icsme-reproducibility

ICSME Reproducibility

Use this before submission and again before camera-ready. ICSME's ROSE-Festival culture makes

reproducibility a valued, sometimes scored dimension rather than a courtesy: double-anonymous

review already expects an inspectable artifact, and the **Joint Artifact Evaluation Track and ROSE

Festival** rewards a permanent, open one with IEEE badges. The goal is that a competent maintainer

could rebuild your evidence — re-mine the corpus, re-run the technique — and reach your conclusions.

Evidence map

• Map each research-question answer, technique claim, and reported number to a **verifiable

location** — a paper section, a table generated from logged data, or a script in the artifact.

• For techniques, give enough of the algorithm, parameters, and environment that a reader could

re-implement or re-run it on their own systems.

• For mining/evolution studies, report subject systems and their selection, the extraction pipeline,

preprocessing, metrics, statistics, and the analysis scripts.

• Keep the data-availability statement truthful and specific: what is shared (corpus, scripts,

subject list), where it will live after acceptance, and — if something cannot be shared

(proprietary legacy code, private history) — exactly why.

• Keep the paper and the artifact consistent: a number in the PDF that no script in the artifact

produces is the contradiction reviewers read as carelessness.

Data-availability statement audit

| Claim in the paper | Weak availability answer | ICSME-ready answer |

|---|---|---|

| "We mine N evolving projects" | "Dataset available on request" | Anonymized archive of the exact project list, SHAs, and extraction scripts |

| "Our tool detects debt hotspots" | "Code will be released" | Anonymized, runnable tool with a README and a small demo on a bundled repo |

| "We surveyed/interviewed P maintainers" | Nothing (privacy cited vaguely) | Anonymized codebook, protocol, and aggregate data; stated ethics limits |

| "The model summarized the change" | Live API described | Cached prompts and raw responses, model IDs and access dates |

"Available on request" is treated as not available under ROSE norms; convert every such line into

a concrete, anonymized artifact or an explicit, justified exception.

Provenance pinning

[Mining]   pin repository SHAs; record extraction date and the studied time window; archive the
           extracted corpus, not just the query; document fork/duplicate/bot handling
[LLM]      record exact model identifiers + access dates; cache raw inputs and outputs; report
           sampling settings; prefer post-cutoff subjects to bound contamination
[Compute]  state hardware, mining/runtime cost, and number of runs so a reader can size a reproduction
[Randomness] log seeds for any stochastic step; say what is and is not deterministic

Degrees of reproducibility (state the one you achieved)

• Turnkey: one documented command regenerates each table/figure from logged data.
• Scripted: scripts exist but require documented manual steps or access to a large mined corpus.
• Descriptive: prose detailed enough that a competent reader could rebuild the pipeline.

For ICSME, aim turnkey for anything a reviewer might rerun quickly (a detection script on a sample

repo, a plot from logged results); a large mined corpus or proprietary history may stay scripted

with access clearly documented. Stating the achieved level honestly beats promising turnkey

behaviour that fails on a clean machine — especially with no revision round to fix it in.

Vignette: a mixed-methods evolution study

Consider a study combining mined release history with a maintainer survey. Its reproducibility

spine: the mining scripts with pinned SHAs and extraction dates; the anonymized extracted dataset

with the exact subject-system list; the survey instrument and anonymized responses; the qualitative

codebook with inter-rater agreement; and the analysis notebooks that turn all of it into the

paper's tables — plus one honest sentence about the parts (identities, private forks) that cannot be

shared and why.

Consistency and camera-ready / ROSE pass

• Before submission: every scored number traces to the artifact; the data-availability statement

matches reality; the artifact is anonymized (no owner strings, cluster paths, lab names, or your

own org's repos).

• Before camera-ready: swap anonymized links for a permanent, DOI-issuing archive, and align the

statement with the IEEE badges you pursue in the ROSE Festival (icsme-artifact-evaluation).

Output format

[Claim inventory] <claim -> evidence location>
[Data availability] concrete / vague / missing
[Provenance gaps] <mining SHAs / LLM caching / seeds / compute>
[Reproducibility level] turnkey / scripted / descriptive, stated honestly
[Paper fixes] <must appear in the PDF, since there is no revision round>
[Artifact fixes] <additions before upload>