micro-reproducibility

MICRO Reproducibility

In microarchitecture research the "experiment" is usually a software artifact

pretending to be hardware. That makes results perfectly reproducible in

principle — and silently unreproducible in practice, because a simulator commit, a

config diff, or a trace-generation flag changed between the headline run and the

camera-ready. This skill installs the record-keeping that prevents that.

What must be pinned, by evidence type

| Evidence | Pin these or the number is unrecoverable |

|---|---|

| Cycle-level simulation | Simulator name + exact commit hash; full config files (not "8-wide OoO" prose); patch set applied; host compiler for the simulator build |

| Workload traces | Source binary + compiler flags; trace tool + version; SimPoint interval size, K, and seed; warmup instructions per region |

| Power / area estimates | McPAT/CACTI/Accelergy version; technology node; the XML/cfg input actually fed in; any manual scaling factors |

| RTL results | HDL commit; synthesis tool + version; target library/PDK; constraints file; which corners were run |

| FPGA / silicon | Board/chip stepping; bitstream hash or firmware; frequency; measurement method for power (rail vs counter model) |

| DRAM behavior | Ramulator/DRAMsim3 version + timing config (device datasheet parameters) |

The rule of thumb: **a result is reproducible when a stranger can regenerate the

exact bar in Fig. 3 from the manifest alone**, without asking you anything.

The run manifest

Emit one manifest per experiment batch, generated by the launch script — never

written by hand after the fact:

# results/2026-04-02_headline/manifest.yaml
simulator: {name: champsim, commit: 3fa9c21, patches: [prefetch-fix.diff]}
config:    {file: configs/8w_ooo_baseline.json, sha256: 9be1...}
mechanism: {file: configs/ours_32kb.json, knobs: {table_entries: 2048, tag_bits: 12}}
workloads: {suite: spec2017_rate1, traces: traces/v3/, simpoints: k30_int200M_seed7,
            warmup_M: 200, detailed_M: 1000}
power:     {tool: mcpat, version: "1.3", node_nm: 22, input: mcpat/ours.xml}
host:      {machine: lab-node-14, compiler: gcc-13.2}
outputs:   {raw: stats/, figures: fig3_headline.pdf, script: plot_headline.py}

Commit the manifest with the raw stats files. Figures are regenerated from stats by

a script named in the manifest — a figure that cannot be regenerated does not go in

the paper.

Disclosure lines the paper itself owes

The methodology section (see micro-writing-style, ~1 page) must state, in text:

• Simulator and version, with your modifications summarized honestly ("we added a

cycle-accurate model of X; the interconnect remains the default model").

• The full baseline configuration in a table (pipeline width, ROB, caches with

latencies, predictor, prefetchers, DRAM timing).

• Sampling recipe: how regions were chosen, warmup, and measured instruction counts.
• Power/area model, node, and the sentence "these are model estimates" — the

community's tools (CACTI, McPAT — both MICRO papers themselves) are respected but

known to be approximations, and pretending otherwise reads as naivety.

Determinism and drift checks

• Two identical launches must produce bit-identical stats; if the simulator has any

nondeterminism (thread interleaving in parallel sims), pin it or report run count

and spread.

• Re-run the baseline after every simulator patch — a patch that changes baseline

IPC by 0.5% invalidates every relative number already collected.

• Keep a results-ledger.md: date, manifest path, headline geomean, what changed.

Rebuttal season (June) will ask "what config produced Fig. 7?" — from three

months earlier; the ledger answers in minutes, memory does not.

Validating the instrument itself

Reproducibility of a wrong number is worthless; record the evidence that the

simulator models what the paper says it models:

• Cross-check the baseline against published behavior (see

micro-experiments): store the comparison table — suite, metric, published

range, your value — in the repo, not just in someone's memory.

• Unit-test the mechanism model. Hand-construct access sequences with known

correct outcomes (a burst that must train the table, a pattern that must not)

and assert the simulator's counters match. These micro-tests are the first

thing an AE evaluator can run and the best evidence the model is not a stats

bug.

• Record known fidelity gaps in a LIMITS.md: default interconnect model,

simplified TLB, unmodeled prefetcher interactions. The paper's claims must

stay inside these limits; writing them down keeps deadline-week edits honest.

Anonymity interaction (submission phase)

MICRO 2026 requires artifact links to be fully anonymized or removed. Before the

April upload, sweep the repo you intend to link: git author fields, hostnames in

manifests (lab-node-14 is fine; patt-lab-utexas-14 is not), cluster paths,

funding strings in READMEs. The manifest discipline above makes this sweep

mechanical rather than archaeological.

Storage economics of keeping everything

Raw cycle-level stats for a full paper (dozens of configs × dozens of workloads ×

sweeps) usually fit in single-digit gigabytes — trivially archivable, and the

cheapest insurance in research. Policy: keep all raw stats files, forever,

compressed; regenerate figures on demand. Delete only simulator checkpoints and

intermediate trace files that the recipes can rebuild. When a reviewer asks in

June about a bar from a January run, or AE asks in August, or a follow-up paper

asks in two years, the manifest + stats pair answers; the alternative is

re-simulating weeks of compute under deadline pressure.

Output format

[Pinning audit] per evidence type: pinned / partially pinned / loose (list)
[Manifest coverage] experiments with manifests: N/M
[Figure regen] all paper figures scripted from raw stats: yes / no (list unscripted)
[Methodology text] simulator+mods / baseline table / sampling / model-estimate
                   disclosure: present-absent each
[Determinism] identical-rerun check passed: yes / no / untested
[Ledger] results-ledger current through: <date>
[Anonymity sweep] repo clean for anonymized linking: yes / issues listed