asplos-reproducibility

ASPLOS Reproducibility

Systems results decay fast: a kernel update, a microcode revision, or a silently

changed simulator default can move numbers by more than the paper's claimed margin.

Reproducibility work at ASPLOS is therefore state capture — recording the full

machine, model, and toolchain state behind every figure — done while the experiments

run, not reconstructed at camera-ready time. It also front-loads artifact

evaluation: the badge criteria (asplos-artifact-evaluation) are exactly a demand

that this state capture exists and works.

The state ledger

Maintain one ledger row per experimental platform, committed alongside results:

| Layer | Capture | Why it moves numbers |

|---|---|---|

| Silicon | CPU model + stepping, memory config/topology, device (e.g. CXL expander) firmware | Steppings differ in errata and prefetch behavior |

| Firmware/BIOS | Microcode revision; SMT, turbo, prefetcher, C-state, NUMA settings | Any one knob can swamp a 10% effect |

| OS | Kernel version + full config, relevant sysctls, mitigations state | Speculation mitigations alone shift syscall-heavy results |

| Toolchain | Compiler + flags, libraries, runtime versions | -O level and allocator choice are classic silent variables |

| Simulator | Exact commit, all config files, region/checkpoint method, warm-up length | Defaults change across releases without notice |

| FPGA | Board, toolchain version, constraints, bitstream hash, achieved clock | Re-synthesis at a different clock is a different experiment |

| Workloads | Suite versions, input sets, trace provenance and preprocessing | "SPEC" without input class is unrepeatable |

| Randomness | Seeds for any stochastic component + run counts | Needed for the dispersion numbers to mean anything |

Scripted capture beats remembered capture

Run at the start of every measurement session; store output next to the data:

#!/bin/sh
# state-capture.sh — commit this file and its output with each result set
uname -a; cat /proc/cmdline
grep -m1 'model name' /proc/cpuinfo; grep microcode /proc/cpuinfo | sort -u
cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u
cat /sys/devices/system/cpu/smt/control 2>/dev/null
numactl --hardware 2>/dev/null | head -5
cc --version | head -1
git -C "$SIM_DIR" rev-parse HEAD 2>/dev/null   # simulator commit
sha256sum "$BITSTREAM" 2>/dev/null              # FPGA bitstream identity

The hardware-access problem, named honestly

ASPLOS artifacts often need hardware an independent evaluator will not have. The

honest pattern is a three-tier availability statement drafted at submission

time:

1. Repeatable anywhere: simulator experiments and analysis scripts — full

configs and one command per figure.

1. Repeatable with named hardware: the exact platform requirements (board,

expander, CPU family), plus what to expect if the evaluator's part differs.

1. Not independently repeatable: results on lab-only or pre-production

hardware — say so, and provide either supervised access, raw logs with the

analysis pipeline, or a scaled-down proxy. Silence here reads as concealment;

a stated limitation reads as engineering.

Claim-preservation, not number-worship

State which conclusions should survive environmental drift and which are

environment-specific: "the ordering of policies is stable across kernels 6.6-6.9;

absolute runtimes are not." This single sentence pattern prevents the most common

failed-reproduction dispute — an evaluator matching your ordering but not your

absolute numbers and calling it a failure.

Timing across the ASPLOS cycle

• Before September 9: ledger current; capture script in the repo; availability

tiers drafted (they inform the paper's own text).

• Response window: the ledger is your defense when a reviewer doubts a number —

you can state the exact conditions instead of hand-waving.

• Major Revision: re-run under the captured original state where possible;

where the environment has drifted, disclose the drift in the change note.

• After acceptance: the ledger becomes the Artifact Appendix's dependency

section nearly verbatim; AE calendars for 2027 were 待核实 at pack-check time, so

confirm dates when notified.

One command per figure

The internal gold standard that makes everything downstream cheap: every figure

and table in the paper regenerates from a single committed command that reads

raw results and emits the exact plot. It catches stale-figure bugs before

submission, turns response-window questions into lookups, and becomes the

Reproducible-badge run script with a rename. Institute it at the first result,

when it costs minutes — retrofitting it at camera-ready costs days.

Trace and dataset provenance

Workload inputs decay independently of code. For each trace or dataset, record

origin (public suite version, generated-by script + seed, or production source),

preprocessing steps as scripts rather than prose, and a checksum of the exact

bytes used. Production traces that cannot be released need a characterization

(rate, skew, working-set curves) plus a matched synthetic generator committed to

the repo — this is also the anonymity-safe form for submission, since a raw

trace can identify its owner.

When numbers drift between submission and revision

The Major Revision window arrives months after the original runs, and

environments drift. Protocol:

1. Re-run a sentinel subset (three representative experiments) under the

captured original state before starting revision work; if the sentinels

reproduce, extend confidently.

1. If they do not, bisect the ledger — kernel, microcode, simulator commit —

until the moved variable is found; the ledger exists for exactly this moment.

1. Disclose in the change note which results were re-collected and under what

changed conditions, and re-state the claim-preservation sentence for the new

environment. Silent regeneration of all numbers invites a reviewer to ask

which version was real.

Output format

[Ledger coverage] platforms with complete rows: N/N · gaps listed
[Capture automation] script committed + outputs stored with data: Y/N
[Simulator pinning] commit + configs + region method + warm-up recorded: Y/N
[Availability tiers] anywhere / named-hardware / not-repeatable — each populated
[Claim preservation] drift-stable vs environment-specific conclusions stated: Y/N
[Badge readiness] which badges the current package could already earn