osdi-reproducibility

OSDI Reproducibility

Make the numbers survivable. OSDI-specific hooks below (artifact timing, badge scope,

open-access exposure) are 2026-cycle facts verified 2026-07-08; the provenance

discipline is venue-independent engineering.

Why the bar is high at OSDI specifically

Two venue mechanics raise the stakes beyond generic good practice:

• USENIX proceedings are open access from day one. Every reader on the internet —

including the teams whose systems you outperformed — gets the free PDF immediately

and can attempt your numbers. Errors get found publicly and fast.

• Artifact evaluation happens after acceptance (May 8, 2026, 8:59 pm PDT in the

'26 cycle), when the experiments are months old. If provenance was not recorded

while runs happened, the artifact reconstructs a memory, not an experiment.

Reproducibility at OSDI is therefore a recording problem during the project, not a

packaging problem at the end. Packaging is osdi-artifact-evaluation's job; this

skill makes packaging possible.

The provenance ledger

Maintain one machine-readable ledger, committed beside the code, updated by the run

scripts themselves — never by hand after the fact:

# runs/2025-11-14-recovery-scale/ledger.yaml (written by the harness, per experiment)
experiment: recovery-vs-cluster-size      # maps to RQ in the experiment matrix
commit: 4f2c9e1 (system) / 8a11d02 (harness)
hardware: 64x c6525-25g (CloudLab), 25 GbE, NVMe model+fw recorded per node
os_kernel: Ubuntu 22.04, 5.15.0-91; mitigations=on; governor=performance
baseline_versions: replayfs v2.3.1 (tag), ckptstore rebuilt from paper (SHA)
workload: trace block-2025-w2, reconstruction script + source documented
runs: 10 per point; seeds 1..10; outliers kept, plotted as distribution
raw_output: s3://bucket/runs/2025-11-14/... (checksummed)
figures: fig7 <- plot_recovery.py @ 8a11d02 on raw_output

The last line is the anti-drift rule: **every figure in the paper regenerates from

checksummed raw output by a committed script**. If a figure cannot name its script and

input, the number it shows is unverifiable — by the AE committee and by you in June.

What systems papers must pin down

| Dimension | Must record | Common omission that kills reruns |

|---|---|---|

| Hardware | Node model, NIC, storage device + firmware, topology | The NIC/firmware detail that made the difference |

| Software | Kernel version + relevant knobs, dependency lockfile | Sysctl and IRQ-affinity settings applied by hand |

| Baselines | Exact version/tag, tuning applied, build flags | "Default settings" that were quietly edited |

| Workloads | Trace provenance, generation seed, licensing | The preprocessing script that shaped the trace |

| Measurement | Warmup policy, window, timer source, run counts | Which runs were discarded and why |

| Environment | Cluster sharing, power/turbo state, time of run | Co-located tenants distorting tail latency |

Hardware access is the honest limit of systems reproducibility: a result needing 64

specific machines will not rerun on a laptop. The discipline is *disclosure plus

graceful degradation* — document the full testbed, and provide a scaled-down

configuration that exercises every code path even if it cannot reproduce headline

magnitudes.

Determinism where it is cheap, honesty where it is not

Distributed systems are not bitwise-reproducible; do not pretend otherwise. Seed what

can be seeded (workload generation, placement decisions, fault-injection schedules),

report distributions over repeated runs for what cannot, and state which class each

reported number belongs to. A paper that says "recovery time varies ±8% run to run;

we report 10-run distributions" pre-empts the reviewer who reruns and gets a

different point value.

Traces, data, and the licensing wall

Workload provenance is where systems reproducibility most often dies quietly:

• Production-derived traces usually cannot be redistributed. Decide the release

posture before the evaluation depends on them: either obtain redistribution

rights early, or build a documented reconstruction pipeline (statistical profile →

generator → validation against the original) and treat the generator as part of

the artifact.

• Public traces still need version pinning and checksums — public archives

reorganize, and "the standard trace" is not an identifier.

• Sensitive measurements (multi-tenant clusters, user-facing services) need the

anonymization step documented as code, because it changes distributions and a

reproducer must know how.

• Whatever the posture, the paper states it in one honest sentence; discovering an

unreleasable dataset during artifact evaluation reads far worse than declaring it

in the submission.

The shared smoke checker in

[../../resources/code/README.md](../../resources/code/README.md) catches structural

gaps (missing README/manifest/license) but none of the above — licensing and

provenance are judgment calls only the authors can make.

Timing against the 2026 cycle

• During experiments (autumn) — the ledger above, enforced by the harness.
• Silent review window (Dec–Mar) — freeze the testbed image, trace archives, and

environment; the '26 Call for Artifacts encouraged preparing artifacts while the

paper was under consideration, and a conditional accept may demand new runs on the

frozen setup (osdi-author-response).

• After notification (Mar 26) — packaging sprint to the May 8 artifact deadline;

in 2026 the badge evaluated was Artifacts Available, so permanent archiving is

the floor — but a ledger-backed artifact is what makes the optional two-page

Artifact Appendix in the final paper worth writing (osdi-camera-ready).

The handoff test

The standing acceptance test for all of the above: a new group member, given only

the repository and the ledger, regenerates one paper figure on the scaled-down

configuration without asking anyone anything. Run it quarterly and before each

gate (submission, artifact deadline, final paper). Every question they are forced

to ask is a missing ledger entry; every mismatch they hit is drift between paper

and artifact that an AE evaluator — or a public reproducer holding the open-access

PDF — would have found later, with an audience.

Output format

[Ledger] exists + harness-written? gaps: <dimensions from the table>
[Figure regeneration] all figures script+input traceable? failures: <list>
[Determinism statement] seeded vs distributional numbers classified? yes/no
[Testbed freeze] image/trace archive frozen for the review window? yes/no
[AE readiness] distance from ledger to packageable artifact: <low/med/high>