eurosys-reproducibility

EuroSys Reproducibility

Use this while experiments are still running — reproducibility retrofitted in

deadline week is transcription, not engineering. The venue context: EuroSys

papers live on measured performance claims, the community runs a badge-granting

artifact evaluation (sysartifacts.github.io), and SIGOPS has publicly digested

five years of EuroSys AE lessons (sigops.org blog, 2025; rendered 2026-07-08).

A paper whose numbers cannot be regenerated by its own authors three months

later fails both review-time scrutiny and post-acceptance AE.

The provenance ledger

Keep one machine-readable record per reported number. Minimum fields:

| Field | Why EuroSys reviewers care |

|---|---|

| Commit hash of system under test | "Which version got 2.1x?" is a real AE question |

| Baseline name + version + config | Untuned-baseline suspicion is the venue's default |

| Hardware: CPU/RAM/NIC/storage, topology | Systems results rarely transfer across boxes |

| OS/kernel, key library versions | Kernel changes move I/O and scheduler numbers |

| Workload/trace + generator seed | Trace provenance is checked, not assumed |

| Repetitions, warm-up policy | Distinguishes measurement from anecdote |

| Timestamp + raw-output path | Lets you rebuild any figure from raw logs |

Variance discipline

Single-run numbers are the most common silent reproducibility failure in

systems evaluation:

• Repeat every headline measurement enough times to see its spread; report

median plus an explicit dispersion measure (stdev, IQR, or min–max), and say

in the caption which one it is.

• Isolate noise sources you control: pin frequencies, disable turbo where it

distorts comparisons, note co-located load, randomize run order across

systems so drift does not favor yours.

• Tail metrics (p99 and beyond) need far more samples than means; state the

sample count whenever a tail latency is claimed.

• When a difference is within run-to-run spread, say so — EuroSys reviewers

reward calibrated claims over uniform victory narratives.

Automation floor

The practical bar: any figure regenerates from raw data with one command.

# Layout that keeps figures honest
experiments/
  fig7_throughput/
    run.sh        # executes the sweep, writes results/*.csv with metadata header
    plot.py       # reads results/, emits fig7.pdf — no hand-edited numbers
    results/      # raw outputs, never overwritten, one dir per run timestamp
make fig7         # the only path by which fig7.pdf ever changes

If a plot was ever touched manually, its provenance is broken and the AEC will

find the discrepancy before you do.

Availability statement, two audiences

• Review time (double-blind): describe what exists — "an anonymized

repository containing the system, workload generator, and run scripts

accompanies the submission" — without leaking the lab's identity through

URLs, paths, or commit authors.

• Camera-ready / AE time: replace with the DOI-backed archive and the badge

set being sought. Restricted traces (production data, partner NDAs) need an

honest fallback: a synthetic generator calibrated to the trace's published

statistics, with the calibration method described.

Restricted evidence, stated honestly

Some EuroSys evidence legitimately cannot ship — production traces under

NDA, partner clusters, proprietary workloads. The honest pattern:

• Name the restriction and its scope precisely ("the ingestion trace from

operator X cannot be released; its summary statistics are in Table 3").

• Ship a calibrated synthetic substitute and the calibration procedure, so

external readers can approximate the regime.

• Keep at least one headline result on fully public inputs; a paper whose

every number depends on unreleasable data asks reviewers for faith the

venue does not trade in.

• Never let the availability paragraph imply more openness than the AEC

will find; the badge process makes overstatement visible in print.

Pre-deadline reproducibility drill

One week before the paper gate, run the drill on a machine that never ran

the experiments:

1. Clean checkout, environment build from the lockfile alone — record every

undocumented step it turns out to need.

1. Regenerate two figures end to end: one cheap, one expensive.
2. Diff regenerated numbers against the draft's numbers; investigate any

drift beyond the reported dispersion.

1. Fix the documentation, not just the outcome — the drill's product is the

README the AEC will eventually read.

Quick self-test

1. Can a new student regenerate Figure 7 from a clean checkout in one command?
2. Does every table cell trace to a raw log file with hardware metadata?
3. Are repetition counts and dispersion visible for every performance claim?
4. Would the availability paragraph survive both anonymity and the AEC?

Output format

[Repro grade] regenerable / scripted-with-gaps / manual
[Ledger coverage] <numbers with full provenance / total reported numbers>
[Variance findings] <single-run claims, missing dispersion, undersampled tails>
[Workload provenance] <traces and generators with version + seed status>
[Availability draft] <review-time text and camera-ready plan>