nsdi-reproducibility

NSDI Reproducibility

A networked-systems result is a function of topology, traffic, timing, and code — and

three of those four are absent from the PDF unless deliberately recorded. NSDI

rewards the discipline institutionally: artifact badges after acceptance and a

Community Award for the best paper whose code and/or dataset is public by the

final-papers deadline. But the work happens during the experiments, not after the

decision email.

The four provenance ledgers

Keep each as a versioned file next to the results it explains:

| Ledger | Contents | Loss mode it prevents |

|---|---|---|

| Topology | node specs, NIC/switch models, link speeds, RTT matrix, kernel + NIC settings | "worked on our cluster," unreproducible knee points |

| Traffic | trace source + collection context, scaling/anonymization transforms, synthetic-generator parameters + seeds | headline numbers tied to data nobody can regenerate |

| Configuration | every config diff from defaults, per system and per baseline | untuned-baseline accusations you cannot rebut |

| Run | per-experiment: commit hash, config snapshot, fault schedule, raw-log location, analysis-script hash | figures that cannot be regenerated at revision time |

The run ledger has a second life NSDI makes unusually likely: a one-shot revision

letter can demand new measurements on the same setup, 3-4 months after submission,

at a subsequent deadline. Teams with ledgers rerun in days; teams without them

rebuild the testbed from memory.

Determinism is different here

Unlike ML reproducibility, networked-systems results are legitimately

non-deterministic — background load, timer jitter, and cross-traffic vary. The honest

posture is not "identical numbers" but characterized variance:

• Fix what can be fixed: seeds for generators and fault schedules, pinned software

versions, isolated or documented-shared testbeds.

• Measure what cannot: repeat headline experiments across days/trace slices; publish

the spread with the artifact so an evaluator knows a 7% delta is noise, not

failure.

• Timestamp everything; time-of-day and neighbor effects are real explanations, but

only if recorded.

What can actually ship?

Decide per data item before the camera-ready crunch, because the answer shapes the

paper's claims:

1. Public now — code, synthetic generators, testbed configs, analysis scripts.

Default to shipping.

1. Public after transformation — production-derived traces that can be

anonymized/aggregated. Document the transform and what properties it preserves;

state in the paper that released traces differ from raw ones.

1. Never public — proprietary traces, customer data. Plan the fallback: a

synthetic workload with matched characteristics, shipped alongside a fidelity

note. Do not let a never-public trace be the only support for a headline claim

if a shippable proxy can corroborate it.

Anonymize the artifact itself for review-time supplements (repo owners, hostnames,

paths, company strings in configs); de-anonymize only at final-paper time.

Wiring it into the repo

# Layout that keeps paper and artifact from drifting
experiments/
  <exp-id>/run.sh          # topology + config + fault schedule, self-describing
  <exp-id>/provenance.json # commit, trace id, seeds, dates, operator
figures/
  Makefile                 # every paper figure regenerated from raw logs:
                           #   make fig6 -> pulls exp logs, runs analysis, emits PDF
paper/
  claims.md                # claim -> exp-id -> figure mapping, reviewed at freeze

The claims.md cross-map is the cheapest anti-drift device: at submission freeze,

walk it once; any claim whose exp-id is stale gets rerun or reworded. The shared

smoke-checker

([../../resources/code/README.md](../../resources/code/README.md)) covers package

hygiene but not topology/trace fidelity — those checks are yours.

Blind now, open later

Reproducibility material crosses the double-blind boundary twice, and each crossing

has a checklist:

• At submission (if any package is uploaded as auxiliary material): scrub

repository owners, commit-author emails, hostnames that embed the institution,

and license files naming the organization; keep the scrubbed and true versions as

separate branches so de-anonymization is a merge, not a rewrite.

• At final papers: flip to the public, attributed release — DOI'd archive,

real repository, institutional acknowledgment — dated before the final-papers

deadline if the Community Award matters (nsdi-camera-ready).

Cheap wins ordered by payoff

1. provenance.json per experiment, written by the runner script (an hour to

automate).

1. Figure Makefile with raw-log inputs (removes an entire class of revision pain).
2. Baseline config diffs recorded (rebuts the most common reviewer attack).
3. Variance characterization for the two headline results.
4. Early legal check on trace release — licensing takes longer than deadline gaps.

Signs the discipline is slipping

• A figure in the draft that nobody can regenerate this week.
• "The good run" language — selecting runs by outcome rather than by protocol.
• Baseline numbers copied forward from an earlier paper draft instead of re-measured

on the current topology.

• Testbed changes (kernel upgrade, NIC swap) mid-campaign without a ledger entry

splitting before/after results.

Any one of these, found at submission freeze, costs days; found by an artifact

evaluator or a revision reviewer, it costs the result's credibility.

Output format

[Ledger status] topology / traffic / configuration / run — each present/partial/absent
[Variance] headline results with characterized spread? which lack repeats?
[Shippability] data items -> public / transformable / never (fallback named)
[Drift check] claims.md walked? stale claim list
[Award posture] on track for public code+data by final-papers deadline? 
[Next actions] ordered by payoff-per-hour