sigcomm-reproducibility

SIGCOMM Reproducibility

Use this before submission and again before the artifact deadline. SIGCOMM's culture treats a

result as a claim a stranger should be able to rebuild; reproducibility here is largely about

whether the network conditions behind a number are recorded well enough to recreate.

Reopen the current Call for Artifacts to confirm the process for this edition.

Evidence map

• Map each reported number to a rebuildable location: a script, a logged run, a config, and

the figure it feeds.

• For every measurement, record the topology (physical or emulated), the traffic

(workload generator or trace, with provenance), the configuration (every parameter),

and the environment (kernel, NIC, switch firmware, data-plane toolchain).

• For any stochastic result, report the run count, the seed or workload driver, and the

variance behind each percentile — a lone 99th-percentile number with no replication is not

reproducible.

• Decide early what can legally ship: production traces and topologies often cannot, so plan a

substitute with matching statistical character rather than discovering the block at the

deadline.

• Keep the paper and the artifact consistent; a figure the artifact cannot regenerate is a

reproducibility gap reviewers and the AEC both notice.

The four ledgers

| Ledger | What it records | Failure it prevents |

|---|---|---|

| Topology | Nodes, links, capacities, buffer sizes, emulation vs. hardware | "Which topology produced Figure 6?" |

| Traffic | Workload distribution or trace, provenance, which figure used it | Numbers that cannot be tied to an input |

| Configuration | Every parameter and its value per experiment | Silent knob changes between runs |

| Run | Seeds, replication counts, timestamps, variance | Percentiles with no reproducible basis |

Maintaining these while you run is cheap; reconstructing them after the deadline is

error-prone and often impossible.

Degrees of reproducibility

Turnkey     : one command rebuilds each figure from logged runs on a documented setup
Scripted    : scripts exist but need hardware, private data, or documented manual steps
Descriptive : prose detailed enough that a competent networker could rebuild the pipeline
Fragile     : results depend on unrecorded conditions -> a red flag to fix before submission

For SIGCOMM, downscaled testbed results should be turnkey because evaluators actually run

them; full-scale hardware results may stay scripted with the deviation documented. State the

achieved level honestly rather than promising turnkey behavior that fails on a clean machine.

Vignette: a wide-area measurement result

A paper characterizes loss and latency on a production overlay, then proposes a routing tweak.

Its reproducibility spine: the measurement window and vantage points, the sampling method, the

anonymization applied before any release, a public-trace or synthetic substitute for the parts

that cannot ship, and a driver that regenerates the loss-latency figures from logged samples —

plus one honest sentence about which production condition the substitute cannot capture.

Legal and ethical release

• Real user or third-party-infrastructure measurement needs its handling addressed in the

paper; anonymize before release and record what was removed.

• Prefer an archival host with a DOI for anything you can ship, and document the exact reason

for anything you cannot.

Output format

[Claim inventory] <number -> rebuildable location>
[Ledger status] topology / traffic / configuration / run — complete or gaps
[Reproducibility level] turnkey / scripted / descriptive / fragile
[Data-release plan] shippable / substitute (+ provenance) / withheld (+ reason)
[Paper fixes] <must appear in the main PDF>
[Artifact fixes] <what the package still needs>