sigmetrics-reproducibility

SIGMETRICS Reproducibility

Use this before submission and again before the POMACS camera-ready. SIGMETRICS reproducibility has

a distinctive shape: the "artifact" is often a proof plus a simulator plus a trace, not only

running code. The goal is that a competent reader could re-derive your bound, re-run your simulation

to the same curves, and re-analyze your measurement to the same conclusions.

Evidence map

• Map each theorem, bound, and reported number to a verifiable location — a proof in an

appendix, a figure regenerated from a seeded simulation, or a script that turns the trace into the

table.

• For analytic results, give the full derivation and every assumption; a reader should be able

to check the proof and see which assumptions each step uses.

• For simulations, ship a seeded simulator whose scripts regenerate each figure and overlay

the analytic prediction, so a reviewer sees model and measurement agree.

• For measurement studies, document the trace source, collection window, sanitization, and the

processing scripts; archive the processed dataset or document access.

• Keep the paper and the artifact consistent: a p99 number in the PDF that no simulator run

reproduces is the contradiction reviewers read as carelessness.

Reproducibility-claim audit

| Claim in the paper | Weak reproducibility answer | SIGMETRICS-ready answer |

|---|---|---|

| "Theorem 1 bounds the tail" | Proof sketch only | Full proof (appendix) + a simulation that matches the analytic curve |

| "We simulate policy X" | "Simulator available on request" | Seeded simulator + scripts that regenerate each figure from logged runs |

| "We measured system Y" | "Data on request" | Processed dataset (or documented access) + provenance + processing scripts |

| "The learner has low regret" | Empirical curve only | Regret proof + code plotting empirical regret against the bound |

"Available on request" is treated as not available; convert every such line into a concrete,

anonymized artifact or an explicit, justified exception (e.g. a proprietary trace, with the

methodology fully documented).

Provenance and determinism pinning

[Proof]       state every assumption; give the full derivation; note which lemmas each step needs
[Simulation]  log seeds; state steady-state/warm-up handling; make figures regenerate deterministically
[Measurement] pin the trace source, collection window, sanitization; archive processed data
[Compute]     state hardware, runtime, and number of independent runs so a reader can size a rerun
[Agreement]   ship the overlay of analysis vs. simulation so the match is reproducible, not asserted

Degrees of reproducibility (state the one you achieved)

• Turnkey: one documented command regenerates each figure/table from logged simulation runs and

reproduces the analytic overlay.

• Scripted: scripts exist but require documented manual steps or access to a restricted trace.
• Descriptive: proofs and methodology detailed enough that a competent reader could rebuild the

pipeline.

For SIGMETRICS, aim turnkey for anything a reviewer might rerun quickly (a simulation regenerating a

figure, a script producing a table); a proprietary industrial trace may stay scripted with access

documented, but the methodology and the analysis code should still be turnkey.

Vignette: a queueing-theory-plus-measurement paper

Consider a paper with a scheduling theorem and a trace-driven evaluation. Its reproducibility spine:

the full proof with stated assumptions in an appendix; a seeded simulator whose notebook regenerates

the analysis-vs-simulation figure; the trace-processing scripts with pinned provenance; the

anonymized processed dataset (or documented access to a restricted one); and the analysis notebooks

that turn logged runs into the paper's tables — plus one honest sentence about any assumption that

only approximately holds and how §6 bounds it.

Consistency and camera-ready pass

• Before submission: every reported number traces to a proof, a logged simulation run, or a

measurement script; the artifact is anonymized (no owner strings, cluster paths, group names).

• Before camera-ready: swap anonymized links for a permanent, DOI-issuing archive, and align the

artifact with any ACM badges you are pursuing (sigmetrics-artifact-evaluation).

Output format

[Claim inventory] <claim -> proof / simulation run / measurement script>
[Reproducibility] concrete / vague / missing, per claim
[Provenance gaps] <proof assumptions stated? seeds logged? trace provenance pinned?>
[Reproducibility level] turnkey / scripted / descriptive, stated honestly
[Paper fixes] <must appear in the PDF/appendix>
[Artifact fixes] <additions before upload>