sigmod-reproducibility

SIGMOD Reproducibility

PACMMOD's author guidelines state the culture plainly: sharing research

artifacts should be the norm, and papers are expected to make code, data,

scripts, and notebooks available where possible — encouraged rather than

mandatory for acceptance, but reviewers read availability as a credibility

signal. This skill covers reproducibility as engineered into the paper;

the post-acceptance badge process lives in sigmod-artifact-evaluation.

Provenance chain, not vibes

A database paper's result is a function of code version, configuration,

dataset, workload, and hardware. Reproducibility means the paper pins all

five for every number it prints.

| Layer | Must be recoverable from paper + artifact | Where it usually hides |

|---|---|---|

| Code version | Commit or tag behind each experiment | "latest" at submission time |

| Configuration | Buffer sizes, thread counts, compaction/GC settings, flags | Defaults nobody recorded |

| Dataset | Source, version, generator seed, scale factor | "standard benchmark data" |

| Workload | Query mix, arrival pattern, skew parameters, warm/cold state | The harness script |

| Hardware | CPU model, cores, RAM, storage class, network, OS/kernel | A single sentence, if any |

The practical test: could a competent stranger fill this whole table from

your materials? Every empty cell is a review question waiting to be asked in

the feedback phase.

Systems numbers need distributions

Throughput and latency are noisy. The SIGMOD-credible floor:

• Repeated runs with the repetition count stated; medians or means with a

spread measure, never a single lucky run.

• Latency reported at named percentiles (p50/p95/p99), since tail behavior

is often the actual contribution.

• Warm-up policy stated: what was cached, JIT-compiled, or pre-compacted

before measurement started.

• Cross-run controls named: same machine, isolated tenancy, pinned cores,

disabled turbo — whatever was actually done, said explicitly.

Fairness to baselines is reproducibility too

The most contested numbers in a data-systems paper are the competitor's.

Record and disclose: which version of each baseline, who tuned it and how,

which of its features were enabled, and whether its authors' recommended

configuration was used. An artifact that reproduces your system but ships

an untuned strawman baseline reproduces the wrong thing.

Repro debt ledger

Track gaps while writing rather than reconstructing at deadline:

# repro-ledger.md (kept in the paper repo)
| Paper item | Script | Data pinned | Config pinned | Runs/variance | Status |
|-----------|--------|-------------|---------------|---------------|--------|
| Fig 6     | exp/f6.sh | yes (sf=100, seed 41) | yes | 5 runs, p50/p99 | OK |
| Fig 7     | exp/f7.sh | yes | NO — flags undocumented | 1 run | DEBT |
| Tab 2     | manual | partial | yes | n/a | DEBT: script it |

Rule: nothing with status DEBT appears in the submitted PDF. The ledger later

becomes the ARI claims map almost verbatim.

Pinning, mechanically

The cheapest insurance is captured at run time, not reconstructed later:

# stamped into every experiment's log directory by the harness
git -C "$ENGINE_DIR" rev-parse HEAD          > meta/engine.commit
"$ENGINE" --dump-effective-config             > meta/config.effective
sha256sum data/*.bin                          > meta/datasets.sha256
lscpu; free -h; uname -r; nvme list          >> meta/hardware.txt
echo "$WORKLOAD_SEED $QUERY_MIX $SKEW"        > meta/workload.params

Five commands, and the provenance table fills itself for every figure. The

--dump-effective-config line matters most: defaults you never set are

still part of the experiment, and engines change defaults between versions.

Data you cannot publish

Industrial collaborations often involve proprietary workloads. The accepted

pattern at SIGMOD: characterize the private data (sizes, distributions, skew,

schema shape), provide a public or synthetic stand-in that exhibits the same

phenomena, run headline experiments on both, and say which conclusions are

supported by the public path alone. A paper whose every claim requires

private data cannot be badged and will be read skeptically.

Multi-round consistency

Because SIGMOD reviewing spans rounds, reproducibility discipline has a

second job: your own revision. Reviewers may demand new experiments with a

one-month window — regenerating the whole evaluation under a new flag is only

survivable if the original runs were scripted, seeded, and logged. Teams that

hand-ran their plots discover this during the revision, at the worst time.

What reviewers can check without running anything

Even reviewers who never open the artifact test reproducibility passively:

do the numbers in the abstract, the results section, and the conclusion

agree; do figure axes and caption units match the prose; does the claimed

hardware plausibly fit the claimed dataset in memory; do percentages in

tables sum sensibly. Internal inconsistency is read as evidence that the

pipeline is hand-operated — and it usually is. A final numeric-consistency

pass over the PDF is reproducibility work, not copyediting.

Output format

[Sharing posture] full artifact / partial / withheld with stated reason
[Provenance table] complete cells vs. gaps, per figure and table
[Variance floor] repetitions, spread measures, percentile reporting
[Baseline fairness] versions, tuning provenance, config disclosure
[Private-data plan] public stand-in coverage of headline claims
[Debt ledger] items that must close before the round deadline