icde-reproducibility

ICDE Reproducibility

Use this before submission and again before camera-ready. ICDE authors are expected to submit

supplemental material, and its availability is weighed in the evaluation — so reproducibility

is not optional polish, it is scored evidence.

Evidence map

• Map each performance claim to a verifiable location: a figure regenerated from logged

runs, a workload script, or a documented measurement in the supplement.

• Pin the environment: CPU, memory, storage device (the NVMe/SSD/HDD distinction changes

results), OS and kernel, database/library versions, compiler flags, and any cluster topology.

• Pin the data: dataset provenance, construction steps, scale factors, and for synthetic

data the generator with its seeds — a workload nobody can regenerate is not reproducible.

• Pin the variance protocol: how many runs, warm-up handling, how outliers are treated, and

whether reported bars are standard deviations, confidence intervals, or percentiles.

• Document baseline tuning: the configuration and tuning budget given to each competitor.

Reproducibility here means a reader can re-run the fair comparison, not just your system.

• Trace figures to raw data: emit tables and plots from logged results so the PDF numbers

and the supplement cannot drift apart.

Systems-reproducibility audit table

| Dimension | Weak answer | ICDE-ready answer |

|---|---|---|

| Hardware | "a modern server" | Exact CPU, RAM, storage device model, and topology |

| Data | "a large dataset" | Named dataset or a seeded generator with scale factors |

| Variance | one median number | N runs with declared spread and warm-up policy |

| Baselines | "we compared to X" | X's config and tuning budget, re-runnable |

| Figures | hand-entered numbers | Plots emitted from logged runs by a script |

Degrees of reproducibility

• Turnkey: run_all.sh regenerates every figure from logged seeds on a documented machine;

run_small.sh gives a fast subset for a reviewer with limited hardware.

• Scripted: scripts exist but need documented manual steps or restricted-data access.
• Descriptive: prose detailed enough that a competent engineer could rebuild the pipeline.

For ICDE, aim for turnkey on the synthetic experiments — a reviewer will re-run a generator

far sooner than they will provision a cluster — and scripted for large real-data or

proprietary-hardware runs, with deviations documented. State the level you actually achieved;

overpromising turnkey behavior that fails on a clean machine is worse than an honest "scripted."

Vignette: a throughput-plus-latency paper

A submission claims higher ingestion at bounded read-latency cost. Its reproducibility spine:

the storage device and queue-depth settings, the workload generator with append-to-scan

parameters and seeds, the run count and percentile policy for the latency tails, the baseline

LSM's compaction configuration, and a run_small.sh that reproduces the headline crossover on

a single machine in minutes — plus one honest sentence on any result that needs the full

cluster.

Single-blind note

• ICDE supplemental material need not be anonymized — names may stay on the repository and

in commit history. Spend the saved effort on making the package actually run, not on scrubbing

identity a double-blind venue would demand.

Output format

[Claim inventory] <claim -> evidence location>
[Environment pinned] complete / partial / missing <what>
[Variance protocol] <runs / spread type / warm-up>
[Baseline fairness] <tuning budgets documented? y/n>
[Reproduction level] turnkey / scripted / descriptive
[Fixes before submission] <ordered list>