icde-artifact-evaluation

ICDE Artifact Evaluation

Use this for evidence packaging around ICDE. ICDE expects authors to submit supplemental

material and considers its availability in the evaluation, so a strong artifact directly

raises the paper's floor even where no separate badge process runs. Confirm whether the current

edition also runs a post-acceptance reproducibility/badge track (待核实) before promising

evaluators anything.

Artifact plan

• Decide what evidence a builder needs to believe the numbers: the system source, the

workload generators, the baselines' configurations, the datasets or their construction

scripts, seeds, and logs.

• Keep decision-critical evidence in the paper or its figures; the artifact demonstrates

reproducibility, it does not replace the argument.

• Provide a minimal reproduction map: build steps, dependencies, hardware assumptions

(especially the storage device), commands, expected outputs, runtime, and known

nondeterminism sources.

• For restricted or proprietary data, give enough provenance and construction detail for a

credible re-run without violating data-use terms.

• After acceptance, publish the public, licensed, tagged version whose commit produced the

paper's numbers.

What ICDE evidence reviewers open first

| Claim type | First artifact inspected | Common failure caught |

|---|---|---|

| Throughput/latency win | run_small.sh and the workload generator | Numbers cannot be regenerated; generator or seeds absent |

| "Mechanism causes the gain" | The ablation toggle in the code | The toggle does not exist; gain not isolable in the artifact |

| Baseline comparison | The baseline's config files | Baseline was untuned or run with defaults |

| Scale claim | The scale-factor sweep script | Only one scale factor is actually runnable |

| Cost/overhead claim | The instrumentation that measures cost | Cost is asserted but not measured anywhere |

Because ICDE reviewers are builders, they will re-run a small benchmark far sooner than they

will provision a cluster — make run_small.sh reproduce the headline crossover on one machine

in minutes before polishing anything else.

Worked vignette: packaging a storage-engine benchmark

A submission proposes a write-optimized index validated on a telemetry trace and a synthetic

sweep.

• Ship the workload as a parameterized generator (append rate, append-to-scan ratio, key

distribution), not constants buried in a driver, so reviewers can vary the regime.

• Record the exact seed sequence and run count behind every throughput and latency-tail

figure; percentile claims are meaningless without them.

• Emit figures directly from logged runs so PDF and artifact numbers cannot drift.
• Include the ablation switch and the baseline configs so a reviewer can reproduce both

the effect and the fair comparison.

Single-blind and logistics anchors

• ICDE is single-blind: the artifact need not be anonymized — leave author names and history

in place; spend the effort on making it build and run.

• Assume, absent a formal badge track, that only the README and one entry script get opened;

design for that. If a badge/reproducibility process does run this edition, read its criteria

before packaging.

• Upload size limits and accepted formats vary by edition; verify against the current CMT

submission form.

Output format

[Artifact role] scored supplement / post-acceptance reproducibility / public archive
[Contents] <source / generators / baseline-configs / logs / claims-map>
[Turnkey check] <does run_small.sh reproduce the headline result? y/n>
[Isolability] <is the mechanism ablation runnable in the artifact? y/n>
[Hygiene] <secrets / caches / bloat removed>
[Fixes before upload] <ordered list>