icde-experiments

ICDE Experiments

Use this before submission when the evaluation is not yet locked. At ICDE, the experiments are

where a systems paper is won or lost; a builder-heavy committee reads them adversarially.

Experiment audit

• Map each performance claim to a figure, table, or measured crossover — no orphan claims.
• Run tuned baselines, not straw men. Document the tuning budget you gave each competitor;

an untuned baseline is the single most common ICDE reject trigger.

• Prefer curves over points: show behavior across scale factors, data sizes, or contention

levels, not one operating point where you happen to win.

• Report throughput and tail latency with variance — multiple runs, and captions that say

whether bars are standard deviations, confidence intervals, or percentiles. A p99 that hides

behind a median is a caught omission.

• Add ablations that isolate the mechanism: turn off the one idea and show the gain

disappears, so the improvement cannot be attributed to unrelated engineering.

• Disclose the cost of every gain: memory, write amplification, CPU, or read-latency

penalty. A loss map is a credibility signal, not a weakness.

• Report **hardware, OS, storage device, software versions, dataset construction, and workload

generators with seeds** so the numbers are reconstructable.

What experiments are for at this venue

• ICDE experiments exist to establish and isolate a systems effect, not to top a

leaderboard. One clean ablation proving the mechanism causes the gain outweighs five more

datasets where it merely correlates.

• The strongest design pairs a synthetic generator you can sweep (to find the crossover and

stress the mechanism) with a real workload (to show practical relevance). State where the

synthetic regime matches the real one and where it does not.

• Reviewers check whether the operating regime — data size, concurrency, hardware — is one

where the claimed effect should even appear; a cache-resident benchmark cannot support a

claim about I/O-bound behavior.

Evidence-burden table

| Claim | Matching evidence | Reject pattern avoided |

|---|---|---|

| "The problem exists" | Baseline stalls/degrades on the target workload | "Solving a problem nobody has" |

| "Our mechanism causes the gain" | Ablation toggling the one idea | "Gain from unrelated engineering" |

| "It scales" | Curve across scale factors, not a single point | "Single-point win, unknown scaling" |

| "The cost is acceptable" | Disclosed memory/CPU/latency cost table | "Free-lunch claim, hidden trade-off" |

| "It generalizes" | A real workload beside the synthetic sweep | "Only synthetic, no real relevance" |

Vignette: evaluating a write-optimized index

A paper claims higher ingestion by deferring ordering. The matching plan: sweep the

append-to-scan ratio synthetically to locate the crossover, run a real telemetry trace at the

claimed rate to show the baseline stalls where the new index does not, ablate the deferral to

prove it is the cause, and report the read-latency tail and memory cost so the trade is legible

— every panel tied to a claim in the text.

Reporting floor

• Multiple runs and reported variance for every performance figure; name the metric precisely

(throughput, p50/p99 latency, amplification).

• Report the hardware and the compute actually consumed, not vague feasibility language.

Output format

[Experiment readiness] strong / adequate / weak
[Claim -> evidence map] <claim: figure/table/crossover>
[Baseline fairness] <tuning budget disclosed? untuned competitors?>
[Missing evidence] <scale curve / variance / ablation / cost disclosure>
[Reproducibility gaps] <hardware / versions / generators / seeds>
[Decision-critical next run] <one experiment>