edbt-experiments

EDBT Experiments

Use this before submission when the evaluation is not yet locked. EDBT reviewers are database-systems

empiricists; the evaluation is where a good idea is won or lost. The organizing principle is

evidence proportional to the claim — the study must measure the thing the paper actually asserts,

on workloads, datasets, and scales a skeptic would accept, against baselines a skeptic would accept.

Evaluation audit

• Match evidence to the claim shape. A claim about latency needs latency measurements under a

realistic workload; a claim about scalability needs runs across a realistic range of sizes/nodes;

a claim about space needs memory/footprint numbers; a claim about accuracy needs a labeled

ground truth. "Faster on a dataset" is not evidence for a scalability claim.

• Use real workloads and datasets, named and sourced (standard benchmarks, real query logs, real

corpora), not a single toy input. Say how the workload was derived.

• Choose fair, tuned baselines, including the strongest current technique and a

simple-but-reasonable alternative, configured with a documented, equal budget. An untuned baseline

is the most common EDBT reviewer objection.

• Measure honestly: report variance across repeated runs, warm/cold state, the metric definition,

and the hardware/cluster configuration. State what you controlled and what you did not.

• Cover the regimes: where the technique helps, where it is neutral, and its overhead or failure

cases — quantified, not asserted.

• Make the harness reproducible (see edbt-reproducibility): the evaluation should re-run from

the artifact rather than being re-measured from scratch.

Claim-to-evidence design table

| Database claim | Matching evidence | Reject pattern avoided |

|---|---|---|

| "Lower query latency" | Latency under a named workload vs. tuned baseline, with variance | "One unnamed dataset, no baseline config" |

| "Scales to large data / many nodes" | Runs across a realistic size/node range | "Only small inputs / single node tested" |

| "Lower space / memory" | Footprint measured under realistic load | "Asymptotic argument, no measurement" |

| "Robust to skew / adversarial input" | Results across skew levels incl. worst case | "Only uniform / benign inputs" |

| "General across engines / settings" | Multiple engines or configurations + explicit limits | "One engine, claimed universal" |

Measurement discipline (database-systems flavor)

[Workload]     name it, source it, say how it was derived; prefer real logs / standard benchmarks
[Baseline]     the strongest current technique, TUNED, with the configuration documented
[Scale]        realistic sizes and node counts; report where behavior changes
[Variance]     repeated runs; report spread, not a single best number
[State]        warm vs cold, cache effects, and what was controlled
[Environment]  hardware, memory, network, engine build/commit — enough to size a reproduction

The Experiments & Analysis paper (a distinct, higher bar)

When the paper's contribution is the study — a benchmarking, repeatability, or comparative

analysis — the methodology is not support, it is the deliverable:

[Subjects]     the systems/techniques compared, chosen by a stated, defensible criterion
[Coverage]     the workload and parameter space actually spanned, and what was left out and why
[Fairness]     every compared system tuned by its own experts' guidance, not just yours
[Repeatability] the harness re-runs the whole comparison from the artifact
[Findings]     the analysis, with the surprising or actionable results foregrounded

An Experiments & Analysis paper that tunes only its authors' preferred system, or spans a workload

space too narrow to generalize, fails on its core contribution, not on a side point.

Vignette: evaluating a query-processing operator

Suppose the paper claims an operator lowers straggler time under skew. The matching plan: derive

workloads from real query logs at several skew levels; run the operator and a tuned skew-aware

baseline across 8-128 workers; report straggler time and total latency with variance; measure the

overhead on skew-free workloads to bound the worst case; and state the boundary (very short queries,

undetectable skew) with a measurement — every number traceable to a logged run in the artifact.

Output format

[Evaluation readiness] strong / adequate / weak
[Claim -> evidence map] <claim: workload / metric / scale>
[Baseline fairness] <baseline -> tuned? equal config? documented?>
[Scale + variance] <realistic range tested? variance reported?>
[Regimes] <helps / neutral / cost / failure all measured? yes/no>
[E&A bar (if applicable)] <methodology, coverage, fairness, repeatability adequate?>
[Decision-critical next run] <one experiment to add>