vldb-experiments

VLDB Experiments

Use this before the evaluation section hardens. At this venue the experiments

are the argument: a PVLDB reviewer typically flips from the introduction

straight to the plots and decides how skeptically to read everything else.

What the evaluation must establish

Four distinct burdens, each needing its own experiments:

1. The problem exists — measure the baseline failing on a credible

workload before showing your fix.

1. The mechanism causes the gain — ablate your components; a monolithic

"our system vs. theirs" plot proves selection, not mechanism.

1. The gain survives scale — curves along data size, cluster size, and

concurrency, not one configuration chosen after the fact.

1. The cost is known — measure where your design loses and say so.

Workload realism ladder

| Rung | Example | Reviewer credit |

|---|---|---|

| Micro-benchmark | single operator, synthetic keys | Mechanism insight only |

| Standard benchmark | TPC-style, YCSB, JCC-H-class | Comparable, but "benchmark-only" is a known flag |

| Benchmark + skew/drift | standard suite with realistic distributions | Solid |

| Trace or production-derived | replayed real workload | Strongest, disclose provenance |

Climb as high as the data you can legally use allows; state the rung honestly.

Baseline fairness protocol

The people who built your baselines review here. For every competitor:

• Latest stable version (or a justified pin), same hardware, same data.
• Tuning effort comparable to what you gave your own system — document the

knobs tried in both cases.

• If a competitor is excluded, one honest sentence why (license, no support

for the workload) beats silence.

• Include the strong-but-inconvenient baseline: the hand-tuned config, the

single-node engine that wins at small scale, the "just add an index"

answer. Reviewers propose these in their first pass; preempt them.

Reporting floor

• Throughput and latency, with tails (p95/p99) where users feel tails.
• Repetition counts and spread (see vldb-reproducibility for the variance

protocol); medians for skewed metrics.

• Axes from zero or clearly marked; log scales labeled; error bands defined

in the caption.

• Every number in the abstract traceable to a specific figure or table.

Ablation and sweep matrix

For each design component C1..Cn:
  full system  vs  full-minus-Ci        (mechanism attribution)
For each claimed robustness dimension D:
  sweep D across its realistic range    (skew, size, concurrency, selectivity)
Loss map:
  identify >=1 region where a baseline wins; measure it; discuss it

The loss map is deliberately mandatory. A paper whose system wins everywhere

in every plot triggers reviewer search behavior — they will find the losing

region themselves, without your framing.

Scale honesty

Run the largest experiments your infrastructure permits, then scope claims to

what was run. "Designed for larger deployments" is acceptable prose;

extrapolated curves presented as measurements are not. If cloud credits capped

the study, say so — builders on the panel have lived that constraint.

Output format

[Burden coverage] problem-exists / mechanism / scale / cost — evidence per burden
[Workload rung] <ladder position, justification>
[Baseline fairness] <competitor -> version, tuning parity, exclusions>
[Reporting floor] tails, reps, spread — gaps
[Loss map] present (region, magnitude) / missing (risk)
[Next decisive run] <the one experiment that most changes acceptance odds>