mlsys-experiments

MLSys Experiments

Use this before the evaluation is frozen. An MLSys evaluation exists to answer one

compound question: *does the named mechanism deliver the claimed system-level payoff on

workloads that matter, at acceptable cost, for understood reasons?* Each clause needs its

own evidence, and this venue — which published the MLPerf Training methodology in its own

proceedings — holds measurement to benchmark-committee standards.

Workload selection: the make-or-break choice

"Unrepresentative workload" is the modal fatal objection at this venue. Defend the

choice explicitly:

• Anchor on recognizable workloads: MLPerf-family tasks, widely used open models, or

published request traces. Bespoke workloads need a characterization section showing

their statistics (request rates, sequence lengths, arrival burstiness, model mix).

• Cover the axis your mechanism exploits: a scheduler for bursty load must be tested on

bursty and uniform load; the uniform case bounds your claim, it does not weaken it.

• Include at least one workload where you expect little or no gain, and show it. A

non-win row is the cheapest credibility purchase available.

• State workload licensing/provenance — traces you cannot release constrain your future

artifact badges (see mlsys-artifact-evaluation).

Baseline discipline

• Compare against the current strongest systems, version-pinned, in their recommended

configurations — and tune them with the same budget you gave your own system.

Document both budgets; asymmetric tuning is treated as invalidating.

• Include the "do nothing" configuration where meaningful (e.g., default framework

behavior); reviewers use it to sanity-check absolute numbers.

• If a relevant baseline cannot run in your setting, demonstrate the incompatibility

rather than asserting it.

The reporting quartet

Never report throughput alone. Each headline claim carries four coordinates:

| Coordinate | Report as | Classic gaming pattern it prevents |

|---|---|---|

| Throughput/goodput | Under a stated latency constraint | Batch-size inflation that destroys tails |

| Latency | p50 and p99 (p99.9 for serving), distributions not means | Mean-only tables hiding stragglers |

| Memory | Peak device + host, at the measured configuration | Wins that only fit on 80GB parts, unstated |

| Cost/efficiency | $ or joules per unit work, with price/meter basis | "Faster" systems that are 4x more expensive |

Quality (accuracy/perplexity or task metric) rides along whenever the technique could

plausibly change model output — quantization, sparsity, scheduling with dropping,

approximation of any kind. "No quality change" is a claim requiring a measurement.

Ablation = attribution

Systems papers stack techniques; reviews ask which component pays.

• One ablation row per named mechanism, cumulative or leave-one-out — pick one scheme

and say which.

• Ablate the insight, not only the code: if the claim is "lookahead scheduling wins,"

include the no-lookahead scheduler, not just the whole-system-off row.

• Sensitivity sweeps over the parameters a deployer must set (window sizes, thresholds);

a mechanism needing per-workload hand-tuning must say so.

Scaling and sensitivity evidence

# experiment-matrix.yaml — one row per plotted point, generated, never hand-edited
sweep:
  models:    [llama-class-8b, llama-class-70b, moe-16e]
  hardware:  [1xA100, 8xA100-nvlink]        # state interconnect explicitly
  load:      [uniform-qps, bursty-trace-A]
  precision: [bf16, int4]
  trials: 5
  warmup_steps: 20
  metrics: [goodput@p99<250ms, p50, p99, peak_mem_gb, gpu_hours, quality]
  seeds: [1, 2, 3, 4, 5]

• Plot trends, not points: gain versus model size, versus load, versus GPU count. A

single-configuration win invites the "will this survive scale?" objection you cannot

answer in the four-day response window.

• Log everything; regenerate every figure from logs by script (measurement mechanics

and variance rules live in mlsys-reproducibility).

Statistical floor for systems numbers

Systems papers rarely need hypothesis-testing machinery, but they always need variance

honesty; the floor this venue's reviewers apply:

• Five or more trials for any number in a comparison table; report spread (stdev or

IQR) and say which in the caption.

• Overlapping spreads mean the difference is not a finding — either run more trials or

report the tie honestly; a "win" inside noise is worse than a reported tie because

one reviewer rerun destroys it.

• For latency distributions, trials are not enough: within-run percentiles need enough

requests that p99 is estimated from hundreds of samples, not three.

• Seed variation matters only where model quality enters the claim; systems noise

(placement, thermal state, co-tenants) dominates latency variance and is controlled

by measurement design, not seeds — the split is detailed in mlsys-reproducibility.

Evaluation-design review before running

Run this table exercise before burning GPU-hours: for each claim in the abstract, write

the figure that will support it, the workload it needs, and the strongest objection a

systems reviewer could raise — then check the matrix covers that objection. Claims

without a planned figure get cut from the abstract, not padded later.

Common reject patterns to design against

• Microbenchmark gains that never appear end-to-end; always include the end-to-end run.
• Gains measured only at one scale/hardware and claimed generally.
• Quality silently degraded by the optimization, discovered by a reviewer from your own

appendix table.

• Cost omitted where the honest accounting would erase the win.

Output format

[Evaluation readiness] strong / adequate / weak
[Workload defense] <named anchors + characterization + non-win case present?>
[Baseline symmetry] <versions pinned, tuning budgets equal + documented?>
[Quartet coverage] <throughput/latency-tails/memory/cost per headline claim>
[Attribution] <mechanism -> ablation row>
[Decision-critical missing run] <one experiment>