asplos-experiments

ASPLOS Experiments

An ASPLOS evaluation answers to three communities at once: architects who will audit

the modeling, OS people who will audit the workload realism, and PL people who will

audit what the software layer actually does. The section's core discipline is

matching each claim to an instrument whose error model can carry it — and saying

what that error model is.

The instrument ladder

| Instrument | What it can prove | What it cannot | Must be reported |

|---|---|---|---|

| Real silicon | End-to-end effects, OS interactions, true tails | Designs needing hardware that doesn't exist | CPU/stepping, kernel + config, microcode, BIOS knobs (SMT/turbo/prefetchers), memory topology |

| FPGA prototype | Feasibility, cycle behavior of new logic at the prototype's clock | Absolute performance of an ASIC-class part | Board, clock, resource utilization, what was scaled down and why |

| Cycle-level simulator (e.g. gem5-class) | Relative effects of microarchitectural change under stated configs | Anything outside modeled fidelity — I/O, OS noise, firmware behavior are commonly stylized | Simulator + exact version/commit, config files, warm-up and region-selection method, validation against a real machine where possible |

| Analytical/energy models (McPAT-class, first-order area) | Trend-level energy/area comparisons | Absolute mW or mm² as truth | Model version, technology node assumptions, and the claim written as trend not absolute |

The cardinal sin is a claim-instrument mismatch: absolute latency claims from an

unvalidated simulator, or OS-interaction claims from a user-space harness. Rapid

and full reviewers both hunt for it.

Cycle-accuracy caveats are content, not apology

When simulation carries a claim, the paper must state: which structures are modeled

in detail vs stylized; how simulation regions were chosen (full runs, checkpoints,

sampled regions à la SimPoint-style methodology); how long the warm-up was; and —

strongest of all — a validation experiment showing the simulator tracks a real

machine on a measurable subset. A one-paragraph validation against silicon buys

credibility that no amount of extra benchmarks can.

Workloads and baselines that survive three audiences

• Draw workloads from suites the communities recognize (SPEC-class CPU suites,

parallel suites, cloud/graph/serving workloads appropriate to the claim) and

include at least one full application or kernel-integrated scenario — accelerator

papers evaluated only on extracted kernels routinely get the "where is the rest of

the system" review.

• The baseline is the **strongest deployed alternative configured by someone who

wants it to win**: current kernel policy with its tunables set properly, the

vendor library, the state-of-the-art accelerator at an honest technology

normalization.

• Technology normalization must be explicit when comparing across nodes or clocks:

state the scaling assumptions rather than silently converting.

Attribution: ablate the mechanism you credit

Every "X improves Y because of mechanism M" needs a run with M removed, weakened,

or transplanted onto the baseline. In cross-layer papers this means ablating **each

side of the boundary separately** — hardware hints without the new policy, policy

without the hints — because the venue's whole premise is that the coupling matters;

prove the coupling, not just the sum.

The claim-instrument matrix

Freeze this before writing; it becomes the evaluation section's skeleton and the

rebuttal's ammunition:

claim                          instrument        workloads          baseline(+config)      metric + spread          where
end-to-end speedup             real 2-socket+CXL  SPEC17 + graph(5)  Linux 6.9 tiering,     runtime, gmean, 10 runs, §6.2
                                                                     tuned per docs         95% CI
coupling is necessary          same               subset(6)          each-half ablation     delta vs full design     §6.4
generality across latency      gem5 (pinned cfg)  subset(6)          same policy            trend, sim-validated     §6.5
overhead where design idles    real hardware      non-tiered set     stock kernel           <=2% regression bound    §6.6
energy trend                   McPAT-class model  subset             baseline design        trend only, node stated  §6.7

Report dispersion for anything measured on real hardware (runs, variance source,

CI); report sensitivity for anything simulated (which config parameters move the

result). Include the workload where the design loses and explain the boundary — a

measured regression with a mechanism story is evidence of understanding, and its

absence is conspicuous to reviewers who build systems themselves.

Measurement noise on real hardware is a design input

Silicon experiments carry noise sources that simulators hide, and the paper's

run protocol must name its countermeasures: pin frequency governors or report

the governor used; control or randomize NUMA placement; interleave A/B runs

rather than batching (thermal and cache state drift over a session); and

distinguish warm-start from cold-start numbers explicitly. When an effect is

within the machine's observed run-to-run variance, the honest sentence is that

the experiment cannot distinguish the designs — reviewers respect the sentence

and pounce on its absence.

Energy, power, and area claims

• On silicon, name the meter: RAPL-class counters, wall-power instrumentation,

or board-level telemetry — each has known blind spots worth one caveat clause.

• Model-derived energy or area numbers (McPAT-class, synthesis estimates)

support comparisons under stated assumptions, not datasheet-grade values;

write them as ratios with the technology node and model version attached.

• FPGA utilization (LUTs, BRAM, DSPs) is evidence of feasibility at the

prototype's scale — extrapolating it to ASIC area needs an explicit argument,

or the claim should stay at feasibility.

Evaluation-methodology papers

Note that "experimental methodologies" is itself on the 2027 topics list: if

the most defensible contribution turns out to be the measurement approach — a

validation harness, a workload characterization, a simulation-sampling method —

consider promoting it from a subsection to the paper, with asplos-topic-selection

re-run on the promoted claim.

Sweeps and knees

Cross-layer designs live or die on regime boundaries, so at least one sweep per

load-bearing parameter (device latency, core count, working-set size, offered

load) should run past the knee — the point where the benefit saturates or

inverts. A curve truncated before its knee is read by systems reviewers as a

curve hiding its knee. State where the knee is and why it sits there; the

mechanism story at the boundary is often the most-cited sentence in the paper.

Output format

[Matrix] every claim has instrument+baseline+location: Y/N (orphans listed)
[Instrument audit] any claim exceeding its instrument's error model? list
[Simulator hygiene] version/config/regions/warm-up stated · validated vs silicon?
[Baseline strength] strongest deployed alternative, tuned: Y/N per claim
[Attribution] per-layer ablations present: Y/N
[Adverse results] losing workload + boundary explanation in paper: Y/N