asplos-artifact-evaluation

ASPLOS Artifact Evaluation

Artifact evaluation at ASPLOS is a post-acceptance, opt-in, collaborative

process: an independent committee works with authors to validate the paper's key

results, and successful artifacts carry badges on the published paper (AE pages,

checked 2026-07-08). It is also a tradition the venue itself highlights — systems

readers increasingly treat an unbadged systems paper as a weaker citation. Treat AE

as part of the publication, budgeted like a small sixth section.

The three badges and what each actually demands

| Badge | 2027 criterion (paraphrased from the AE pages) | Practical bar |

|---|---|---|

| Available | Artifact placed on a publicly accessible archival repository | A DOI-issuing archive (institutional or Zenodo-class); a GitHub URL alone is not archival |

| Functional | Evaluators can prepare and run the artifact; they document the steps they followed | Clean-machine install + a smoke experiment that completes in minutes, not hours |

| Reproducible | Evaluators validate the paper's key results | Per-claim run scripts whose output maps visibly onto specific figures/tables |

Evaluators assign scores per requested badge and record what they could and could

not reproduce — so the artifact's job is to make their success path short and their

failure modes diagnosable.

The Artifact Appendix is the contract

ASPLOS 2027 expects an Artifact Appendix built from the provided ae.tex template

(or equivalent sections) covering: all **software, hardware, and dataset

dependencies; the key results to be reproduced; and how to prepare, run,

and validate** the experiments. Write it as if the evaluator is competent, busy,

and using different hardware than yours:

• Dependencies include the awkward ones: kernel versions, privileged access, BIOS

settings, board models, expander firmware — everything from the state ledger in

asplos-reproducibility.

• "Key results" means a selected subset: pick the claims that define the paper,

not all 40 bars of every figure. Ambition here creates failure reports.

• Validation must be decidable: state the expected output and the tolerance within

which the claim holds ("ordering preserved; absolute times ±15%").

Package layout that evaluators can navigate blind

artifact/
  README.md            # 10-minute quick start + full map
  APPENDIX.pdf         # the ae.tex appendix as submitted
  env/                 # container/VM recipe OR exact install script
  hardware.md          # tiered requirements + what to do without them
  run/
    smoke.sh           # minutes-scale end-to-end sanity check
    claim1_fig6.sh     # one script per key result, named for its figure
    claim2_tab3.sh
  expected/            # reference outputs + tolerance statement per claim
  data/ or data.md     # datasets, or archival pointers + checksums

Hardware-dependent claims: give evaluators a path

The recurring ASPLOS AE failure is a paper whose headline number needs silicon the

committee lacks. Acceptable mitigations, in descending order of strength:

1. Provide remote access to the platform for the evaluation window (with an

anonymity-safe access route if the process requires it).

1. Ship the simulator-backed subset as the reproducible core, and mark the

silicon results as demonstrably-run (logs + analysis pipeline included).

1. Offer a scaled-down proxy (smaller FPGA, reduced workload) with an explicit

argument for why the proxy's behavior transfers.

Say which mitigation applies in the appendix, per claim — evaluators score

against what you requested, so calibrated requests outperform hopeful ones.

Collaboration protocol

• Expect rounds: evaluators report blockers, authors fix and respond. Reserve

maintainer time in the weeks after camera-ready (exact 2027 AE dates: 待核实 —

confirm at notification).

• Fix-forward, do not re-argue: an evaluator's confusion is a defect in the README.
• Keep the artifact frozen at a tagged version during evaluation; hotfixes go on a

branch the evaluators are told about.

Common evaluator blockers, pre-empted

Field experience across systems AE committees converges on a short list of

first-hour failures, all preventable:

• Undeclared credentials or licenses — a workload, simulator model, or

dataset that needs a registration the evaluator lacks; declare it in the

appendix and provide an alternative path.

• Hidden network assumptions — builds that fetch from internal mirrors or

rate-limited hosts; vendor the dependencies or provide the container image.

• Root-only steps without warning — kernel-module or BIOS-adjacent steps

must be flagged up front so the evaluator can pick a sacrificial machine.

• Hour-scale first feedback — if the smoke test takes an evening, the first

blocker report costs a full round trip; minutes-scale smoke tests keep the

collaboration inside the calendar.

• Output the evaluator must interpret — raw logs with no comparator; every

claim script should end by printing PASS/FAIL against the tolerance.

Anonymity boundary

AE runs after acceptance, so evaluator-facing materials need not be anonymous —

but any artifact pointer placed in the submission itself (an appendix teaser,

a footnoted repository) falls under the double-blind rules and must be

anonymized end to end: repository owner, commit author strings, container

registry paths, and dataset hosting all leak identity. The clean pattern is to

keep the submission's artifact story descriptive ("we will submit an artifact

covering claims 1-3") and materialize the links only in the Artifact Appendix

after notification.

Dry-run protocol

Before submission to the AEC, have a colleague who did not build the artifact

execute the README on a clean machine, timing each stage and noting every

question they had to ask. Their questions are defects; fix the README, not the

colleague. Two such passes typically halve the evaluation rounds.

Output format

[Badges requested] available / functional / reproducible — with rationale
[Appendix status] dependencies / key results / prepare-run-validate all drafted: Y/N
[Smoke test] clean-environment runtime: N min · passes: Y/N
[Claim scripts] one per key result, mapped to figure/table: list
[Hardware path] per silicon-dependent claim: access / sim-subset / proxy
[Archive] DOI-issuing repository chosen + deposit dry-run done: Y/N