isca-artifact-evaluation

ISCA Artifact Evaluation

The verified 2026 arrangement: authors of accepted papers could voluntarily

submit artifacts, assessed under the ACM Artifact Review and Badging policy,

with earned badges printed on the paper and attached as ACM Digital Library

metadata — and with no effect on the acceptance decision itself. The conference

framed AE as a recent, deliberate import into the architecture community

(following ASPLOS from 2020 and MICRO from 2021). Voluntary does not mean

optional in practice: badges are becoming the community's default trust signal,

and an architecture paper without them increasingly gets asked why.

Decide scope first: which claims does the artifact defend?

Architecture artifacts fail most often from over-promising. Before packaging,

split the paper's results three ways and say so in the artifact documentation:

| Tier | Contents | Evaluator experience |

|---|---|---|

| Reproduce | Headline figure(s) + key ablation, runnable on evaluator-accessible machines within hours | Runs it, matches within stated tolerance |

| Regenerate-with-resources | Full sweeps needing cluster-days; scripts and configs complete, cost stated honestly | Inspects, spot-runs a slice |

| Inspect-only | Anything needing hardware evaluators won't have (FPGA boards, silicon, internal traces) | Reads code + recorded raw outputs; verifies the pipeline from raw to figure |

Declaring tier boundaries up front converts "couldn't run everything" from a

failure into the documented plan. Never gate the Reproduce tier on proprietary

inputs — substitute an open workload subset and show it preserves the trend.

Simulator-heavy artifacts: the specific frictions

• Wall-clock honesty. Cycle-level simulation of full suites is

CPU-months; evaluators have days. Ship reduced-input or single-region variants

of the headline experiment, with the full-run manifests alongside. State

expected runtimes per step, measured, not guessed.

• Build fragility. Simulator trees with local patches break on new

toolchains. Ship a container image with everything pre-built, and the

from-source path for evaluators who distrust images.

• Tolerance statement. Define "match": bit-identical stats for

deterministic runs; a stated percentage band where sampling or threading

varies. An evaluator staring at 33.7% vs the paper's 34% needs the band in

writing.

• Third-party licenses. SPEC-class suites cannot be redistributed; ship

build recipes plus checksums, and include one freely redistributable workload

so the pipeline is exercisable end to end regardless.

The package an evaluator can drive blind

artifact/
  README.md          # claims-to-experiments map; runtimes; tolerance bands
  INSTALL.md         # container route + source route, both tested cold
  run_all.sh         # Reproduce tier end to end, one command
  experiments/
    f07-headline/    # regen.sh + manifest.ini per figure (from
    f09-ablation/    #   isca-reproducibility — same manifests, reused)
  expected/          # our raw outputs + final CSVs, for diffing
  workloads/         # free subset included; licensed suite: recipe + checksums
  LICENSE

The README's first section is a claims table: paper claim → figure → command

→ expected output → tolerance. Evaluators grade against exactly this table;

making them reverse-engineer the mapping from the paper costs goodwill and,

frequently, a badge.

# Cold-start rehearsal — run on a machine that has never seen the project
docker load < artifact-image.tar.gz
docker run --rm -it artifact:v1 bash -lc './run_all.sh --tier reproduce'
diff <(csvcut -c workload,ipc results/f07.csv) \
     <(csvcut -c workload,ipc expected/f07.csv)   # within README tolerance?

Have a group member outside the project do this rehearsal before submission;

every question they ask is a missing README paragraph.

The badge set under ACM's policy

The ACM Review and Badging framework distinguishes availability badging (the

artifact is permanently retrievable from an archival location) from functional

and results-oriented badging (the artifact works as documented; the paper's

results can be obtained with it). Practical mapping:

• Availability is cheap and unconditional: deposit the evaluated snapshot

with a DOI-issuing archive (a lab GitHub alone is not archival). Do this even

if you pursue nothing else.

• Functionality rides on documentation and the cold-start test.
• Results reproduction rides on the Reproduce tier's design: modest

runtime, clear tolerance, deterministic where possible.

Which badges ISCA 2027 requests, the AE calendar, and the committee's platform

offerings (cloud credits, provided machines) are per-edition decisions — 待核实

against the 2027 AE page when it exists.

Working with evaluators

Architecture AE is conventionally collaborative and iterative: evaluators hit a

wall, authors fix and resubmit within the AE window. Budget author-days for this

in the post-acceptance calendar (isca-workflow), respond within a day while

the evaluator's context is warm, and version every fix (artifact:v1.1,

changelog in the README) so the final badge attaches to an identifiable object.

Timing and division of labor

AE happens in the same weeks as camera-ready preparation and talk writing.

Assign the artifact to someone who ran the original experiments — packaging is

recall-heavy — and start from the submission-time snapshot that

isca-reproducibility froze in November, not from the current dev tree, which

has drifted.

Submission gate

• [ ] Claims table complete: every badge-relevant figure mapped to a command
• [ ] Cold-start rehearsal passed by a non-author on a clean machine
• [ ] Runtimes measured and stated per step; tolerance bands written
• [ ] Container + source routes both build
• [ ] Licensed workloads handled by recipe; free subset runs end to end
• [ ] Archival deposit made; DOI in the camera-ready's artifact statement

Venue facts verified 2026-07-08 (../../resources/official-source-map.md);

badge definitions follow ACM's published badging policy — check the version the

2027 AE chairs cite.