hpca-artifact-evaluation

HPCA Artifact Evaluation

Use this to prepare an HPCA artifact for the badge round. HPCA's artifact evaluation

is voluntary, post-acceptance, and run on a separate HotCRP

(hpca<year>ae.hotcrp.com) under IEEE reproducibility badging — not the ACM

Artifact Review and Badging policy the ACM-touched siblings use. Package for the IEEE

pipeline and for an evaluator who has never seen your infrastructure.

Scope the reproducibility tiers

Not every claim can be reproduced cheaply. Decide, per headline result, what an

evaluator can realistically achieve and say so:

• Full reproduction — the evaluator regenerates the headline numbers from source.

The strongest badge target; reserve it for results whose pipeline you can shrink to

fit an evaluator's budget.

• Regeneration from logs — the evaluator rebuilds figures from provided raw

output when a full run is too long or needs licensed workloads.

• Inspection — the evaluator confirms the code implements the mechanism when

hardware or license constraints block execution.

State each result's tier in the artifact README so evaluators are not surprised.

Package for a cold-start evaluator

The evaluator has no license to your workloads, no copy of your machine, and a fixed

time budget. Design for that:

| Obstacle | What to ship |

|---|---|

| No workload license | Recipe + checksums, plus one free workload that runs the full pipeline |

| No access to your machine | Container/script that builds on a clean host; pinned dependencies |

| Long simulations | Reduced-input variants of the headline runs, with measured runtimes |

| Silicon-only results | Captured machine-state logs + an inspection path when re-running is impossible |

| Unclear entry point | A top-level run.sh that reproduces one headline figure end to end |

Budget the evaluator's wall-clock

An artifact that "reproduces everything" in three days of compute will not be fully

reproduced. Provide reduced-input variants of the headline experiments with **stated

per-step runtimes**, so an evaluator can plan and finish inside the AE window. Put the

expensive full runs behind a clearly labeled optional path.

Mirror ↔ release

The anonymized artifact mirror used during review becomes the de-anonymized public

release at the badge round. De-anonymize deliberately: restore author names, add the

real repository, and check that nothing was left blinded that should now be open —

and nothing personal was left in that should not.

Packaging pass

1. Per-result tier declared (full / regeneration / inspection)
2. Cold-start build works on a clean host (container or script, pinned deps)
3. One free workload exercises the full pipeline end to end
4. Reduced-input variants with measured per-step runtimes provided
5. Licensed workloads reduced to recipe + checksums
6. run.sh reproduces one headline figure without hand-holding
7. De-anonymized for release; README states tiers and expected runtimes

Output format

[AE readiness] Ready / Needs work / Not ready
[Tiers] results with a declared tier / total
[Cold start] clean-host build passes? (Y/N)
[Budget] reduced variants + runtimes stated? (Y/N)
[Badge target] full-reproduction results / total
[Top gaps] <ordered>

Reopen the current AE page for the badge set, the calendar, and the submission

mechanics — the IEEE badge names and AE timeline are per-edition.