ppopp-artifact-evaluation

PPoPP Artifact Evaluation

Use this for the artifact track. PPoPP runs a post-acceptance artifact evaluation with its own

deadline (PPoPP 2026: artifact submission 17 November 2025, notification 5 January 2026),

separate from the camera-ready and shared in culture with co-located CGO. Two things to

internalize: PPoPP has a distinctive badge policy, and the hard part of a parallel artifact is

that the results must reproduce on someone else's hardware, where core counts and topology

differ from yours.

PPoPP's badge policy (verify the current set each cycle)

| Badge | Colour | What it certifies | How it is awarded |

|---|---|---|---|

| Artifacts Available | green | The artifact is publicly, permanently retrievable | By the publisher, from a deposited-artifact link — no formal audit |

| Artifacts Evaluated — Functional | lighter red | The artifact runs, is documented, consistent, complete, exercisable | AE committee runs it |

| Artifacts Evaluated — Reusable | darker red | Functional plus quality that exceeds minimal functionality; carefully documented and structured for reuse | AE committee runs it |

| Results Reproduced | darker blue | The paper's main results were obtained by the evaluators (within tolerance) | AE committee re-runs and matches |

Two PPoPP-specific facts to get right:

• The committee awards one of Functional / Reusable (Reusable is the higher bar) plus

possibly Results Reproduced.

• PPoPP does not award the lighter-blue "Results Replicated" badge — do not target or claim it.

Available is not evaluated; it is publisher-granted from a link, so it is the cheapest badge

to secure and worth doing regardless.

What makes a parallel artifact hard

An evaluator does not have your machine. A speedup that depended on your 96-core dual-socket node

will not reproduce on a reviewer's 16-core laptop, and "Results Reproduced" requires agreement

within a stated tolerance. Design for portability of the conclusion, not the absolute numbers:

[Hardware doc]  state the exact machine you used (CPU/GPU model, sockets, cores, NUMA, memory,
                interconnect) and the minimum config on which the trend still holds
[Scaled runs]   provide a small/fast configuration whose *shape* (linear region, saturation)
                matches the paper, so an evaluator can reproduce the claim on modest hardware
[Tolerance]     state explicitly what "reproduced" means for your numbers (e.g. within 10%, or
                "monotone speedup to the machine's core count")
[Determinism]   fix seeds; pin threads; document warm-up; make runs repeatable
[GPU]           document driver/CUDA/ROCm versions and the GPU class needed; provide a CPU
                fallback path where possible

What evaluators open first

| Claim type | First thing inspected | Common failure caught |

|---|---|---|

| A concurrent structure | README + a one-command throughput run | Only builds on the authors' toolchain; hardcoded core count |

| A runtime/scheduler | The script that reproduces a scaling figure | Numbers in the PDF that no script regenerates |

| A GPU technique | Build + a small kernel run | Requires a specific GPU with no fallback; missing driver versions |

| A parallel algorithm | Build + a scaled input run | Data missing; topology assumptions undocumented |

Assume a bounded time budget on a clean machine; make the first ten minutes succeed.

Packaging plan

[Container]   a Dockerfile or pinned environment (spack/conda/lockfile); avoid manual 40-step builds
[README]      one-screen orientation: what it is, how to build, how to run the demo, how to
              reproduce each figure, expected runtime, and the hardware each run needs
[Mapping]     an explicit table: paper claim/figure -> script -> expected result (+ tolerance)
[Scaling kit] scripts that sweep thread/core count and emit the paper's curve on the eval machine
[Provenance]  exact hardware used, compiler/flags, driver versions, seeds, pinning policy
[License]     an OSI-approved license so the artifact can be badged Reusable
[Archive]     deposit in a DOI-issuing repository (Zenodo/figshare/Software Heritage) for Available

Anonymized review artifact vs. badge artifact

• At paper submission: the artifact (if attached for reviewers) is anonymized — no owner

strings, cluster/account paths, lab names, or identifying links.

• At the AE track (post-acceptance): the de-anonymized, licensed, DOI-archived version is what

evaluators badge and the camera-ready cites.

Calibration

• The AE deadline is after acceptance and independent of the camera-ready; do not conflate them.
• Confirm each cycle whether AE is run jointly with CGO, the exact badge set and colours, and

whether Available is auto-granted by the publisher (as verified) or handled by the committee.

Output format

[Target badges] Available (publisher) / Functional or Reusable / Results Reproduced  (NOT Replicated)
[Artifact role] anonymized review artifact / public badge artifact
[Portability] scaled config whose trend matches the paper on modest hardware? tolerance stated?
[Ten-minute test] build + demo succeeds on a clean machine? yes/no
[Claim mapping] each figure -> script -> expected result (+ tolerance) present? yes/no
[Fixes before upload] <ordered list>