sosp-artifact-evaluation

SOSP Artifact Evaluation

Use this once a SOSP paper is accepted and the artifact-evaluation (AE) invitation

arrives. SOSP's AE is optional, runs after acceptance (so it never threatens the

paper decision), and is operated through the shared systems-community infrastructure at

sysartifacts.github.io with its own HotCRP instance (for 2025:

sysartifacts.github.io/sosp2025/, sosp25ae.hotcrp.com; verified 2026-07-08). The

CFP describes the clock plainly: a few days after notification to register the

artifact, then a few more days to polish it before evaluation begins.

The process is cooperative — plan to be online

SOSP AE is explicitly interactive: evaluators report problems, authors fix them during

the window, and reasonable response time is an expectation, not a courtesy. The

strongest predictor of a failed evaluation is not code quality; it is an author team

that treated submission as fire-and-forget during the same weeks the camera-ready and

conference logistics were competing for attention.

• Name an AE owner who is not the shepherd-thread owner.
• Watch the AE HotCRP for evaluator comments daily during the window.
• Fixes during evaluation are legitimate; ship them as tagged revisions so evaluators

can tell what changed.

Badge targeting

SOSP badges follow the ACM artifact-review taxonomy. Choose targets deliberately — each

badge has its own checklist, and chasing all of them with a hardware-bound artifact is

a common own-goal.

| Badge | What evaluators check | Typical blocker |

|---|---|---|

| Artifacts Available | Artifact deposited in a public archival repository with a DOI/stable link | Only a GitHub URL — repos are not archival; cut an archived release |

| Artifacts Evaluated — Functional | Documented, consistent, complete, exercisable; evidence of verification | "Works on our cluster" scripts with hardcoded paths and hostnames |

| Artifacts Evaluated — Reusable | Functional bar plus docs and structure that let others build on it | No way to run on inputs other than the paper's |

Artifacts earning all available badges compete for the Distinguished Artifact

award — worth targeting when the artifact is genuinely the paper's contribution.

Packaging for an OS-adjacent artifact

SOSP artifacts frequently need kernels, custom modules, specific NICs, or multi-node

clusters — things an evaluator cannot conjure. Tier the claims:

1. Anyone-can-run tier: build + unit-level functionality in a container or VM

image, minutes not hours. This alone can carry Functional.

1. Provided-hardware tier: reserved access to your cluster or a public testbed

(CloudLab-style) with exact node profiles, for headline performance results.

1. Documented-only tier: results needing unobtainable hardware or proprietary

traces; state clearly that these are documented, not re-runnable, and do not claim

badges over them.

artifact/
  README.md            # claims map: paper claim -> experiment -> expected output & tolerance
  ARTIFACT-CHECKLIST   # target badges, per-badge self-assessment
  kick-the-tires/      # <30 min smoke path evaluators run first
  build/               # container/VM definitions, pinned toolchain and kernel version
  experiments/         # one dir per paper figure/table, run.sh + plot.sh each
  hardware/            # node specs, testbed profile, reservation instructions
  LICENSE

The claims map matters more than any script: evaluators reproduce claims, so state

for each figure what "success" means numerically (for example, "throughput within 10%

of Fig. 7's trend; absolute numbers vary with hardware").

Kick-the-tires discipline

Most SOSP-style AE processes open with a smoke-test phase. Whatever the current year

calls it, build for it: a top-level path that a stranger on a fresh machine can

complete in under half an hour, exercising the build and one small end-to-end run.

Rehearse it literally on a fresh VM with a lab member who did not write the code, and

time it. Every minute of evaluator confusion in the first session is paid back as

skepticism for the rest of the window.

What AE cannot do

• It cannot change the acceptance decision, and evaluators are anonymous to authors —

keep interactions inside the AE HotCRP.

• It does not certify performance portability; badges say the artifact was exercised,

not that your speedups hold on other silicon.

• Badge names, phases, and dates are re-announced per cycle by the sysartifacts chairs;

the 2026-cycle SOSP AE page details were not yet pinned when this pack was written

(待核实) — read the year's call before promising badges to co-authors.

Output format

[AE decision] participate? badges targeted
[Clock] registration due / polish window / evaluation window
[Tier map] claim -> anyone-can-run / provided-hardware / documented-only
[Kick-the-tires] rehearsed on fresh machine? duration?
[Owner] AE-window responder (distinct from shepherd owner)
[Risk] <single most likely evaluator blocker>