socc-artifact-evaluation

SoCC Artifact Evaluation

Use this for artifact preparation. SoCC — as an ACM venue — follows the **ACM Artifact Review and

Badging scheme where an edition offers evaluation. First, verify whether the current SoCC

edition runs a dedicated artifact-evaluation track, and which badges it offers** — unlike some

sibling systems flagships that run a standing AE process, SoCC's artifact track and badge set are

decided per edition and are 待核实 as of 2026-07-09. The advice below applies once the edition's

call confirms evaluation.

Two things to internalize: badges are earned by evaluators actually reproducing your cloud

results, and the review artifact (anonymized, for the paper's reviewers) is not the same deliverable

as the badge artifact (de-anonymized, permanently archived).

The ACM badges (verify the current set and names)

| Badge | What it certifies | What earns it for a cloud paper |

|---|---|---|

| Artifacts Available | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage) |

| Artifacts Evaluated - Functional | The artifact runs and does what the paper says | A documented testbed setup, a workload replay, and expected outputs |

| Artifacts Evaluated - Reusable | Others can build on it | Functional plus careful docs, structure, licensing, and a portable harness |

| Results Reproduced | An evaluator reproduced the paper's key results | A turnkey path from the artifact to the headline throughput/tail/cost numbers |

Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced

require the evaluator's own run to succeed, so the failure mode is always "did not run on their

testbed," never "the idea was weak."

What a cloud-systems evaluator opens first

| Claim type | First thing inspected | Common failure caught |

|---|---|---|

| A cloud system/mechanism | The README and one setup+run command | Undocumented cluster assumptions; only-runs-on-authors'-testbed |

| A measurement/trace study | The scripts that turn the trace into the paper's figures | Numbers in the PDF no script reproduces; trace missing |

| A scheduling/serverless result | The workload generator + the tail/cost measurement scripts | Only the mean reproduces; tail and cost cannot be regenerated |

| A large-scale deployment | A scaled-down but faithful reproduction path | Requires a proprietary cluster; no smaller-scale replay |

Assume an evaluator has a bounded time budget and cannot reserve your 200-node cluster. Provide

a scaled-down reproduction that still regenerates the shape of the tail and cost results, plus a

clear statement of what needs full scale.

Packaging plan

[Environment] ship a Dockerfile / pinned environment AND a testbed description (node counts,
              instance types, OS, kernel) so a run is reproducible
[Workloads]   the workload generator or the (anonymized, then released) trace, not just a pointer
[README]      one-screen orientation: what it is, how to set up, how to run a small demo, how to
              reproduce each figure, expected runtime and outputs
[Mapping]     an explicit table: paper claim -> script -> expected result (incl. tail and cost)
[Scaled path] a small-scale reproduction that runs without the full cluster
[Provenance]  commit SHAs, trace extraction dates, instance types, seeds, run counts
[License]     an OSI-approved license so the artifact can be badged Reusable
[Archive]     deposit in a DOI-issuing repository for the Available badge

Anonymized review artifact vs. badge artifact

• At submission: the artifact is anonymized for the paper's reviewers — no cluster names,

provider hints, owner strings, or identity-revealing trace provenance, and no live repository that

discloses authors.

• After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing

archive; this is the version any artifact evaluators badge and the camera-ready cites.

Worked vignette: packaging an autoscaler + trace study

A paper contributes a tail-aware autoscaler and a measurement of the cost-tail gap. To target

Reusable and Reproduced: ship a Docker image with the controller pre-built; a run_demo.sh that

replays a short trace slice on a few nodes in minutes and prints p99 and instance-seconds; a

reproduce/ directory whose scripts regenerate each figure from logged runs; a claim-to-script

mapping table in the README; the (released) trace-replay harness with pinned SHAs; and an

MIT/Apache license. State honestly which figures are turnkey at small scale and which need the full

testbed.

Calibration

• Confirm the track exists for this edition before planning; SoCC's AE track and badge set are

待核实 per cycle.

• Reproducing tail and cost, not just the mean, is the cloud-specific bar; design the package so

an evaluator can regenerate them.

• Badge names, the exact set offered, and whether evaluation is single- or double-anonymous vary by

edition — confirm on the current call.

Output format

[Track status] SoCC AE track confirmed for this edition? yes/no/待核实
[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <system/workloads/trace/scripts/testbed-desc/provenance/license>
[Small-scale test] does setup + demo reproduce tail+cost shape without the full cluster? yes/no
[Claim mapping] <claim -> script -> expected result (incl. tail/cost) present? yes/no>
[Fixes before upload] <ordered list>