fse-artifact-evaluation

FSE Artifact Evaluation

Use this for the artifact track. FSE follows the ACM Artifact Review and Badging scheme, and

the artifact evaluation is a separate, post-acceptance process with its own deadline. Two

things to internalize: badges are earned by evaluators actually using your package, and the review

artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact

(de-anonymized, permanently archived).

The ACM badges (verify the current set and names)

| Badge | What it certifies | What earns it |

|---|---|---|

| Artifacts Available | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage) |

| Artifacts Evaluated - Functional | The artifact runs and does what the paper says | A clean-machine install, a demo, and documented expected outputs |

| Artifacts Evaluated - Reusable | Others can build on it | The Functional bar plus careful docs, structure, and licensing |

| Results Reproduced | An evaluator reproduced the paper's key results | A turnkey path from the artifact to the headline numbers |

Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced

require the evaluator's own run to succeed, so the failure mode is always "did not run on their

machine," never "the idea was weak."

What SIGSOFT evaluators open first

| Claim type | First thing inspected | Common failure caught |

|---|---|---|

| A tool/technique | The README and one install/run command | Undocumented dependencies; only-works-on-authors'-laptop |

| An empirical study | The scripts that turn data into the paper's tables | Numbers in the PDF that no script reproduces |

| A mined dataset | The extraction scripts + the extracted data | Query shipped, data missing; provenance unpinned |

| An LLM-based result | Cached prompts/outputs + model IDs | Requires live API keys; not reproducible |

Assume an evaluator gives your package a bounded time budget on a clean machine. Design for the

first ten minutes to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
              "install these 40 things by hand"
[README]      one-screen orientation: what it is, how to install, how to run the demo, how to
              reproduce each claim, expected runtime and outputs
[Mapping]     an explicit table: paper claim -> script -> expected result
[Data]        the extracted dataset itself (or documented access), not just the query
[Provenance]  repo SHAs, extraction dates, model IDs/dates, seeds
[License]     an OSI-approved license so the artifact can be badged Reusable
[Archive]     deposit in a DOI-issuing repository for the Available badge

Anonymized review artifact vs. badge artifact

• At submission: the artifact is anonymized for the paper's reviewers — no owner strings,

cluster paths, lab names, or identity-revealing links, and no live repository that discloses

authors.

• After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing

archive; this is the version the artifact evaluators badge and the camera-ready cites.

Worked vignette: packaging a detection tool + study

A paper contributes a defect-detection tool and an empirical evaluation. To target Reusable and

Reproduced: ship a Docker image with the tool pre-built; a run_demo.sh that detects on a small

bundled project in under a minute; a reproduce/ directory whose scripts regenerate each table

from logged results; a claim-to-script mapping table in the README; the extracted evaluation

dataset with pinned SHAs; and an MIT/Apache license. State honestly which results are turnkey and

which need the full (slow) dataset run.

Calibration

• The artifact deadline is after acceptance and independent of the camera-ready; do not conflate

them.

• Badge names, the exact set offered, and whether evaluation is single- or double-anonymous vary by

cycle — confirm on the current artifact-track call.

Output format

[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <tool/data/scripts/provenance/license>
[Ten-minute test] does install + demo succeed on a clean machine? yes/no
[Claim mapping] <claim -> script -> expected result present? yes/no>
[Fixes before upload] <ordered list>