usenixsec-reproducibility

USENIX Security Reproducibility

Since the '26 cycle, USENIX Security has made openness a structural requirement:

every submission carries an Open Science appendix stating where the artifacts

behind the paper live, and acceptance is later conditioned on that availability

verifying (Phase-1 AE). This skill covers the appendix itself and the underlying

engineering that makes the statement true. Policy text is per-cycle — reread the

current CFP section before relying on wording.

The Open Science appendix is a contract, not a caption

Reviewers read it during evaluation; the AEC enforces it after acceptance. It

should answer, concretely:

• What exists: code, datasets, configurations, analysis notebooks, hardware

designs — enumerated, not gestured at.

• Where: resolvable locations (anonymized mirrors at submission; permanent,

ideally DOI-backed archives in the final paper).

• What is withheld and why: the policy accepts justified omission, and security

work legitimately generates a lot of it.

\section*{Open Science}
All scanner source code, the analysis pipeline, and the aggregated measurement
tables (Sections 4--6) are available at \url{<anonymized-archive>} and will be
deposited with a DOI upon publication. The raw scan captures contain
per-endpoint identifying data for still-unpatched hosts and are withheld;
Appendix~B documents the aggregation procedure so the tables can be
regenerated from an independent scan. The 214 disclosure emails are withheld
as human-subjects correspondence; templates are included.

Withholding decisions that hold up

| Material | Default posture | Acceptable justification pattern |

|---|---|---|

| Attack/PoC code | Share, targeted at a bundled testbed | Weaponizable against unpatched population → share after patch window, state the date |

| Vulnerability details | Share post-disclosure | Coordinated disclosure incomplete → embargo with timeline in the Ethical Considerations appendix |

| Scan/measurement raw data | Share aggregated | Raw data identifies vulnerable hosts or users → aggregate + publish the aggregation code |

| User-study transcripts | Withhold; share instruments | IRB/consent scope → release codebooks, surveys, and quantitative summaries |

| Malware corpora | Hashes + provenance | Redistribution illegal or dangerous → document acquisition path others can follow |

| Vendor-provided datasets | As NDA allows | Contract limits → say so and provide synthetic or public-subset substitutes |

Two disciplines make these defensible: decide them at experiment time (retrofit

justifications read as excuses), and always ship the procedure even when the

data stays closed — an independent team with their own vantage point should be

able to re-derive your tables.

Determinism engineering for security experiments

Security results are unusually time- and environment-coupled. Pin what you can and

timestamp what you cannot:

• The internet moves. Record scan dates, target-list snapshots, and blocklist

versions; a reproduction in 2027 measures a different network than 2026 did.

• Fuzzing and probabilistic attacks: fix seeds where the harness allows, and

report distributions over repeated campaigns (count, median, IQR) rather than a

best run; state the compute budget per campaign.

• ML-based pipelines (classifiers, LLM-assisted analysis): pin model versions

and weights; a hosted-API dependency is a reproducibility hole — snapshot outputs

and say so.

• Version-sensitive exploits: record exact kernel/firmware/browser builds, and

keep the vulnerable version archived in the artifact since upstream will patch.

# Capture the environment fingerprint alongside every experiment run
{ date -u +%FT%TZ; uname -srmo; git rev-parse HEAD;
  sha256sum targets.txt config.yaml; pip freeze | sort; } > runs/$(date +%s).env

Paper-side reporting that reviewers check

• Every headline number traces to a script in the artifact; the claims table in

the artifact README (see usenixsec-artifact-evaluation) is the index.

• Ranges and repetition counts for stochastic results; single-run numbers flagged

as such.

• Dataset construction fully specified: collection window, filters, dedup rules,

ground-truth labeling procedure and inter-rater agreement where humans labeled.

• Negative-result honesty: configurations where the attack fails or the defense

costs too much belong in the paper, and reviewers at this venue notice absence.

Reverify each cycle

• Exact Open Science appendix requirements and page allowance in the current CFP.
• Whether availability verification remains acceptance-conditional ('27 待核实).
• Current AEC guidance on acceptable archives and anonymized hosting.

Output format

[Inventory] artifacts enumerated: code / data / configs / instruments
[Open items] each withheld artifact + justification + procedure substitute
[Determinism] seeds, environment fingerprints, time-coupling documented
[Trace] headline claims ↔ regeneration scripts: n/m covered
[Appendix draft] Open Science text ready: yes / gaps listed