ndss-artifact-evaluation

NDSS Artifact Evaluation

NDSS added artifact evaluation to the symposium in recent editions (running on the

community secartifacts platform, verified for 2024-2026), and it is **opt-in after

conditional acceptance** — a separate track, not a submission requirement. Papers that pass

carry badges on the first page and add a 2-page artifact appendix (Call for Artifacts,

checked 2026-07-08). Decide early whether you are engineering for it, because the packaging

choices are cheap during the project and expensive afterward.

The three badges

| Badge | Claim to the committee | What you must ship |

| --- | --- | --- |

| Available | The artifact is publicly, permanently retrievable | Deposit in an archival repository with a DOI (Zenodo, figshare); a bare GitHub URL is not "available" |

| Functional | It does what the paper says, is usable and documented | Build/run instructions that work on a clean machine; the components matching the paper's claims |

| Reproduced | The paper's results can be regenerated by others | One documented path per headline result, with expected output and tolerances |

Badges are independent — you can earn Available and Functional without Reproduced when the

world (live targets, proprietary data) genuinely prevents re-execution. Claim only what you

can defend; an evaluator who cannot reproduce a "Reproduced" claim writes that in the

report.

From claims ledger to artifact

The claims.yml ledger built in ndss-reproducibility is the artifact's spine. Convert it

into an evaluator-facing package:

artifact/
  README.md          # what this is, badges sought, hardware/time needs, safety notes
  INSTALL.md         # clean-machine setup; pinned deps / container image
  claims/            # one entry per headline result:
    fig4/  run.sh    # → expected_output/, runtime + resource note
    tab2/  run.sh
  data/              # snapshots or synthetic substitutes (never raw victim traffic)
  src/               # the code, with the dangerous parts gated (below)
  ETHICS.md          # disclosure status, gating rationale, usage boundaries

Give evaluators the shortest true path to each result. A "kick-the-tires" phase means the

first five minutes decide the evaluator's mood — make the smoke test one command.

Packaging dangerous material responsibly

NDSS artifacts routinely include working exploits, attack tooling, or malware-adjacent

code. This is allowed; unbounded release is not.

• Gate the weaponized parts. Ship the analysis and measurement pipeline openly; put

live exploit modules behind documented access conditions or a defanged demonstration

mode consistent with your Ethics Considerations section.

• Align with the disclosure clock. If a vulnerability is under a coordinated-disclosure

embargo, the artifact's release timing must respect it — state the date the full module

becomes available.

• Sandbox by default. Anything that touches a network should target a supplied local

testbed, not the Internet, out of the box. Document the blast radius.

• No live victim data. Replace with prefix-preserved or synthetic traces carrying the

statistics the result needs (see ndss-reproducibility).

When re-execution needs your lab

Hardware-bound results (SDRs, specific switches, IoT devices) cannot be reproduced by a

committee at their desks. Bridge the gap: provide recorded traces plus a replay harness so

the analysis is reproducible even when the capture is not, and say plainly in the

README which results are trace-replayable and which need the physical setup. This commonly

yields Functional + Reproduced-on-recorded-data rather than a failed full Reproduced.

The 2-page artifact appendix

Written after evaluation for the camera-ready. It states the badges earned, the artifact's

scope, access instructions (the DOI), hardware/software requirements, and the mapping from

paper claims to artifact components. Keep it factual and navigational — it is a map for

future readers of the open-access paper, not a second results section.

Timing

The opt-in comes shortly after (conditional) acceptance and runs on its own schedule

(2027 dates 待核实 — confirm on the Call for Artifacts). It overlaps the camera-ready

window (Jan 6, 2027), so a team that starts packaging at acceptance, not at the AE

deadline, avoids a two-front crunch.

Output format

[Badges targeted] Available / Functional / Reproduced — with feasibility per badge
[Package status] README / INSTALL / per-claim runners / data substitutes — done or gaps
[Danger handling] gated modules, sandbox default, disclosure-aligned release date
[Hardware gap] results needing physical setup + trace-replay bridge
[Appendix] 2-page artifact appendix drafted? claim→component map complete?