ndss-reproducibility

NDSS Reproducibility

Network-security results decay: targets patch, providers change behavior, botnets die, and

the population you measured in August is not the population of next March. Reproducibility

at NDSS therefore means two different promises, and conflating them is the classic mistake:

1. Reconstruction — anyone can re-derive your numbers from what you recorded.
2. Re-execution — anyone can re-run your pipeline against the world and get their

numbers, understanding why they differ from yours.

Promise (1) unconditionally. Promise (2) only where the world cooperates.

Freezing the measured world

| Volatile thing | What to freeze at experiment time |

| --- | --- |

| Scanned population | Input list + source + retrieval date; per-target response snapshots |

| Target software | Exact versions, build hashes, config files; patch level on the test date |

| Network path | Vantage descriptions, traceroute-level context where relevant, ASN of probes |

| Testbed | Topology file, firmware images (or their hashes), kernel/NIC settings |

| Toolchain | Container image or lockfile for every analysis script; seeds for anything sampled |

| Third-party feeds | Copies (or hashes + dates) of blocklists, zone files, certificate logs used |

The snapshot habit converts "trust us, it was exploitable in July 2026" into an auditable

record — which is also what the rebuttal will need when a reviewer asks whether the fix

released in September invalidates the paper.

Traces are radioactive

Packet captures, flow logs, DNS transcripts, and crawl outputs embed user identities,

internal hostnames, and your institution's fingerprints — an anonymity leak against

double-blind review and a privacy harm on release. Rules that hold up:

• Scrub at capture time, not release time; a raw pcap on a laptop is a liability, not an

asset.

• Prefix-preserving IP anonymization where analysis needs structure; deletion where it

does not. Document which was applied — reviewers of measurement work will check.

• Replace real victim traffic with regenerated synthetic equivalents whenever the result

tolerates it, and say so.

• Grep every release candidate for institutional domains, usernames, and cloud-account

identifiers before it leaves the repo.

The claims ledger

Maintain, from the first experiment, a machine-checkable mapping between paper claims and

regeneration paths:

# claims.yml — one entry per number/figure the paper depends on
fig4_takeover_success:
  claim: "takeover succeeds against configs A-C"
  inputs: [snapshots/2026-07-scan/, configs/targets.yml]
  command: "make fig4"          # runs inside container ndss-artifact:v3
  runtime: "35 min, no network" # replays recorded traces
  status: verified 2026-07-05
prevalence_table2:
  claim: "condition present in N/M sampled domains"
  inputs: [snapshots/2026-06-population.csv.gz]
  command: "make table2"
  status: verified 2026-07-02

If a claim has no entry, either add the pipeline or soften the claim. The ledger later

becomes the artifact-evaluation appendix almost verbatim (see ndss-artifact-evaluation).

Honest availability statements

Some NDSS work legitimately cannot release everything: unpatched-exploit details under

embargo, traces that cannot be de-identified, vendor NDAs. The venue's culture accepts

limits that are named and mitigated, not gestured at:

• State precisely what is withheld, why, and until when (e.g., "exploit module released

after the coordinated-disclosure window closes").

• Ship the largest safe substitute: redacted configs, synthetic traces with matched

statistics, the analysis code even when the data stays private.

• Never write "code available upon request" as the entire plan — at this venue it reads as

"not available".

Because NDSS proceedings are open access (Internet Society model, no paywall), your

artifact link and the paper will be read together by the whole community; the availability

statement is a public commitment, not review-stage decoration.

Cheap habits that pay at rebuttal time

• Date-stamp every scan directory; the timeline question always comes.
• Re-run the full pipeline from clean checkout monthly during the project — drift found

early is drift fixed cheaply.

• Keep one LIMITS.md recording every known non-determinism (timing-sensitive exploits,

load-dependent measurements) and how the paper's statistics absorb it.

Output format

[Reconstruction status] claims with regeneration paths: N/M; missing listed
[World snapshot] frozen / partial / absent per volatile category
[Trace hygiene] scrub method, leak grep result, synthetic substitutions
[Availability statement] withheld items + reason + mitigation + release date
[Drift check] last clean-checkout rerun date and result