ndss-experiments

NDSS Experiments

Evidence at NDSS answers one question: would this hold on the real network? Every design

choice in the evaluation — target selection, vantage, scale, baseline — either supports

that answer or advertises that you avoided it.

Evidence by contribution type

| You claim... | Reviewers require... | Weak substitute they will flag |

| --- | --- | --- |

| A new attack | End-to-end demonstration on real implementations, named versions, measured preconditions and cost | PoC against a config you weakened |

| A prevalent condition | Defined population, sampling method, validated vulnerability test, false-positive analysis | Extrapolation from anecdotes |

| A defense | Security and utility: adaptive-adversary evaluation + overhead on realistic workloads | Blocking only the published attack |

| A detection system | Base-rate honest metrics on traffic resembling deployment, drift discussion | Accuracy on one stale benchmark |

| A protocol change | Interop and incremental-deployment story, behavior under partial adoption | "Assume everyone upgrades" |

Choosing the experimental substrate

Match the substrate to the claim, and say why in the paper:

• Live Internet — the only substrate for prevalence claims. Requires the ethics

machinery (below) and a recorded snapshot: measurement results are time-indexed facts, so

log the observation window, vantage points, and population source.

• Testbed / physical devices — for exploit reliability and defense overhead, where you

need ground truth and repeatability. Pin firmware, kernel, and topology; a testbed nobody

can reconstruct is an anecdote with racks.

• Emulation / simulation — legitimate for scale sweeps and what-if topologies after

the mechanism is shown real elsewhere. A paper whose only evidence is simulated packets

is, at this venue, a proposal.

Mixed designs are the norm: demonstrate on real systems, generalize on the testbed, sweep

in emulation, and keep the chain of custody between the three explicit.

The adaptive-adversary requirement

For defenses, the reviewer's first move is to play attacker against your design. Do it

first, in print:

1. Restate the defense as the constraint it imposes on the adversary.
2. Enumerate strategies that respect capabilities from the threat model but target the

mechanism itself — evasion, mimicry, resource exhaustion, downgrade, oracle abuse.

1. Implement the strongest of these, not the most convenient, and report where the defense

bends. Cost asymmetry (defender cents vs. attacker weeks) is a legitimate result;

silence about adaptation is not.

Measurement hygiene

• One vantage point is a case study; state how many you used and what each can and cannot

see (anycast, geo-blocking, and CDN behavior all bite here).

• Validate the classifier or vulnerability test against labeled ground truth and report

both error directions — a scanner's false positives are the headline number's error bar.

• Repeat over time before claiming stability; the Internet's diurnal and patch-cycle

rhythms confound single-shot scans.

Ethics as an experimental parameter

Live experiments are bounded by harm, and the CFP backs this with an Ethics Review Board.

Design limits in from the start: rate caps and opt-out honoring for scans, no serving of

real user traffic during interception experiments, synthetic victims wherever the

demonstration allows, notification pipelines for anything you confirm exploitable, and IRB

engagement where humans are observed — while remembering NDSS treats IRB sign-off as

necessary context, not sufficient mitigation. Every one of these choices belongs in the

paper's method text, not only in the ethics section.

Reporting discipline

For every headline number, the text or appendix must pin:
  target set + versions + selection method
  vantage(s) and observation window
  trials, variance, and the aggregation rule
  hardware/software of the measurement or attack host
  configuration deltas from defaults (ideally: none, or justified)
  the command or procedure that regenerates the number (→ artifact)

Numbers that move between runs get intervals; comparisons against baselines get identical

conditions or an explanation of why identity was impossible.

Failure modes this venue punishes

• Evaluating on the environment you developed on, and nowhere else.
• Baselines run with defaults while your system got tuning.
• Success rates without denominators; "up to X" as a summary statistic.
• Prevalence claims from a population chosen because it was scannable, described as if it

were representative.

Output format

[Claim → evidence map] each headline claim + substrate + status
[Adaptive evaluation] strategies enumerated / strongest implemented / result
[Measurement validity] population, vantage, validation, error directions
[Ethics parameters] rate caps, synthetic victims, notification, IRB state
[Repro pinning] items from the reporting block still missing