ieeesp-experiments

IEEE S&P Experiments

Use this to build or audit the evidence an S&P paper stands on. The venue's

reviewers are professional adversaries; an evaluation that would satisfy a

systems or ML PC often leaves an Oakland reviewer's central question — "does

this survive a real, adaptive attacker?" — unanswered.

Match the evaluation to the contribution type

| Contribution | Evaluation that closes the loop | Fatal gap |

|---|---|---|

| Attack | End-to-end demonstration against a realistic, current target | Toy target; unrealistic preconditions |

| Defense | Adaptive adversary who knows the design; cost/overhead | Only non-adaptive or prior attacks |

| Measurement | Representative sampling + validation + ethics | Convenience sample presented as population |

| System | Security property demonstrated and performance | Property asserted, not tested |

| SoK | Systematic re-analysis under one framework | Cherry-picked coverage |

The adaptive-adversary rule dominates defense papers

A defense evaluated only against existing or non-adaptive attacks is the most

common S&P defense rejection. The standard:

• Define the adaptive adversary explicitly: knows the mechanism, the

parameters, and the deployment.

• Show your defense against attacks designed to break it, not just

yesterday's attacks it happens to stop.

• Report the cost of adaptation for the attacker and the overhead for the

defender — both are part of the security claim.

• If a class of adaptive attack is out of scope, say so in the threat model

and own the boundary; do not leave it for a reviewer to discover.

Measurement validity is an evidence question and an ethics question

For measurement papers, the sampling story and the ethics story are the same

paragraph in reviewers' minds:

• State the population, the frame, and the sampling method; quantify coverage

and bias.

• Validate a subsample by an independent method where possible.
• Active measurement (scanning, probing) must respect opt-out norms, rate

limits, and the ethics record (ieeesp-review-process) — an experiment

that harms the systems it measures is a reject regardless of results.

• Human-subjects components need IRB determination before running, not a

post-hoc note.

Statistics for attacks and fuzzing

Security evidence is often probabilistic and gets held to a real bar:

Attack-success reporting:
  n trials (state n) · success rate ± dispersion · target set described
  → "worked" without n is an anecdote, not a result

Fuzzing / bug-finding comparison (the field's known pitfalls):
  - equal budgets (CPU-time, not wall-clock)
  - ≥ 5–10 campaigns per configuration; report variance
  - identical seed corpora across compared tools
  - a ground-truth or triage method for "unique" bugs
  → a single-run bug count comparison is not evidence of superiority

Timing / side-channel:
  noise floor stated · machine quiescence (isolated cores, freq pinning)
  · distinguisher's statistical test named

Baselines and ablations Oakland reviewers ask for

• The strongest prior attack/defense, at its best configuration, not a

weakened reimplementation.

• An ablation isolating the component you claim is responsible for the

security gain.

• A cost baseline: what does the attacker/defender spend, and is it

realistic at the claimed scale?

• Negative results where they bound the claim (attack fails against target

class Y — state it; it strengthens the scoped claim).

Ethics as an experimental design constraint, not an afterthought

Some experiments are simply not runnable as first imagined:

• Testing an exploit against live third-party systems without authorization

is out; build a representative testbed instead.

• Collecting user data beyond what the IRB and the ethics record cover is

out.

• Disclosure timing constrains when certain measurements can be published

— design the timeline so the evidence and the fix do not collide

(ieeesp-reproducibility).

Audit worksheet

For each experiment:
  claim it supports | contribution type | adaptive adversary evaluated? |
  strongest baseline used? | n trials + dispersion | ethics clearance |
  realistic target? | rerunnable? (→ ieeesp-reproducibility)
Flag any row with: non-adaptive-only defense · anecdotal success rate ·
  weakened baseline · unmet ethics precondition

Output format

[Contribution type] attack / defense / measurement / system / SoK
[Loop closed?] <the demonstration/eval that proves the claim> ✓/✗
[Adaptive adversary] evaluated ✓/✗/n-a — scope stated?
[Baselines] strongest prior used ✓/✗ · ablation ✓/✗ · cost baseline ✓/✗
[Statistics] trials+dispersion ✓/✗ · fuzzing pitfalls avoided ✓/✗/n-a
[Ethics preconditions] IRB ✓/✗/n-a · authorization ✓/✗ · disclosure timing ok ✓/✗