icdm-experiments

ICDM Experiments

Design the evaluation an ICDM reviewer will trust: a defined mining task, baselines tuned as

carefully as your method, ablations that isolate the mechanism, a measured scale story, and a

discovery-validity argument. ICDM's data-centric reviewers punish leaderboard-only wins and

un-checkable discovery claims, and the whole evaluation must fit inside the 10-page

all-inclusive cap.

Define the mining task before the metric

• State the task operationally: inputs, outputs, and what a correct answer is. "Anomaly

detection" is a genre; "rank edges by anomalousness in a one-pass stream, evaluated against

injected ground truth" is a task.

• Fix the evaluation protocol — splits, negatives, thresholds, ranking cutoffs — before

running anything, and describe it precisely enough to reproduce inside the page cap.

The four evidence axes

| Axis | Question it answers | Typical evidence |

|---|---|---|

| Quality | Is the mining result good on the task? | Ranking/accuracy vs baselines with variance |

| Scale | Does the scale claim hold? | Latency/memory curves across data sizes |

| Mechanism | Is the named mechanism the reason? | Ablations toggling exactly that component |

| Validity | Is the finding real, not an artifact? | Controlled injections, known-truth checks |

A strong ICDM paper touches all four; missing "mechanism" or "validity" is the usual reason a

methodologically fine paper reads as thin.

Baselines and tuning symmetry

• Compare against current strong baselines, and tune them with the same budget you gave

your method; an under-tuned baseline is the fastest way to lose reviewer trust.

• Include the obvious simple baseline. If a cheap method nearly matches you, say so and argue

the regime where your mechanism pays off.

• Report every number with variance over seeded runs; a single-run table invites the "is this

noise?" review.

Ablations that isolate the mechanism

The mechanism-attached novelty of icdm-writing-style must be demonstrated, not asserted.

Mechanism: single-pass isolation sketch with m random partitions.
Ablation grid:
  - remove the sketch, keep full storage      -> isolates the streaming contribution
  - vary m (partition count)                  -> maps the accuracy/memory knob
  - swap the hash family                       -> tests sensitivity to the mechanism's core
  - replace isolation score with density score -> isolates the isolation principle
Each row answers "was THIS the reason it worked?"

Test the scale claim, do not assert it

• If you claim scalability, plot behavior across at least an order of magnitude of data size,

and report the cost model (linear, sub-linear memory, amortized constant update).

• Separate wall-clock from asymptotic claims; hardware-dependent speedups need the hardware

stated and, ideally, an operation count that is not hardware-dependent.

Discovery validity: the ICDM instinct

• Where truth is unknown, build a setting where it is: injected anomalies, planted patterns,

synthetic graphs with known structure — so you can show the method recovers known signal.

• Guard against leakage: temporal tasks need time-respecting splits; graph tasks need to avoid

train/test edge overlap. State the guard explicitly.

• Do not overclaim when differences are within variance; an honest "matches at lower cost" is

stronger here than a fragile "outperforms."

Vignette: an ablation that saved the claim

A team reports strong stream-anomaly numbers but reviewers cannot tell whether the sketch or

the underlying isolation criterion did the work. Adding two ablation rows — full-storage

isolation (isolating the streaming contribution) and a density-score swap (isolating the

isolation principle) — showed the sketch preserved batch quality while the isolation principle

drove detection. The claim survived because the mechanism was shown, not stated, and both

rows fit in the appendix inside the 10-page cap.

Output format

[Task] <operational task definition>
[Axes covered] quality / scale / mechanism / validity - list gaps
[Baselines] strong + tuned symmetrically: yes / no
[Ablation] isolates the named mechanism: yes / no
[Validity] known-truth or leakage-guarded: yes / no
[Top evidence gap] <single most important missing experiment>