webconf-experiments

Web Conference Experiments

The empirical bar at this venue is not "more datasets" — it is **evidence whose

scale, freshness, and realism match the claim**. A method claiming web-scale

efficiency must show the scaling curve; a measurement claiming platform-general

behavior must show more than one platform; a system claiming deployability must

show cost under realistic load. Audit the claim-evidence match before adding

anything.

Claim-to-evidence contract

| Claim type | Minimum honest evidence | Habitual shortfall |

|---|---|---|

| "Outperforms" (quality) | Tuned recent baselines, repeated runs, variance, significance | Untuned baselines from 3-year-old code |

| "Scales" (efficiency) | Cost curves across ≥2 orders of magnitude, hardware stated | One big-dataset wall-clock number |

| "Generalizes" (external validity) | ≥2 platforms/domains or a stated single-platform scope | Silent single-platform universality |

| "Measures" (phenomenon) | Construct definition, sampling frame, bot/spam handling, error bars | Convenience crawl treated as census |

| "Deploys" (system) | Load, latency percentiles, failure behavior; A/B where claimed | Demo-grade throughput on toy traffic |

Dataset selection with provenance

• Prefer datasets with documented collection: source, crawl window, sampling

rule, known biases. An undocumented Kaggle mirror of a platform dump is a

provenance liability reviewers now flag.

• Freshness is evidential here: user behavior and platform mechanics drift, so

a 2015 interaction dataset supports historical claims, not claims about current

platforms. Either add a recent corpus or scope the claim in time.

• Report per-dataset statistics (nodes/edges/users/items/time span) in a table, and

state why this set of datasets spans the claim — heterogeneity in domain, size,

and density is the point, not the count.

• For data you collected: consent/ToS posture and aggregation level belong beside

the dataset description (body text, per webconf-writing-style).

Leakage: the venue's most-caught methodological bug

Web data is temporal, popularity-skewed, and duplicated across the crawl. Three

specific leaks to audit:

1. Temporal: random splits on interactions or evolving graphs train on the

future. Use a time-based split rule and state it ("train < T ≤ test").

1. Popularity: negative sampling that mirrors test popularity inflates ranking

metrics; report at least one popularity-debiased or full-ranking metric.

1. Duplication/contamination: near-duplicate pages across splits, and — for

LLM-era pipelines — benchmark text present in a foundation model's pretraining.

State the dedup rule; where an external LLM is a component, discuss

contamination explicitly rather than hoping no one asks.

Baselines and ablations

Baselines come from the sibling circuit's current cycle (WWW, WSDM, SIGIR,

KDD, CIKM within ~2 years), tuned with the same budget as your method — state the

search space for both, since asymmetric tuning is the most common quiet unfairness.

Ablate the claimed mechanism, not everything: if the paper's story is "the

crawl-time signal is what helps," the ablation table must contain exactly the row

that removes it.

Experiment matrix skeleton (fill per claim, not per dataset):
  claim C1 "quality"  : datasets D1-D3 x {ours, B1..B4} x 5 seeds -> mean±sd, test
  claim C2 "scales"   : D_synthetic sized 10^6..10^9 edges x {ours, B1} -> cost curve
  claim C3 "mechanism": ours minus {signal S, module M} on D1-D3    -> delta table
  claim C4 "external" : platform P2 replication of headline result  -> one table
Every table row answers a named claim; rows answering nothing get cut.

Statistics that survive review

• Repeated runs with different seeds wherever training is stochastic; report

mean ± sd and the number of runs.

• Significance tests matched to the design (paired across datasets/queries), with

effect sizes — at web scale everything is "significant," so the delta's

magnitude and cost carry the argument.

• Percentiles, not means, for latency and exposure-type quantities; web

distributions are heavy-tailed and means mislead.

When offline evidence is not enough

Claims about user response (satisfaction, engagement shifts, behavior change)

need a user study or an online experiment, with ethics posture stated; claims about

production viability need load realism. If neither is obtainable, weaken the

claim to what offline evidence supports — "improves offline ranking quality" is

publishable; "improves user satisfaction" without users is a rebuttal-week wound

that cannot be healed in seven days.

Predictable objections and their cheap preemptions

Four objections recur across this venue's tracks; each has a preemption that

costs a sentence or a table row, not a new experiment campaign:

• "Gains may come from capacity, not the proposed component" — include one

parameter-matched ablation row and state parameter counts in the caption.

• "Datasets are all from one platform family" — either add the second-family

corpus or write the scope sentence in the conclusion; silence converts a

limitation into a discovered flaw.

• "The baseline numbers differ from the original paper" — declare the

re-implementation and the protocol difference (split, metric variant,

preprocessing) in a footnote at first mention; unexplained deltas read as

either sloppiness or gaming.

• "Efficiency claims lack a cost axis" — every quality table involving a

method whose selling point includes scale gets a companion column (time,

memory, or queries), because at web scale a 0.5-point win at 10x cost is a

negative result.

The meta-rule: reviewers here are drawn from a circuit that reviews the same

methods at WSDM, SIGIR, and KDD in the same year; they have seen this quarter's

common failure modes several times already. The paper that names its own

weaknesses first is the one whose rebuttal week is quiet.

Audit checklist

• [ ] Every headline claim mapped to a table/figure that can carry it.
• [ ] Split rules stated; temporal/popularity/duplication leaks addressed.
• [ ] Baselines recent, tuned symmetrically, search spaces disclosed.
• [ ] Variance, tests, and effect sizes present; percentiles for tails.
• [ ] Scope sentences where evidence is single-platform or offline-only.

Output format

[Claim-evidence map] C1..Cn -> table/figure or GAP
[Leakage audit] temporal / popularity / duplication: clean or findings
[Baseline fairness] recency, tuning symmetry: pass/fail
[Scale realism] does evidence scale match claim scale? <notes>
[Required additions] <ranked by review risk, with cost estimate>