wsdm-reproducibility

WSDM Reproducibility

Make a web-search/data-mining paper re-derivable. WSDM has no reproducibility

checklist to fill (none surfaced for current editions; 待核实 each cycle) - which

raises the bar rather than lowering it, because reviewers apply the norm

without giving you a form to hide behind. The venue-specific twist: WSDM

evidence usually comes from behavioral data (queries, clicks, follows,

purchases), and behavioral data has failure modes that generic ML

reproducibility advice never mentions.

Provenance: behavioral data decays

A log dataset is a measurement of a platform at a moment - the platform's

ranker, UI, and user base are all baked into it. Reproducibility therefore

starts with recording what generated the data:

• Collection window, platform surface (web vs app, market/locale), and any

known ranker or UI changes inside the window.

• The logging policy: what produced the exposures users could click on. A

click log is a logged-policy artifact; results on it are conditional on

that policy (this is the entire lesson of the position-bias and unbiased

learning-to-rank literature born at this venue).

• Filtering steps with counts at every stage: bots removed, sessions

segmented, minimum-activity thresholds. Two labs "using the same dataset"

routinely diverge purely on preprocessing counts.

Temporal discipline

Random splits on interaction data leak the future into training. Default to

time-based splits and document them to the day:

# split-manifest.yaml - ship with the artifact, cite in the paper
dataset: platform-logs-v3
train:  {start: 2025-01-06, end: 2025-05-31}
valid:  {start: 2025-06-01, end: 2025-06-14}
test:   {start: 2025-06-15, end: 2025-06-28}
user_handling: users may span splits (temporal, not user-disjoint)
item_handling: cold items in test retained; reported separately
leakage_checks:
  - no feature computed over any window overlapping valid/test
  - global statistics (IDF, popularity) frozen at train end
notes: one ranker deployment change on 2025-04-12 inside train window

State whether users are shared across splits (temporal split) or disjoint

(generalization-to-new-users split) - the two answer different questions and

mixing them is a classic silent irreproducibility source in recommendation

papers.

Bias assumptions are part of the method

If the paper estimates relevance or preference from clicks, its results depend

on an exposure/position-bias model. Reproducibility means naming it:

| What you assume | What must be reported |

|---|---|

| Position bias (examination model) | The propensity model, how it was estimated, on what data |

| No exposure bias (rare, say so) | Why the setting justifies it |

| Popularity/selection bias corrected | The correction estimator and its hyperparameters |

| Offline metrics proxy online value | The known gap, plus any online evidence |

An unstated bias model makes the numbers unreproducible even with the code,

because a re-implementer will pick a different default.

Runs, seeds, and variance

• Report the number of runs and the seed policy for every learned component;

ranking metrics on sparse test sets are noisy, and single-run nDCG deltas of

under a point are routinely within seed variance.

• Give variance (std or CI) for headline comparisons; where a system-scale

experiment genuinely cannot be repeated, say "single run" in the table note

rather than letting the reader assume otherwise.

• Statistical tests over query/user-level paired differences beat aggregate

deltas; state the unit of analysis (query, session, user) - it changes the

test.

The honesty ladder for unrerunnable results

Industrial WSDM papers often include numbers nobody outside can regenerate

(online A/B tests, full-traffic logs). Use graded language that matches the

evidence tier, and put the tier in the paper:

1. Rerunnable: public data + released code; anyone can regenerate tables.
2. Rebuildable: released code + documented proprietary pipeline; an insider

could regenerate, an outsider can audit the logic.

1. Attested: online/production results reported with measurement protocol

(traffic share, duration, metric definitions, guardrails) but not

regenerable. Attested numbers support deployment claims, not method-ranking

claims - do not let an A/B win stand in for a missing offline comparison.

Privacy is a reproducibility constraint, not an excuse

WSDM requires an ethical-considerations section; user-data handling belongs in

it. De-identification, aggregation thresholds, and consent/ToS basis for the

data should be stated - and any released sample must survive a re-identification

sniff test (rare queries and long-tail items are quasi-identifiers). "We cannot

release anything" is acceptable only alongside rung 2-3 evidence above and a

public-benchmark mirror where feasible (see wsdm-artifact-evaluation).

Pre-submission reproducibility sweep

Run once when experiments freeze, once on the final PDF:

[ ] Data provenance paragraph: window, surface, logging policy, filters+counts
[ ] Split manifest shipped and cited; user-sharing across splits stated
[ ] Bias/exposure model named, with estimation procedure and data
[ ] Seeds and run counts per learned component; variance on headline deltas
[ ] Unit of analysis named for every statistical test
[ ] Each result family labeled: rerunnable / rebuildable / attested
[ ] Attested results carry protocol: traffic %, duration, metric definitions
[ ] Released sample re-identification check done (rare queries, tail items)
[ ] Ethics section covers user-data basis and mitigations, specifically
[ ] Repo numbers regenerate paper tables (spot-check two tables end-to-end)

Items that fail with no time to fix become limitation sentences, not

silence - at a no-rebuttal venue, a disclosed gap is survivable and a

discovered one usually is not.

Output format

[Provenance] window / surface / logging policy / filter counts: recorded? 
[Splits] temporal manifest present; user-sharing stated: yes / no
[Bias model] named + estimation reported: yes / no / not applicable
[Variance] runs, seeds, CI/test + unit of analysis per headline table
[Tier] rerunnable / rebuildable / attested per result family
[Privacy] ethics-section coverage of user data: adequate / gaps listed