recsys-reproducibility

RecSys Reproducibility

Reproducibility is unusually load-bearing at RecSys for two reasons. First, the venue runs a

dedicated Reproducibility Track for papers that repeat, refute, or re-scope prior results and

for datasets and frameworks that enable future reproduction. Second, the field's own literature

documented how often reported recommender gains fail to survive a fair re-evaluation, so reviewers

of regular papers read reproducibility signals as a proxy for whether the gains are real.

Why recommender results drift: the usual suspects

| Drift source | Typical symptom | Pin it by |

|---|---|---|

| Dataset version / filtering | "MovieLens" numbers differ across papers | Exact version id, k-core filter, and checksums |

| Split protocol | Session results inflated | Temporal or leave-one-last split, not random; document it |

| Baseline tuning asymmetry | Neural method "beats" everything | Equal search budget per system, grid and selected config logged |

| Sampled vs full-ranking metrics | Recall/nDCG off by a lot | Rank over the full item set, or state sampling and cutoff |

| Seeds and nondeterminism | ±0.005 nDCG run to run | Multiple seeds; report mean ± sd, not the best run |

| Metric implementation | nDCG differs at the 3rd decimal | One canonical scorer with cutoff and tie-handling recorded |

| Off-policy propensities | Counterfactual estimate irreproducible | Release logged propensities and the estimator variant (IPS/SNIPS/DR) |

A reproducibility-strong RecSys paper closes each row with an artifact, not a promise: the config

file is the documentation.

Minimum reporting block for any empirical RecSys paper

Put this in the paper (it fits in ~0.3 page and pre-empts three review objections):

• Datasets with versions and filtering; train/validation/test usage; the split protocol.
• Tuning protocol: search space, budget, selection metric, validation split — **symmetric across

systems, baselines included**.

• Metrics: which ones, the cutoff, and whether ranking is over the full item catalog or a sampled

candidate set.

• Seeds: how many, and whether tables show mean, sd, and a significance test on close results.
• For any deployment or off-policy claim: the estimator, its assumption, and the logged

propensities.

The Reproducibility Track specifically

A reproduction paper is first-class work here, not a lesser contribution.

[reproduction scope]  which prior papers, which claims, which datasets
[matched setup]       identical splits, metrics, and — critically — equal tuning budget
[divergence report]   where results reproduce, where they do not, and the diagnosed cause
[new context]         a new domain, dataset, or baseline that tests generality
[artifact]            a runnable pipeline others can extend

Honest divergence is the contribution: "we could not reproduce claim X under matched tuning, and

here is why" is publishable when the diagnosis is rigorous.

Vignette: a session-model reproduction

Consider reproducing three published session recommenders. Its reproducibility spine: the exact

dataset versions and k-core filter, a temporal split shared across all systems, an equal tuning

budget so no method is advantaged by search, full-ranking metrics at a fixed cutoff, five seeds

with mean ± sd, and one honest paragraph on the one result that did not reproduce and the

preprocessing difference that explains it.

Degrees of reproducibility

• Turnkey: one command regenerates each ranking table from logged seeds.
• Scripted: scripts exist but need documented manual steps or an external dataset download.
• Descriptive: prose detailed enough that a competent reader could rebuild the pipeline.

For RecSys, the offline pipeline should be turnkey because reviewers actually re-run it; a

production A/B result may stay descriptive with the deviation documented. State the achieved level

honestly rather than overpromising a turnkey build that fails on a clean machine.

Output format

[Claim inventory] <claim -> evidence location>
[Reproducibility status] strong / partial / weak
[Drift risks] <dataset version / split / baseline tuning / sampled metrics / seeds / propensities>
[Paper fixes] <must appear in main PDF>
[Repository fixes] <anonymous-repo additions>