sigir-reproducibility

SIGIR Reproducibility

Reproducibility is unusually load-bearing at SIGIR for two reasons. First, the venue

runs a dedicated Reproducibility track (its own track in 2026, split out of the

former combined Resource & Reproducibility track — budget and dates 待核实 on the

current page), so reproduction studies are publishable first-class work. Second, the

field's own literature documents how often reported IR gains fail to replicate under

matched tuning — reviewers of regular papers therefore read reproducibility signals

as a proxy for whether the gains are real.

Why IR results drift: the usual suspects

| Drift source | Typical symptom | Pin it by |

|---|---|---|

| Collection version | "MS MARCO" numbers off by points | Exact version/split ids, ir_datasets identifiers, checksums |

| Index-time analysis | BM25 baseline differs across papers | Scripted index build; record stemmer, stopwords, k1/b |

| Doc processing for neural models | Recall@k shifts | Max length, stride, title concatenation recorded as config |

| Truncated vs judged pools | Inflated dense-retrieval scores | State pooling; report judged@k alongside nDCG |

| Seeds and nondeterminism | ±0.005 nDCG run-to-run | Multiple seeds; report mean ± sd, not the best run |

| Eval tool discrepancies | MAP differs at 4th decimal | One canonical scorer (trec_eval/ir_measures) with flags recorded |

| Hyperparameter asymmetry | Baselines lose by under-tuning | Equal tuning budget, documented per system |

A reproducibility-strong SIGIR paper closes each row with an artifact, not a promise:

the config file is the documentation.

Minimum reporting block for any empirical SIGIR paper

Put this in the paper (it fits in ~0.3 page and pre-empts three review objections):

• Collections with versions; train/dev/test usage; any filtering.
• Index/build settings for every system, baselines included.
• Tuning protocol: search space, budget, selection metric, dev split — symmetric

across systems.

• Seeds: how many, and whether tables show mean, sd, and the significance test.
• Compute: hardware, wall-clock for index+train+retrieve, and query latency setup.
• Pointer to the repository with run files (see sigir-artifact-evaluation).

# config-as-artifact: one file per reported system, committed to the repo
system: ours-dense-v2
collection: msmarco-passage/dev/small        # ir_datasets id
index: {tokenizer: bert-base-uncased, max_len: 256, stride: 128}
train: {seeds: [13, 42, 71], batch: 64, lr: 2e-5, epochs: 3}
eval: {tool: ir_measures, metrics: [nDCG@10, RR@10, R@1000], qrels: official}
significance: {test: paired-t, correction: bonferroni, alpha: 0.05}

Writing a Reproducibility track paper

A reproduction study is not a re-run; it is an investigation. The track rewards:

1. Target selection with stakes — results the community builds on (a widely

cited ranker, a standard baseline configuration, a claimed efficiency win).

1. Faithful reimplementation first: reproduce with the original artifacts where

they exist; document every forced deviation and why.

1. Divergence analysis as the contribution: when numbers differ, isolate the

cause (collection version? tuning? eval tool?) with controlled toggles — the

drift table above is your experimental design.

1. Generalization probes: does the original conclusion survive new collections,

matched tuning, or current baselines? "Holds, but the margin halves under equal

tuning" is a publishable, community-serving finding.

1. Respectful register: the genre convention is scientific, not gotcha — criticize

configurations, not authors, and give original authors' artifacts credit where due.

Anti-patterns the track's reviewers flag: reproducing only the headline number while

skipping ablations; declaring "failure to reproduce" without exhausting configuration

space; and shipping a reproduction whose own pipeline is unpinned (the irony reject).

Decay planning: the three-year audience

A SIGIR paper's reproducibility has two audiences with different failure modes: the

reviewer this spring (needs the 10-minute runnable path) and the researcher in three

years (meets link rot, dataset takedowns, deprecated toolkit APIs, and vanished model

checkpoints). Plan for the second audience explicitly:

• Cite collections by stable identifiers (ir_datasets ids, TREC track names,

dataset DOIs), never by lab-server URLs.

• Freeze an archival copy of your own artifacts (Zenodo/institutional DOI) at

camera-ready; GitHub is a working mirror, not an archive.

• Record toolkit versions in the paper text, not only in the lockfile — the paper

outlives the repository more often than authors expect.

• If a resource you depend on has restrictive terms (query logs, commercial APIs),

state what a future reproducer can do without it: which tables survive, which die.

• Leave the qrels/run checksums in the repo README; three years later they are the

only way to prove a re-download matches the evaluated state.

Reproducibility as review defense for regular papers

• Close comparisons without variance are the most common SIGIR experimental

criticism; three seeds with mean ± sd is cheap insurance for neural systems.

• An honest "we could not reproduce baseline X's published number; we report our

best faithful configuration (details in repo)" reads as strength, not weakness —

the community knows the drift problem intimately.

• Never copy baseline numbers across collections or eval setups from other papers'

tables without saying so; mixed-provenance tables are a known reject trigger.

Quick self-audit before submission

• [ ] Every number in every table regenerates from a shipped run file plus one

documented command.

• [ ] Every system row has a config file; diffs between systems are visible as

config diffs, not prose.

• [ ] The seeds behind each neural row are enumerable, and the shipped run is

identified (which seed, or the per-topic mean).

• [ ] A colleague outside the project reproduced Table 1 from the repo README

without asking questions (the strongest cheap test available).

• [ ] The paper's reporting block and the repo's configs agree — reviewers diff

them when suspicious.

Output format

[Mode] hardening a regular paper / Reproducibility track study
[Drift audit] rows closed with artifacts: <k>/7 (collection/index/processing/pool/seed/tool/tuning)
[Reporting block] present in paper y/n; missing items <list>
[For repro studies] target + stakes / faithfulness log / divergence causes isolated
[Variance] seeds <n>, mean±sd shown y/n, test named y/n
[Biggest residual risk] <the one unpinned thing a reviewer will find>