eacl-reproducibility

EACL Reproducibility

Use this to audit an EACL paper against the Responsible NLP checklist, which ARR files at

submission and which the action editor and reviewers read alongside the PDF. The checklist is

not paperwork: misleading answers are desk-rejection grounds, and inconsistencies between

the checklist and the paper are what careful EACL reviewers hunt for. Reopen the current

checklist at aclrollingreview.org/responsibleNLPresearch before auditing.

Treat the checklist as a claims audit

Every "yes" in the checklist implies a location in the paper. Walk the paper claim by claim and

confirm each is backed:

| Claim type | Must disclose | Common EACL failure |

|---|---|---|

| Model results | Hyperparameters, tuning, model size | "Default settings" with no numbers |

| Compute | Hardware, run time, total budget | Silent on cost of large runs |

| LLM prompts | Verbatim prompts + decoding params | Paraphrased or omitted prompts |

| Data | Source, license, splits, preprocessing | Undocumented or "on request" data |

| Metrics | Variance over seeds / significance test | Single-run deltas reported as wins |

| Multilingual | Languages, resource levels, per-language results | Aggregate hides where it fails |

Contamination stance (LLM era)

• State explicitly whether evaluation data could have leaked into training — for closed LLMs

this is often unknowable, and the honest move is to say so and bound the risk rather than

claim clean evaluation. The "Leak, Cheat, Repeat" exemplar in

../../resources/exemplars/library.md is the reference discipline.

• Where feasible, run an overlap or decontamination check and report it.

Variance and significance floor

Reporting rule of thumb:
  - >= 3-5 seeds for any headline comparison
  - report mean +/- CI or std, not a single run
  - a significance test when two systems are "close"
  - never claim a win on an unreplicated single-run delta

Multilingual coverage honesty

• If the paper claims a cross-lingual or multilingual result, the checklist audit must

confirm the languages are named, the resource levels are stated, and per-language results

exist somewhere — an aggregate average is not evidence for every language.

• For lower-resourced languages, confirm dataset provenance and annotation context are

documented; this is a recurring EACL reviewer expectation.

Checklist-to-paper consistency sweep

[ ] Every checklist "yes" maps to a section/appendix number
[ ] Hyperparameters + search space stated
[ ] Compute budget stated for expensive runs
[ ] Prompts + decoding params verbatim (if LLMs used)
[ ] Data source, license, splits, preprocessing documented
[ ] Variance/significance reported for headline claims
[ ] Contamination risk addressed honestly
[ ] Per-language results present for multilingual claims
[ ] AI-assistance use disclosed truthfully

Output format

[Reproducibility risk] Low / Medium / High
[Checklist-paper mismatches] <specific "yes" answers not backed in text>
[Disclosure gaps] <hyperparameters/compute/prompts/data>
[Evidence floor] <seeds/variance/significance findings>
[Contamination] <stance + any check run>
[Fix order] <what to add before the cycle deadline>