eacl-experiments

EACL Experiments

Use this to make an EACL paper's evidence hold up under NLP review. EACL rewards **well-scoped

questions answered with careful controls** over leaderboard maximalism — its best papers include

analyses and critiques, not only state-of-the-art systems (see

../../resources/exemplars/library.md). Design the evidence to match the claim exactly, no

broader.

Baselines that make a comparison fair

• Include a tuned baseline, not a strawman: an under-tuned competitor makes a win

meaningless. State the search space for both your method and the baselines.

• For LLM-based work, include the obvious prompt/few-shot baseline and report its prompts and

decoding settings; a gain over an unreported baseline is not credible.

Match breadth to the claim

| Claim | Required breadth |

|---|---|

| "Works for language L" | Solid results on L, honestly scoped |

| "Cross-lingual / multilingual" | Enough languages across resource levels; per-language results |

| "General method" | Multiple tasks/datasets, not one convenient benchmark |

| "Robust" | Stress tests / shifts, not just in-distribution |

A multilingual claim backed by two high-resource languages is the classic EACL over-reach — the

morphology-across-57-languages exemplar shows the bar.

Significance and variance floor

Evidence floor for a headline comparison:
  seeds:        >= 3-5 runs
  report:       mean +/- CI (or std), never a lone run
  significance: a test when systems are close
  ablations:    isolate each component's contribution

Contamination controls

• For any benchmark evaluated with LLMs, address whether the test data could have leaked.

Report an overlap/decontamination check where feasible, or bound the risk honestly for

closed models. This is a live EACL concern, not a formality.

Human evaluation done properly

• If human judgments are a result, report the **number of annotators, guidelines, pay, and

inter-annotator agreement** — an unmeasured human eval is a soft target for reviewers.

• Release the annotation materials (see eacl-artifact-evaluation).

Error analysis as a first-class result

• A quantified error taxonomy ("X% agreement errors, Y% named-entity errors, examples in

Table N") often carries more scientific weight than another decimal of accuracy, and plays to

EACL's analysis-friendly reviewing.

Audit checklist

[ ] Baselines tuned, search spaces stated
[ ] LLM baselines with verbatim prompts + decoding
[ ] Breadth matches the claim (per-language results if multilingual)
[ ] >= 3-5 seeds; variance/CIs reported
[ ] Significance test where systems are close
[ ] Ablations isolate each component
[ ] Contamination addressed
[ ] Human eval: annotators, agreement, pay reported
[ ] Error analysis quantified

Output format

[Evidence strength] Strong / Adequate / Weak
[Baseline fairness] <tuned? LLM baseline reported?>
[Breadth vs claim] <matched / over-reaching>
[Variance + significance] <seeds, CIs, tests>
[Contamination + human eval] <controls present?>
[Fix order] <experiments to add/scope before the cycle deadline>