eacl-artifact-evaluation

EACL Artifact Evaluation

Use this to turn a paper's evidence into artifacts that survive review and become a public

release. EACL runs through ACL Rolling Review, so the artifact lives two lives: an **anonymized

supplement attached at ARR submission, and a public release** after commitment acceptance.

Both are audited against the Responsible NLP checklist. Reopen the current checklist before

packaging.

The two lives of an EACL artifact

| Stage | Form | Must be | Owner |

|---|---|---|---|

| ARR submission | Anonymized .zip/.tgz supplement | Fully de-identified, self-contained | Authors |

| Commitment acceptance | Public repo + Anthology link | Licensed, versioned, reproducible | Authors |

Do not conflate them: the review supplement must contain no author-identifying strings,

while the public release must contain **exactly the identifying and licensing information the

supplement omitted**.

What belongs in an EACL artifact

• Code to reproduce the headline tables, with a top-level entry point.
• Data: the dataset or a loader plus a documented path to it; if redistribution is

restricted, document access precisely rather than implying release.

• Prompts and decoding settings verbatim for any LLM-based result — these are part of the

method, not an afterthought.

• Model outputs retained so scores can be re-computed without re-running expensive models.
• Annotation materials: guidelines, interface, pay information, and inter-annotator

agreement.

Anonymized-supplement checklist

[ ] No author names in paths, file headers, LICENSE, or notebook metadata
[ ] Git history stripped or repo re-initialized
[ ] No personal hosting URLs (Drive/Dropbox) that identify authors
[ ] Prompts + decoding params included verbatim
[ ] Model outputs included for re-scoring
[ ] A README that reproduces at least one reported table
[ ] Smoke-checked (see resources/code/README.md)

Run the shared smoke checker before upload:

python3 ../../../shared-resources/ml-conference-methods/code/check_repro_package.py /path/to/anonymous-supplement

Licensing and documentation for the public release

• Choose a license appropriate to code (e.g. permissive) and data (respecting upstream

dataset terms); the paper text should state it.

• Document intended use and known limitations of any released dataset — required by the

checklist and expected by the European community's data-governance norms.

• Version the release with a tag that matches the camera-ready, so the Anthology PDF and the

repo cannot drift.

Multilingual and lower-resource specifics

• If the artifact covers lower-resourced languages, document **provenance and speaker/annotator

context** carefully; thin documentation of a low-resource dataset is a common EACL reviewer

concern.

• Keep language codes and scripts explicit (ISO codes, script variants) so the artifact is

usable by others working on those languages.

Output format

[Artifact stage] Anonymized supplement / Public release
[Contents] <code/data/prompts/outputs/annotation coverage>
[Anonymization] <pass/fail with specific leaks>
[Reproduces] <which reported table the README regenerates>
[Licensing + docs] <license, dataset terms, intended-use note>
[Gaps] <what a reviewer could still not reproduce>