naacl-artifact-evaluation

NAACL Artifact Evaluation

NAACL has no separate artifact-badging track; artifacts are judged inside

the ARR review itself, through the supplement upload and section B of the

Responsible NLP checklist ("scientific artifacts"). That placement matters:

your artifact documentation is not an optional extra but a set of sworn

answers reviewers cross-examine against the PDF.

What the artifact must let a reviewer do

• Trace provenance. Where did every corpus come from, under what

license, and does your use match the terms and the creators' intent?

• Inspect the instrument. Prompts, annotation guidelines, interface

screenshots, and pay rates are artifacts too — a human-evaluation claim

without its instrument is unverifiable.

• Rerun the cheap parts. Scoring scripts and metric code should execute

from the archive alone; nobody will retrain your model, but everybody can

re-score your outputs if you include them.

• Audit the data card. Language varieties, dialect coverage, speaker

demographics where relevant, known gaps, and intended use.

Language-data documentation ladder

| Data situation | Minimum documentation for a NAACL reviewer | Extra step |

|---|---|---|

| Standard public benchmark | Version, split, license, citation | Note any known contamination reports |

| Web-scraped text | Collection dates, filtering rules, deduplication, license basis | PII handling statement |

| New annotated corpus | Guidelines, annotator recruitment and pay, agreement scores | Release the guidelines verbatim in the supplement |

| Dialectal / code-switched data | Variety labels and how they were assigned | Native-speaker validation description |

| Indigenous or community-owned language data | Consent and partnership terms, community approval for release | Verify whether public release is permitted at all |

The last row is a NAACL signature concern. Work on languages of the Americas

increasingly follows community-controlled data norms: some corpora may be

used but not redistributed, some require named attribution (which conflicts

with anonymous review — use a placeholder and restore at camera-ready), and

some communities set conditions on derived models. "We release everything"

is not automatically the ethical high ground here; the checklist rewards

accuracy about constraints, not maximal openness.

Anonymous packaging that survives inspection

artifact.zip
├── README.md          # one-screen orientation: what, how, how long
├── data/
│   ├── data_card.md   # provenance, license, varieties, gaps
│   └── samples/       # enough rows to judge quality, not the corpus
├── prompts/           # exact strings, all variants tried
├── eval/
│   ├── score.py       # runs on outputs/ with no network access
│   └── outputs/       # raw model outputs backing the main tables
└── annotation/
    └── guidelines.pdf # the instrument, scrubbed of institution marks

Scrub before zipping: repository history, notebook execution metadata,

absolute paths with usernames, license headers naming the lab, and any

consent form carrying institutional letterhead (replace with a redacted

copy; note that the original exists).

Vignette: a Quechua-Spanish parallel corpus package

A submission introduces a 40k-pair Quechua-Spanish parallel corpus built

with two community organizations, plus MT baselines. The packaging calls

that follow from this skill:

• The corpus itself does not ship in the review archive — the

partnership terms permit research use but defer public release to a

community decision. The data card states this, and section B of the

checklist answers "no, with reason" for artifact release.

• What does ship: 200 sample pairs cleared for review purposes, the

collection protocol, annotator recruitment and payment description, the

cleaning scripts, and the full MT evaluation pipeline with outputs.

• The consent-form template ships with the letterhead redacted and a note

that the original is held by the partner organizations.

• The README's first paragraph tells the reviewer exactly which claims the

archive can and cannot let them verify — pre-empting the "authors refuse

to release data" misreading with a governance explanation instead.

The result is an artifact that scores as honest and inspectable rather

than incomplete, which is the realistic best outcome for

community-governed data.

Cycle-volatile mechanics

Archive size caps, accepted formats, and whether supplements upload as one

file or several are OpenReview-form details that shift between ARR cycles;

read the live submission form before building the final zip, and never

reverse-engineer the limits from a previous cycle's folklore.

Post-acceptance conversion

At camera-ready, the anonymous bundle becomes the public record: move it to

a persistent host with a DOI or a tagged release, apply the real license,

restore attribution the community partnership requires, and update the

checklist-facing statements in the paper if the release scope changed

between review and publication.

Output format

[Artifact inventory] <data / code / prompts / guidelines / outputs>
[Checklist B alignment] <each B answer -> where the artifact proves it>
[Provenance gaps] <unlicensed, undocumented, or unclear-consent items>
[Community constraints] <redistribution / attribution / approval terms>
[Anonymity sweep] clean / issues found
[Release plan] <anonymous now -> public form at camera-ready>