facct-artifact-evaluation

FAccT Artifact Evaluation

Use this for the material that backs a FAccT paper's transparency and accountability claims. Note

the venue difference up front: FAccT does not run the SIGSOFT-style ACM Artifact Review and

Badging track that software-engineering venues use, and it does not hand out Available/Functional/

Reusable/Reproduced badges. 待核实: confirm on the current Author Guide whether any optional

artifact/reproducibility appendix or badge scheme has been added for your cycle. What FAccT does

have is a strong norm of accountability documentation — datasheets, model cards, data

statements, audit trails, and impact assessments — plus released code and data. Treat those genres

as your artifact and make each one credible on its own.

The FAccT documentation genres (know which your paper needs)

| Genre | What it documents | When your paper needs it |

|---|---|---|

| Datasheet for a dataset | Motivation, composition, collection, preprocessing, uses, distribution, maintenance | You release or rely on a dataset |

| Model card | Intended use, training data, evaluation disaggregated by group, ethical considerations, limits | You release or audit a model |

| Data statement (for language data) | Speaker/annotator demographics, curation rationale, language variety | You build or study a text/NLP corpus |

| Audit / evaluation report | Method, subgroup metrics, thresholds, what was and was not tested | Your contribution is an audit |

| Impact / risk assessment | Foreseeable harms, affected populations, mitigations, residual risk | Deployment or dual-use is plausible |

Pick the genres your claims actually require; a model audit with no model card, or a dataset paper

with no datasheet, reads as incomplete to this community.

What a credible documentation artifact contains

[Provenance]   where the data/model came from, when, under what terms and consent
[Composition]  who/what is in it, who is absent, and the resulting blind spots
[Disaggregation] evaluation broken out by protected/affected subgroup, with uncertainty
[Intended use]  what it is for — and an explicit "off-label" / do-not-use list
[Limits & harms] known failure groups and foreseeable adverse impacts, not just accuracy
[Maintenance]  who updates it, how issues are reported, how long it persists
[License]      a clear license for released code/data so others can lawfully reuse it

Released code and data (the reproducibility half)

• Ship the analysis that turns data into the paper's disaggregated findings, with pinned data

versions and seeds, so a reader can re-run the harm claim.

• Deposit released data or a public archive in a persistent location (e.g. a DOI-issuing repository)

for the camera-ready; keep it consistent with the datasheet.

• For model-generated or scraped inputs, cache raw outputs and record model IDs, dates, and terms —

a study that needs a live API or a since-changed website re-samples rather than reproduces.

Anonymized review version vs. public release

• At submission: any documentation or code shipped for reviewers must be anonymized — no

author names, institution paths, cluster URLs, or identity-revealing repositories, and the

Positionality statement stays out entirely (it is not anonymous).

• After acceptance: replace anonymized placeholders with the public, licensed, persistently

archived versions the camera-ready cites, and finalize the datasheet/model card so it matches the

released artifact exactly.

Consistency with the paper's harm claims

The artifact's job at FAccT is to make the paper's accountability claims checkable. Every

disparity, harm, or transparency benefit the paper asserts should be traceable into the

documentation or released analysis. A model card whose disaggregated numbers disagree with the

paper's table, or an impact assessment that omits the harm a reviewer can foresee, undercuts the

paper more than having no artifact at all.

Vignette: an audit paper's artifact set

A paper auditing a commercial classifier ships: a datasheet for the evaluation dataset (how

assembled, subgroup composition, consent basis); a model card-style report for the audited

system as the authors understand it (intended use, disaggregated error, failure groups); the

audit code with pinned data and seeds regenerating each subgroup table; and a short **impact

assessment** naming who is harmed by both the system and by publishing the audit, with mitigations.

All anonymized for review, all public and licensed at camera-ready, all consistent with the paper's

tables.

Calibration

• FAccT's artifact expectations are documentation- and release-centered, not badge-centered; do

not import a Docker-image/badge checklist as if it were the bar.

• Whether any optional artifact appendix, reproducibility checklist, or badge exists is

cycle-volatile — confirm on the current Author Guide (待核实).

Output format

[Genres needed] <datasheet / model card / data statement / audit report / impact assessment>
[Artifact role] anonymized review version / public release
[Contents] <provenance / disaggregation / intended-use / limits / license>
[Claim mapping] <paper harm claim -> where in the documentation/analysis it is checkable? yes/no>
[Consistency] <artifact numbers match the paper's tables? yes/no>
[Fixes before upload] <ordered list, kept anonymous for review>