webconf-artifact-evaluation

Web Conference Artifact Evaluation

The Web Conference's 2026 artifact program was an **availability badge, not a

reproducibility audit**: a volunteer committee performed a light verification that

the artifact exists, is downloadable from a publicly accessible archival

repository, and carries minimal instructions. Accepted papers from all tracks of

the main or companion proceedings — except workshop papers — could apply, with the

artifact submitted at camera-ready time. Design the artifact to clear that bar

first, then exceed it for your actual readers.

What the badge checks vs. what readers need

| Dimension | Badge bar (2026) | Reader bar |

|---|---|---|

| Location | Public archival repository | Same, plus a mirror for big files |

| Persistence | Link resolves at check time | DOI/versioned, survives repo moves |

| Instructions | "Minimal" access instructions | Environment, run commands, expected output |

| Completeness | Artifact "exists" | Enough to regenerate headline tables |

| Legality | Not assessed | Licenses and platform ToS actually permit sharing |

"Archival" is the operative word: a personal GitHub repository can be deleted or

rewritten and historically has not counted as archival for ACM badging purposes.

Deposit a release snapshot in Zenodo, figshare, or an institutional repository that

mints DOIs, and let GitHub be the development mirror the README points to.

Web-data artifacts are legally different

This venue's artifacts are dominated by crawls, platform datasets, interaction

logs, and derived embeddings — objects with third-party rights attached. Decide the

sharing tier per component before promising anything in the paper:

1. Freely shareable: your code, your synthetic data, aggregate statistics.
2. Shareable as derivatives: URL lists, item IDs, hydration scripts — release

the pointers plus the pipeline, not the content, when platform terms forbid

redistribution (the standard pattern for social-media datasets).

1. Shareable on request / gated: user-level data behind a data-use agreement;

document the request path in the README so the artifact is still "available."

1. Not shareable: proprietary logs. Say so in the paper's availability

statement and ship the measurement code anyway.

A takedown-resilient design states the crawl window, preserves checksums of the

original corpus, and includes the re-crawl script — so when 15% of URLs die (they

will), a later user can quantify exactly what changed.

Minimum viable package

artifact-v1.0/  (deposited at Zenodo, DOI 10.5281/zenodo.XXXXXXX)
├── README.md          # what this is, paper title, 3-command quickstart
├── LICENSE            # code license + per-dataset terms table
├── environment.yml    # or Dockerfile / requirements.txt with pins
├── data/
│   ├── MANIFEST.md    # per-file: source, crawl window, row counts, sha256
│   └── ids/           # hydration pointers where content can't ship
├── src/               # training/analysis code, seeds surfaced as flags
├── scripts/
│   ├── reproduce_table2.sh
│   └── recrawl.py     # dead-link accounting for future users
└── CITATION.cff

The single highest-leverage file is reproduce_table2.sh — one command per

headline result. Badge reviewers may not run it; the citing researcher eighteen

months later definitely will.

During review vs. after acceptance

Review at this venue is double-blind and the appendix is the sanctioned home for

reproducibility detail, so during review the artifact appears as an anonymized

link (if at all) with no author-identifying paths, git history, or platform account

names. After acceptance, the flow inverts: mint the DOI first, cite it in the

camera-ready, then file the badge application alongside the final files — the

badge's availability check needs the link that will live in the published paper.

Sequence details sit in webconf-camera-ready; content standards for what the

package proves sit in webconf-reproducibility.

Vignette: the crawl that got a takedown notice

A team releases a 40M-page news crawl with their WWW paper. Eight months later, a

publisher demands removal of its articles. Because the artifact followed the

tiered design, the response costs an afternoon, not the artifact: the Zenodo

deposit v1.1 removes the publisher's content files, the manifest documents the

removal (URLs retained as pointers, checksums preserved), the hydration script

still lets rights-holding users rebuild the full corpus, and the paper's DOI

citation now resolves to a version history that explains itself. The

counterfactual design — one monolithic tarball with raw content — would have

forced a full withdrawal and orphaned every downstream citation. Design for the

takedown on day one; on the Web it is a when, not an if.

Badge application dry run

Before filing with the camera-ready, simulate the committee's light check from a

clean machine and a logged-out browser:

1. The link in the paper resolves to the archival record (not a login page, not a

personal homepage redirect).

1. The record shows a version, a license, and a README visible without download.
2. One file downloads successfully and matches its manifest checksum.
3. The README's first screen states what the artifact is and which paper it

accompanies — the committee volunteer decides in minutes.

Honest labeling

The Artifacts Available badge certifies existence, not correctness. Do not write

"our results are independently verified" on the strength of it. Conversely, an

artifact too encumbered to badge (tier 3-4 data) does not bar publication — the

paper just needs an availability statement that says precisely what is released,

what is gated and why, and what a reader can still verify with the released parts.

Output format

[Artifact tiering] code=<tier> data=<tiers per set> models=<tier>
[Badge eligibility] track eligible? archival deposit? DOI minted?
[Legal check] platform ToS / license conflicts found: <list or none>
[Decay plan] crawl window stated? checksums? recrawl script?
[One-command repro] present for headline tables: yes/no
[Availability statement] drafted for the paper: yes/no