uai-artifact-evaluation

UAI Artifact Evaluation

UAI posted no separate artifact-evaluation track or badge system for the 2026 cycle

(existence of one in later cycles: 待核实). What it did post is sharper than a badge: code

and data release is strongly encouraged, a 50 MB anonymous ZIP rides with the submission,

and reviewers judge whether claims are "backed up convincingly." The artifact's job is to

convert a skeptical probabilistic-ML reviewer's spot-check into confirmation.

Design for the five-minute skeptic

Because reviewers are explicitly not required to consult supplementary material, assume

whoever opens the ZIP gives it five minutes. Optimize for that reader:

• A top-level README that states, in its first ten lines, what claim each script

reproduces and how long it takes.

• One command per headline result, each with a small-scale mode that finishes on a

laptop: a reviewer who reproduces Figure 2 at n=500 in ninety seconds will extend you

trust for the n=50,000 version.

• Pinned dependencies and one canonical entry point. A Makefile or single driver

script beats a directory of loose notebooks.

• Expected outputs stored beside the scripts, so "did it work?" needs no judgment.

What inference artifacts must expose

Probabilistic-modeling papers fail reproduction in venue-specific ways. Package

accordingly:

| Contribution type | Must be in the artifact | Spot-check the reviewer will try |

|---|---|---|

| MCMC / SMC sampler | Seeds, chain configs, convergence diagnostics code | Re-run short chains; compare R-hat and ESS to reported |

| Variational method | ELBO logging, initialization scheme, optimizer settings | Reproduce the ELBO trace shape, not just the endpoint |

| Causal discovery | Graph-generation scripts, SHD/SID evaluation code | Regenerate synthetic graphs; verify metrics on one seed |

| Calibration / conformal | Score functions, split definitions, coverage computation | Recompute empirical coverage at one α |

| Probabilistic programming / PGM inference | Model source, query set, exact-baseline harness | Run the exact baseline on the smallest instance |

| Theory with simulations | Every constant used to instantiate the bound | Check simulation matches theorem conditions |

Anonymity inside the archive

The double-blind requirement covers all supplementary material explicitly. Archive leaks

are quieter than PDF leaks, so build from a clean export:

# Build an anonymous artifact from a clean tree, never from the working repo
git archive --format=tar HEAD | tar -x -C /tmp/uai-artifact
cd /tmp/uai-artifact
grep -rniE 'university|\.edu|author|thanks|grant' --include='*.py' --include='*.md' . | head
find . -name '*.ipynb' -exec grep -l '"authors"' {} \;      # notebook metadata
rm -rf .git .github; find . -name '.DS_Store' -delete
zip -r ../supplement.zip . && du -h ../supplement.zip        # must be ≤ 50 MB (2026 cap)

Watch for: license headers with names, pyproject.toml author fields, conda environment

exports embedding usernames, wandb/MLflow run URLs tied to an account, dataset paths

containing /home/<name>/, and model checkpoints whose training config JSON names a

cluster.

README skeleton for the archive

The archive's README is the artifact's abstract. A shape that fits the five-minute

reader:

# Supplementary code for submission #<OpenReview number>

## Claim → command map (small-scale modes; full-scale flags noted)
| Paper claim | Command | Runtime (laptop) |
|---|---|---|
| Fig. 2 (coverage vs n) | `make fig2-small`  | ~2 min |
| Table 1 (SHD, 10 seeds) | `make table1-small` | ~4 min |
| Thm. 3 simulation | `make thm3-check` | ~1 min |

## Environment
- `pip install -r requirements.txt` (pinned versions; Python 3.11)
- No GPU required for small-scale modes.

## Expected outputs
- `expected/` holds reference CSVs; each command prints PASS/FAIL against them.

## Full-scale reproduction
- Flags, hardware used, and total runtime per experiment in `FULL_RUNS.md`.

Everything above stays anonymous by construction — no names, no lab conventions in

paths, no acknowledgement of infrastructure that identifies an institution.

Data that cannot ship

• Under the 50 MB cap, ship generators, not corpora: the script that produces the

synthetic SCMs or the subsampled benchmark, plus checksums for the full data.

• For licensed or private datasets, include the loader, the preprocessing pipeline, the

schema, and summary statistics, and state access terms in the README.

• Never bypass a data-use agreement to help reviewers; explain the constraint instead.

A stated limitation is a clarity point, a violated license is misconduct under the

AUAI code of conduct.

Licensing notes

• The review-time ZIP needs no license drama, but the post-acceptance release does:

choose deliberately between permissive (MIT/BSD/Apache-2.0) and copyleft, and check

employer policy before the camera-ready deadline forces a rushed default.

• Third-party code vendored into the archive keeps its own license; verify

compatibility now, since reviewers occasionally notice a GPL file inside an

"MIT" archive and read it as carelessness.

• Data licensing is separate from code licensing; a permissive code license does not

launder a restricted dataset.

After acceptance

Convert the anonymous ZIP into a public artifact worth citing: a tagged repository, an

archival DOI where your institution supports one, a license chosen deliberately, and the

README rewritten from "reviewer instructions" to "user instructions." Link it from the

camera-ready — post-acceptance is when the CFP's release encouragement costs you nothing

and earns citations.

Output format

[Artifact scope] supplement ZIP / public release / both
[Five-minute path] <command a reviewer runs first, and its runtime>
[Claim coverage] <headline results reproducible / total>
[Anonymity scan] clean / leaks: <files>
[Size] <ZIP size vs current cap>
[Data strategy] shipped / generated / loader-plus-terms