cvpr-artifact-evaluation

CVPR Artifact Evaluation

CVPR has no badge-granting artifact committee; the "evaluation" of your artifacts is

done informally, twice, by two different audiences. At review time, reviewers poke at an

anonymous supplement while deciding whether to trust your tables. After acceptance, the

entire field — 2026 drew 44,011 authors alone — decides whether your method becomes a

baseline or a citation nobody reproduces. Package for both audiences deliberately.

Review-time artifacts: sealed-box rules

Everything must ride inside the uploaded supplement, because the 2026-verified policy

bans external links that expand content or subvert review. That kills the usual

anonymous-repo-link workflow many venues tolerate. Practical consequences:

• Code goes in the supplement archive, scrubbed: no .git history, no usernames in

paths, no institutional cluster hostnames, no wandb entity names in configs.

• Model weights usually exceed sane archive sizes; ship configs + training commands +

exact dependency pins instead, and say which checkpoints will be released.

• Demo videos are first-class CVPR evidence (see cvpr-supplementary) but strip

container metadata — video files carry author tags more often than PDFs do.

• The 2026 supplementary size cap was not verifiable at check time (待核实); design

the package assuming a strict cap, then confirm on the current page.

# Build a clean review artifact from a working repo
git archive --format=tar HEAD | tar -x -C /tmp/artifact       # no .git, no untracked junk
cd /tmp/artifact
grep -rniE "$(whoami)|<lab-name>|<cluster-host>|wandb\.(ai|entity)" . | head   # identity scan
pip-compile --quiet requirements.in                            # pin, don't approximate
zip -r ../supplement_code.zip . -x '*.ckpt' '*.pth'            # weights out, recipe in

The runnable-by-a-stranger bar

A reviewer who opens your archive gives it minutes, not an afternoon. One entry point,

one config, one expected number:

| Artifact layer | Minimum viable | Gold standard |

|---|---|---|

| Environment | pinned requirements.txt + CUDA/driver note | Dockerfile reproducing the CRF hardware row |

| Inference | script + 5 sample images + expected outputs | notebook rendering figure-quality results |

| Training | full config + command + seed | resumable run with logged curves |

| Evaluation | script that recomputes one main-table row | all-tables harness with dataset download stubs |

| Data | loader + split files + provenance note | checksummed archive or release plan with license |

The single highest-leverage file is REPRODUCE.md mapping **each table/figure in the

paper to one command**. It converts a suspicious reviewer into a supportive one faster

than any rebuttal sentence.

Release-time artifacts: promises come due

Two clocks start at acceptance. First, the verified dataset clause: a dataset claimed as

a contribution must be public by the camera-ready deadline — hosting, license, and

consent scrubbing included. Second, the softer but real credibility clock: the gap

between "code coming soon" in the README and actual code is measured publicly at CVPR

scale.

Release decisions to make explicitly rather than by default:

• License: weights and data need one (CC-BY-SA? research-only? non-commercial?);

"no license" means legally unusable for the industrial half of the CVPR audience.

• Weights: which checkpoints — final only, or the ablation grid reviewers saw?
• Provenance: for scraped or generated data, document sources and filtering; for

human data, document consent posture. Vision datasets attract scrutiny years later.

• Benchmarks: if you release an evaluation, freeze the metric code and version it —

silent metric fixes fork the leaderboard.

Weights and hosting decisions

Model releases have their own failure modes at vision scale:

• Host durability: lab web servers die with graduations. Prefer archival or

platform hosting (institutional repositories, model hubs) with a checksum published

in the README; the CVF open-access page will outlive your URL, so choose hosts with

the same life expectancy.

• Checkpoint provenance: name each released checkpoint after the paper's table row

it reproduces (table2_row4_vitb.pth), and state which exact code commit evaluates

it to the printed number.

• Safety posture for generative models: decide before release what you ship —

weights, LoRA deltas, or inference-only API — and write the model card's misuse

paragraph yourself rather than letting the first misuse write it for you.

• Version pinning at release: tag the repo at camera-ready (v1.0-cvpr) so later

refactors don't silently break the commands printed in the paper's supplement.

Anti-patterns with CVPR-specific cost

• A project-page URL in the submission "for videos" — that is the banned external link,

not a convenience.

• Supplement code that hard-codes /home/<name>/data — an anonymity leak and proof

nobody ran it elsewhere.

• Claiming a dataset contribution, then releasing it months after camera-ready — a

verified policy breach, not just bad manners.

• Releasing training code without evaluation code: at a benchmark-driven venue the

evaluation is the artifact people actually reuse.

After release: the maintenance tail

A used artifact generates issues, and the field reads your issue tracker. Budget a

small maintenance window post-conference: pin the environment against dependency rot,

answer the first wave of "can't reproduce Table 2" issues (usually environment

mismatches — point to the Dockerfile), and keep a RESULTS.md of community

reproductions. Six months of light maintenance is what turns a CVPR paper into the

baseline the next cycle's papers must cite.

Reverify each cycle

• Supplement size/format caps and whether anonymized-repo links are (dis)allowed — the

external-link wording can shift by year.

• The dataset-release clause wording.
• Any new artifact, checklist, or reproducibility fields on the OpenReview form.

Output format

[Artifact stage] review-supplement / camera-ready-release
[Identity scan] clean / hits: <files>
[Runnability] entrypoint · pins · expected-output map present?
[Weights & data plan] <ship now / release plan + license>
[Dataset clause] n/a / due at camera-ready: <status>
[Gaps] <ordered fixes>