iccv-experiments

ICCV Experiments

An ICCV experimental program is built under a fixed, unrepeatable date: the

early-March deadline of an odd year (March 7 in 2025, with supplement due the

same day). The program design problem is therefore *sequencing under a

deadline*, on top of the usual question of what evidence convinces a vision

reviewer. Both are handled here.

Start with the benchmark-drift audit

Because the venue is biennial, a project conceived after one ICCV and submitted

to the next spans two years of field motion. Before designing experiments, audit

what moved:

drift audit (fill once, in the autumn before the deadline)
  benchmarks:   which datasets did the last two years of CVPR/ECCV/ICCV papers
                in this area actually evaluate on? any new canonical benchmark?
  baselines:    leaderboard top-3 today vs when the project started
                → any baseline in your draft older than ~18 months is a red flag
  backbones:    what does current SOTA initialize from? (matching this defines
                "fair" for your comparisons)
  metrics:      any metric revision or new evaluation server since last cycle?
  protocols:    resolution/prompt/eval-harness conventions that changed

Papers rejected for "outdated comparisons" are usually not lazy — they froze

their experiment matrix at project start and never re-based. Re-run the audit in

January; two months before an ICCV deadline is exactly when the previous

November's CVPR-cycle preprints flood arXiv.

The fairness ledger

Vision reviewers' most reliable objection is compute-and-pretraining confounds

dressed as method wins. Make fairness auditable with a ledger column per

comparison:

| Axis | Your method | Each baseline | Mismatch handling |

|---|---|---|---|

| Backbone + init checkpoint | | | Match, or add a matched row |

| Pretraining data exposure | | | Disclose; beware test-adjacent leakage in web-scale corpora |

| Input resolution / tokens | | | Match or tabulate both |

| Training schedule + budget | | | Report epochs and GPU-hours side by side |

| Number quoted vs re-run | | | Mark re-runs; footnote protocol deltas |

The foundation-model twist: when everything builds on the same giant checkpoint,

data exposure replaces architecture as the confound reviewers hunt. If your

improvement could plausibly come from the pretrain having seen the test domain,

run the decontamination or cross-domain check before a reviewer asks for it in a

window when you have one rebuttal page to respond.

Ablations that isolate, not decorate

Structure the ablation grid so every row flips one switch, and include the two

rows that distinguish a mechanism from a lucky configuration: the transplant

(your module inserted into a baseline — does the gain travel?) and the

sensitivity sweep (is the headline number a plateau or a spike?). Rows argued

from in the text belong in the body; the full grid goes to the same-day

supplement. If the core ablation shows the mechanism is not doing the work, that

is an October discovery you want in November — which is why it runs first (see

sequencing below).

Qualitative evidence with stated rules

At a venue that reviews with its eyes, image and video evidence carries real

weight and attracts real skepticism. Three requirements: a declared selection

rule on every grid ("first N val images", "random seed 0" — curation without a

rule is what reviewers assume by default); side-by-sides against the two

strongest baselines on identical inputs; and a failure-mode section with a

taxonomy, previewed in the body and cataloged in the supplement. For temporal or

3D claims, the supplement video is the primary exhibit — packaging in

iccv-supplementary.

Sequencing runs toward March

The scarce resource is calendar, not GPUs. Order the program by *decision value

per week*:

1. Falsifiers first (autumn): the core ablation and the strongest baseline

comparison. If the idea dies, it dies while retargeting to CVPR-November is

still possible (iccv-workflow's autumn fork).

1. Headline table (winter): full benchmark suite, matched settings, seeds on

the cheap rows.

1. Breadth pass (January): second domain, transfer, robustness suite —

whatever supports the claim's outer scope.

1. Freeze margin (mid-February): the main table freezes ~2–3 weeks out; the

deadline is same-day for the supplement, so late results have no legal

landing zone except the rebuttal — and reviewers may not request major new

experiments anyway.

1. Rebuttal reserve: hold 10–20% of compute for May; the most common

winnable rebuttal item is a small matched re-run reported as a mini-table

(iccv-author-response).

# deadline_math.py — sanity-check the plan against the calendar
runs = {"core_ablation": 6, "main_table": 21, "breadth": 10}   # GPU-days each
gpu = 8; days_left = (deadline - today).days - 18              # freeze margin
assert sum(runs.values()) / gpu <= days_left, "cut scope now, not in February"

The four questions any ICCV review silently asks

Does it work (main table, matched)? Why does it work (isolating ablations +

transplant)? Where does it break (failure taxonomy, honest transfer results)?

What does it cost (params, latency on named hardware, training GPU-hours —

volunteered, since no form mandates it; see iccv-reproducibility)? Draft the

experiments section as answers to these four, in this order, and the reviewer's

checklist fills itself.

Reverify each cycle

• The 2027 deadline chain — sequencing above is calendar-shaped and the calendar

is 待核实 until posted.

• Whether supplements stay same-day (changes step 4).
• Evaluation-server rules and submission budgets on your benchmarks.
• Any new ethics/human-data documentation the 2027 forms may require.

Output format

[Drift audit] run on <date>; stale baselines found: <list>
[Fairness ledger] axes matched or disclosed per comparison: n/m
[Ablation] one-switch rows: <n>; transplant + sensitivity present: yes/no
[Qualitative] selection rules stated · failure taxonomy drafted
[Sequencing] falsifiers scheduled before <date>; rebuttal reserve: <GPU-days>
[Cut candidates] <lowest decision-value runs if the calendar slips>