iclr-experiments

ICLR Experiments

Use this before submission or during a revision pass to stress-test empirical claims. ICLR

experiments should answer the scientific question, not merely assemble a leaderboard.

Experiment audit

• Match each experiment to a claim in the introduction.
• Compare against current strong baselines, open-source systems, and the most relevant recent

OpenReview/arXiv papers.

• Add ablations that isolate one mechanism at a time.
• Report variance across seeds or runs when randomness can change conclusions.
• Include robustness checks for dataset shift, prompt changes, architecture variants, hyperparameter

sensitivity, or compute scale when those affect the claim.

• State compute budget, hardware, training time, inference cost, and environmental or access limits

where relevant.

• For human evaluation, document task, annotator instructions, aggregation, quality control, and IRB

or ethics status when needed.

Reviewer questions to pre-answer

• Is the baseline tuned fairly?
• Does the method win because of more compute, data, parameters, or prompt search?
• Does the effect persist outside the easiest benchmark?
• Are negative results hidden?
• Can a reviewer reproduce the headline table from the supplement or artifact?

What ICLR reviewers reward in evidence

ICLR's empirical culture prizes honest ablations and mechanism over leaderboard position. A clean

ablation that explains why a representation works often outscores a larger raw number.

| Claim type | Evidence that convinces ICLR reviewers | Common reject trigger |

| --- | --- | --- |

| New objective helps | Ablate the objective with everything else fixed | Gains confounded with extra tuning |

| Method scales | Several model sizes/tasks with a trend | One large run, no scaling curve |

| Robust representation | Tests across shifts, seeds, prompts | Single-seed peak on one benchmark |

| Beats prior method | Tuned, current, open-source baseline | Stale or under-tuned baseline |

Worked vignette

A paper claims a new self-supervised pretext task yields better linear-probe accuracy. Reviewers

ask whether the gain is the pretext task or simply longer pretraining. The author audit: hold total

pretraining compute fixed, swap only the pretext objective, and report linear-probe accuracy with

error bars over five seeds. The compute-matched ablation isolates the mechanism and is small enough

to post inline during discussion, where the table becomes part of the permanent public record.

Reviewer-pushback patterns

• "You win because of more compute." Add a compute-matched control; report FLOPs, not just wall time.
• "Only one seed." Report mean and spread across seeds; an unstable benchmark needs variance.
• "Baseline is weak." Cite the baseline's own recommended settings and show you matched them.
• "Ablation removes two things at once." Split into single-mechanism ablations a reviewer can read.

Output format

[Claim] <paper claim>
[Experiment evidence] sufficient / needs baseline / needs ablation / needs robustness
[Fairness issue] <compute, tuning, data, prompt, metric>
[Fast fix] <experiment or analysis feasible before deadline>
[Appendix placement] <what can move out of main text>