icml-experiments

ICML Experiments

Use this before submission or rebuttal when the central issue is whether experiments are sound

enough for ICML. The question is not just "does it win"; it is whether the evidence supports the ML

claim under fair comparison.

Experiment audit

• Baselines: current, strong, tuned, and correctly implemented.
• Ablations: isolate mechanism, architecture, objective, data, or optimization change.
• Variance: report seeds, confidence intervals, standard deviations, or a reason variance is not

meaningful.

• Data: check leakage, split construction, duplication, filtering, licensing, and representative

coverage.

• Compute: disclose hardware, training cost, inference cost, and comparison fairness.
• Scaling: show whether gains persist across model sizes, datasets, horizons, or domains when that

supports the claim.

• Negative results: use failures to define boundaries rather than hide them.
• Appendix: put supporting detail there, but keep decisive evidence in the main 8 pages.

Reviewer-pushback patterns and the ICML fix

| Pushback | Why it lands at ICML | Fix |

| --- | --- | --- |

| "Convergence guarantees under assumptions the experiments violate" | Theory paper asserts a rate under smoothness or bounded variance, but the deep-learning runs break it | State assumptions honestly, add a figure showing the rate holds empirically in-regime, flag where it does not |

| "Missing strong, tuned baselines" | The leaderboard win used an undertuned competitor | Re-tune the baseline with matched budget, report the search protocol |

| "No variance, single seed" | One run cannot separate signal from noise | Report seeds with confidence intervals or justify determinism |

| "Compute not disclosed" | ICML expects hardware and training-cost transparency | Add a compute table and confirm comparison fairness |

Worked vignette: optimizer claim audit

A paper claims a new adaptive step-size method beats Adam with a non-convex convergence guarantee.

The audit asks: is Adam tuned with the same budget, do the benchmark losses actually satisfy the

proof's assumptions, and do gains survive across seeds and model sizes? If the win shrinks under a

tuned baseline or the assumptions hold only on toy quadratics, the right move is to narrow the claim

to the regime where both theory and experiments agree, rather than overclaim a universal speedup.

Rebuttal-ready result

During response, prefer a small decisive table, corrected baseline, missing ablation, or concise

error analysis over a broad new experimental section. ICML gives one discussion round, so a single

tuned-baseline row or in-regime variance plot moves a reviewer more than a sprawling new study.

Output format

[Evidence status] strong / adequate / weak
[Most vulnerable claim] <claim>
[Critical missing result] <baseline/ablation/variance/leakage/compute>
[Small response result] <feasible clarification>
[Claim narrowing] <text if evidence is not enough>