colm-author-response

COLM Author Response

In 2026 the COLM rebuttal ran May 22 to June 8: reviews and the response window

opened the same day, giving about eighteen days between first sight of the reviews

and the last word you get before the July 8 decision. A rebuttal in that geometry is

a project with a plan, not a reply-all reflex.

Budget the eighteen days

days 1-2    read only; draft nothing; log every reviewer claim into the four bins
            (misreading / evidence gap / scope fight / correct hit — colm-review-process)
days 3-9    run the winnable experiments — cache-based analyses first, single-GPU
            or small-API-budget runs second; anything needing full retraining is
            out of scope, say so plainly
days 10-13  draft one response per reviewer + a short common preamble for
            cross-cutting points; every paragraph anchored to a page, table,
            appendix section, or new number
days 14-16  co-author cold read: does each response open with its conclusion?
days 17-18  post with margin; OpenReview deadline-hour uploads are self-harm

The highest-yield experiments are the ones your artifact discipline pre-paid for: a

cached-responses archive lets you re-slice results, add a metric, or test a reviewer

hypothesis in hours without touching a model (colm-artifact-evaluation).

The objections COLM reviewers actually raise

These four LM-specific objections dominate borderline cases; each has a right and a

wrong answer shape:

| Objection | Weak answer | Strong answer |

|---|---|---|

| "Benchmark is likely contaminated for these models." | "This benchmark is widely used." | Overlap analysis numbers, or dated-item split showing the effect survives on post-cutoff items |

| "Baselines are under-tuned; better prompting closes the gap." | "We used standard prompts." | The tried-ledger: k prompt variants per system, matched budgets; if the gap narrows, report the narrowed gap |

| "Results are on one model family; is this about LMs or about model X?" | "We expect it generalizes." | One added family inside the window, claim rescoped in the revision either way |

| "API model results won't reproduce." | "We used the public API." | Version strings + query dates already in App. A, plus commitment to release the response cache |

Notice the pattern: every strong answer is either *evidence you can generate this

week or a scope concession that makes the paper truer*. Both move ACs; indignation

moves nothing.

Writing mechanics

• Conclusion first, then support. "The gap survives decontamination (new Table

R1): on items post-dating every model's cutoff, the effect is 7.9 points." Then

the method sentence.

• Number new artifacts R1, R2, ... and reference them consistently; the AC

skims all responses in one sitting and consistent labels are navigation.

• Quote the reviewer's own words for each point you answer — it prevents the

"they dodged my question" reading — and answer the words, not a nastier

paraphrase.

• Concede in the open. A visible "R2 is right; we overclaimed X and have

rescoped to Y" is the single most credibility-generating sentence a rebuttal can

contain.

• Stay anonymous. The discussion is inside double-blind: no lab names, no

"in our previous paper," no links that resolve to you.

• Keep LLM-policy discipline — if a response experiment used LLM-based

evaluation, it is disclosable under the same 2026 policy as the paper.

Handling the difficult cases

• The low-engagement review (two lines, generic, possibly LLM-drafted without

disclosure): respond substantively and briefly, then note factually in a comment

visible to the AC that the review does not engage the paper's content. Flag, do

not accuse.

• The scope-fight reviewer (wants a different paper): one crisp paragraph on

why the paper's question is the right size, pointing at the venue's own genre —

measurement and analysis papers are COLM's award lineage — then stop. Repetition

reads as anxiety.

• The correct hit you cannot fix in eighteen days: say what the fix is, why it

does not fit the window, and how the revision will scope claims meanwhile. ACs

reward authors who can tell the difference between a flesh wound and a broken

spine.

Per-reviewer response skeleton

One response per reviewer, same skeleton, filled with their words and your numbers:

Thank you for the careful reading. We address each point below.

> "<reviewer's sentence, quoted verbatim>"
<conclusion sentence with the page/table/new-artifact anchor>
<one or two support sentences — method, numbers, or the concession>

> "<next quoted point>"
...

Summary of changes committed for the revision:
 1. <rescoped claim / new table R1 folded into §4 / added limitation>
 2. ...
We believe the remaining disagreement (if any) is <one neutral sentence>,
which we leave to the AC's judgment.

Length discipline: reviewers are reading many rebuttals in the same week you are

writing yours. A response that fits on one screen per reviewer, with bolded

conclusions and numbered artifacts, gets read twice; a 2,000-word essay gets

skimmed once. If a point genuinely needs depth, put the depth in a clearly labeled

final section so the main thread stays scannable.

After June 8

Nothing. The reviewer-AC discussion through the July 8 decision has no author

channel — resist the urge to post updates. Spend the month preparing both branches:

the camera-ready plan (due August 7 if accepted) and the resubmission map

(colm-review-process).

Output format

[Window] days remaining: <n> of ~18
[Per reviewer] R<k>: bins A/B/C/D counts + the one thing they must hear
[Experiment queue] <cache-based first, with owner and finish-by date>
[Concessions] <what gets conceded and the rescoped claim>
[AC preamble] <3-sentence summary of what changed during discussion>