colm-supplementary

COLM Supplementary Material

The 2026 format gives you a strict 9-page main text, unlimited citation pages, and

appendix space whose guaranteed audience is nobody — reviewers may read it, the

rebuttal may point into it, but no rule obliges anyone to open it. That asymmetry is

the whole design principle: the main text carries every load-bearing element, and

the supplement carries the audit trail that makes the main text checkable.

The placement rule

Ask of each artifact: is this needed to believe the claim, or to re-run it?

Believe → main text. Re-run → appendix/supplement. Both → summarize in the main

text, deposit fully in the appendix, and cross-reference in both directions.

| Artifact | Placement | Note |

|---|---|---|

| Representative prompt (or skeleton) | Main text | Readers cannot judge an LM evaluation without seeing how the model was asked |

| Full prompt set, verbatim, incl. system prompts | Appendix | Paraphrased prompts are not reproducible; include exact strings |

| Decoding configs per experiment | Appendix table | One row per reported table cell group (colm-experiments) |

| Contamination analysis method + results | Summary in main text; full in appendix | The conclusion is load-bearing; the n-gram machinery is audit trail |

| Per-task / per-model full result grids | Appendix | Main text shows the aggregate + the interesting slice |

| Human-eval protocol: instructions, pay, demographics, agreement | Appendix | Screenshots of annotator UI earn trust |

| Failure cases and transcripts | Appendix, curated | A dozen representative transcripts beat 400 raw ones |

| Code + data package | Supplementary upload / anonymized repo | Must be anonymous at submission (see below) |

| Raw API responses archive | Release artifact (acceptance) | Often too large for review upload; describe it and commit to release |

LM-specific supplement content reviewers actually use

Three appendix sections repeatedly decide borderline LM papers:

1. The prompt appendix. Exact strings, exemplar counts, formatting whitespace

included. Prompt paraphrase is a known variance source, so approximation here

undermines every number.

1. The "what we tried" ledger. Prompt variants and hyperparameters attempted

per system — the fairness evidence for baseline comparisons.

1. The contamination appendix. Overlap statistics per evaluation set, or the

documented reason none could be computed.

Anonymization of the code/data package

COLM's instructions ban identity-revealing links, and supplements are the usual

leak. Before upload, scrub: repository remotes and commit author fields (git log

travels inside .git/), Hugging Face org names in from_pretrained(...) calls,

W&B entity names in configs, API org IDs in cached headers, absolute paths

(/home/<username>/...), and internal cluster hostnames.

# Build the review package from a clean export — never zip a working directory
git archive --format=tar HEAD | (mkdir -p /tmp/colm-pkg && tar -x -C /tmp/colm-pkg)
grep -rniE 'wandb|hf_[A-Za-z0-9]{20,}|api[_-]?key|/home/[a-z]|<lab-name>' /tmp/colm-pkg | head
grep -rlE 'from_pretrained\("[^"]*/' /tmp/colm-pkg --include='*.py' | head   # org-scoped model IDs
(cd /tmp && zip -rq colm-supplement.zip colm-pkg -x '*.git*')

git archive (not cp -r) is the load-bearing choice: it cannot smuggle .git/,

untracked scratch files, or environment files into the package.

Referencing the supplement from the main text

• Point at specific anchors ("App. C.2, Table 9"), never "see appendix".
• In the rebuttal window (May 22 - June 8 in 2026), precise anchors let you answer a

reviewer in one sentence plus a pointer — the highest-leverage use of everything

you deposited.

• Do not promise supplement content that is not there; reviewers who follow a

dangling pointer stop trusting the intact ones.

Ordering the appendix for the reader you hope exists

Appendices get opened with a question in hand, not read linearly. Order sections by

the probability of the question:

1. A. Experimental details — models, revisions, decoding configs, harness

commit: the reproduction question.

1. B. Prompts — verbatim, grouped by experiment: the "how did you ask" question.
2. C. Contamination analysis — the trust question.
3. D. Full results — per-task, per-model grids: the "does it hold on X" question.
4. E. Baseline tuning ledger — the fairness question.
5. F. Human/judge evaluation protocol — instructions, agreement, biases.
6. G. Failure analysis — curated transcripts.
7. H. Compute — the cost question.

Within each, lead with a two-line summary of what the section shows — the reviewer

deciding whether to trust an unread appendix trusts one with visible structure. And

keep appendix numbering frozen after submission: the rebuttal will cite "App. C.2"

and a renumbered revision breaks every pointer in the discussion thread.

What not to include

• Anything load-bearing that appears only in the supplement — if the 9 pages

cannot make the case alone, restructure (colm-writing-style).

• Un-curated dumps (every transcript, every checkpoint) — volume without curation

reads as evasion.

• Data whose license or ToS forbids redistribution — describe it and state the

restriction instead (colm-artifact-evaluation).

• 待核实: no supplementary size cap was verifiable for 2026; check the OpenReview

form's actual limits rather than assuming.

Timing note for the 2026-style calendar

In 2026 the supplement traveled with the paper at the March 31 deadline — no

separate later upload was posted (待核实 whether future cycles add one). Plan

accordingly: the anonymization sweep and package build belong in deadline week's

T-3 slot (colm-submission), not in an imagined grace period. And because the

May 22 - June 8 rebuttal leans on appendix anchors, whatever you deposit in March

is the ammunition you will fight with in June — deposit for your future self, who

will be tired and have eighteen days.

Output format

[Placement audit] load-bearing content in supplement? none / move up: <items>
[Prompt appendix] verbatim + system prompts ▢   [Tried-ledger] ▢   [Contamination appendix] ▢
[Anonymity scan] clean / hits: <files>
[Pointer integrity] all anchors resolve / dangling: <refs>
[Package] built via clean export ▢  size: <n> MB (cap 待核实)