icse-supplementary

ICSE Supplementary Material

ICSE's page model is stricter than most authors coming from ML venues expect,

and it inverts their packing instincts. The 2027 research-track format (read

2026-07-08): main text ≤ 10 pages **inclusive of all figures, tables, and

appendices**, plus up to 2 pages of references only. There is no unlimited

in-PDF appendix. Everything else travels as supplementary material — an

anonymized link in the paper or an upload via HotCRP — and must be assumed

optional reading.

The placement rule

One question decides placement: must a reviewer see it to score the paper?

| Content | Placement | Why |

|---|---|---|

| Anything a criterion score depends on: RQ answers, key tables, protocol essentials, the threat analysis | The 10 pages | Optional material cannot carry mandatory weight |

| Full prompt templates, extended proofs, extra result tables, per-subject breakdowns | Package, summarized in-body | Body states the finding + pointer; package holds the mass |

| Tool source, benchmark, scripts, raw outputs | Replication package | This is the verifiability substrate (icse-reproducibility) |

| Interview guides, codebooks, instruments | Package, named in the availability statement | Reviewers audit method through them when they choose to |

| Demo video, screenshots gallery | Package, only if genuinely clarifying | Never as a substitute for a precise in-body description |

The failure mode is load-bearing supplement: "due to space, the evaluation

methodology is described in the supplementary material" invites the reviewer

to score what the body shows — an unevidenced claim. Compress into the body

instead: a protocol table instead of two prose pages, a summary row instead

of the full matrix.

Summarize-and-point discipline

Every relocation leaves a residue in the body — the conclusion plus a precise

pointer:

> Patches for all 214 bugs, with per-bug generation logs, are in the package

> (results/patches/); Table 5 reports the aggregate. The full 41-item

> codebook is in codebook.pdf; Table 2 shows the 6 categories with ≥10%

> prevalence.

Precise pointers (directory, file, section of the package README) also pay

forward: they become the artifact-evaluation roadmap after acceptance.

Package layout reviewers can navigate in five minutes

replication-package/
├── README.md            # map: claim/table/figure -> command -> output
├── REQUIREMENTS.md      # env: versions, container digest, hardware, runtime
├── LICENSE              # open license even at review time (anonymized holder)
├── code/                # tool source, pinned dependencies
├── benchmark/           # subjects or fetch-scripts with SHAs + mining date
├── scripts/             # one entry point per paper table/figure:
│   ├── run_rq1.sh       #   names match the RQs, not internal jargon
│   └── make_table3.py
├── results/             # raw outputs the paper's numbers derive from
└── appendix.pdf         # extended tables/proofs that did not fit the body

The README's first section is a claims map: each headline claim, the

command that regenerates its evidence, expected runtime, and expected output.

A reviewer who checks one row and succeeds extends trust to the rest; one who

hits a broken path on the first try generalizes that, too.

Anonymity of the archive

The supplement must hold the same double-anonymous line as the paper:

git archive exports (no history), scrubbed notebook metadata, no

/home/username/ paths in configs or logs, no institutional hostnames, no

author names in LICENSE/setup.py/CITATION files, hosting through an

anonymizing service rather than a personal account. Scrub, then *re-run the

entry-point scripts* — anonymization that breaks paths converts your evidence

into a liability.

What must stay out of the package

Screening for exclusions is as important as packing inclusions: third-party

code or datasets whose licenses forbid redistribution (link with pinned

fetch-scripts instead); human-subjects raw data beyond what consent covers

— transcripts, recordings, un-aggregated survey rows; credentials, API

keys, and internal hostnames hiding in config files and CI logs; and

proprietary industrial code shown to you under NDA, even in fragments

quoted in comments. A replication package is a publication: it gets the

same legal and ethical review as the paper, and a leak in the supplement is

harder to retract than a sentence.

Size and dependency realism

No supplement size cap was verifiable for 2027 (待核实 on the live HotCRP

form), so engineer for reviewer patience instead: keep the download small

(fetch-scripts for large corpora, with checksums), keep the smoke test under

minutes on a laptop, and never require credentials, cluster schedulers, or

paid API keys for at least a reduced-scale verification path. Provide cached

outputs beside every expensive step so the analysis chain can be verified

without the compute.

Reverify each cycle

The 10+2 model, the HotCRP supplementary option, and the availability-

statement placement are 2027-cycle facts; page models have changed across

ICSE editions before. Confirm the current call before deciding any placement

that would be expensive to reverse in deadline week.

Output format

[Body/package split] each relocated item -> residue sentence + pointer present?
[Load-bearing check] any criterion-relevant content living only in the package? (must be none)
[Package audit] layout roles present; claims map covers every headline number
[Anonymity + runnability] scrub done; entry points re-run post-scrub
[Reviewer path] smoke test minutes, laptop-feasible, credential-free