interspeech-experiments

INTERSPEECH Experiments

"Not convincing" is the standard Interspeech rejection, and it almost always means

the experimental design — not the idea — failed. Speech evaluation has decades of

conventions per task; an experiment section that ignores them is illegible to the

reviewer pool regardless of how good the numbers are.

Metric-task law

| Task family | Primary metrics | Convention notes |

|---|---|---|

| ASR | WER / CER | normalization + scorer disclosed; CER for unsegmented scripts |

| TTS / VC | MOS, CMOS (+ objective proxies) | panel protocol reported; CMOS for close systems |

| Speaker verification | EER, minDCF | official trial lists; DCF prior/costs stated |

| Diarization | DER / JER | collar and overlap handling stated |

| Enhancement / separation | PESQ, STOI/ESTOI, SI-SDR (+ DNSMOS-style proxies) | wideband vs narrowband named |

| SLU / speech translation | intent acc / F1, BLEU/COMET on ASR output | cascaded vs end-to-end made explicit |

| Paralinguistics / health | UAR, F1 | speaker-disjoint splits are mandatory |

Using a proxy where the community expects the primary (e.g., only neural MOS

predictors for a TTS claim) needs an explicit defense sentence.

Baselines that count

• The stock recipe of a public toolkit on the same corpus (an ESPnet/

SpeechBrain recipe number is a shared, checkable baseline).

• The latest challenge baseline if your task has a running challenge —

reviewers know those numbers by heart.

• Your method's ablated self — at Interspeech, one clean ablation of the

single proposed component often persuades more than two extra datasets.

• Reimplemented prior work must be validated: show your reimplementation matches

its published number before showing you beat it.

Significance: over what randomness?

State which variation your statistics cover — the two are routinely conflated:

• Test-set variation: bootstrap over utterances (or speakers, if claims are

speaker-level) → CI on the metric difference between systems.

• Training variation: multiple seeds → mean ± sd; a 0.2 WER gain with sd 0.3

across seeds is not a result.

• Matched-pairs tests (paired bootstrap; the classic MAPSSWE-style segment test

for ASR) for A-vs-B claims on the same test set.

• Subjective scores: CIs over raters and stimuli; ±0.1 MOS is panel noise

under most protocols (see interspeech-reproducibility).

Condition coverage — the speech-specific axis

A speech claim is implicitly quantified over speakers, acoustic conditions, and

often languages. Reviewers probe the quantifier:

• Speaker-disjointness: train/test speaker overlap invalidates verification,

paralinguistic, and health claims outright.

• Condition breakdown: report clean vs noisy, near- vs far-field, read vs

spontaneous where the corpus offers them — an average hides the regression your

method causes in one condition.

• Language scope: "multilingual" needs a per-language table; English-only

results support English-only claims.

• Domain leakage: SSL pretraining data overlapping the test corpus (the

LibriSpeech-descendant problem) must be checked and stated for foundation-model

work.

Data hygiene

• Official partitions only, or published manifests for custom splits.
• No tuning on test: LM weights, thresholds, and checkpoint selection all happen

on dev — say so in one line.

• License and consent status of every corpus stated (see

interspeech-artifact-evaluation); leaked or scraped audio can sink an

otherwise strong paper on ethics review.

Designing inside 4 pages

Budget roughly one column for the decisive comparison, half for ablation, half for

analysis. The analysis half is what separates accepted Interspeech papers: one

error-pattern finding (where the gains live — short utterances, overlapping

speech, a phone class) converts a benchmark delta into a scientific statement.

Worked micro-example: is 4.9 vs 5.6 WER real?

Claim: proposed 4.9% vs baseline 5.6% WER on test-other (2939 utts).
1. Paired per-utterance errors → paired bootstrap, 1000 resamples.
2. Δ WER 95% CI: [-0.9, -0.5] — excludes 0 → test-set variation covered.
3. Across 3 seeds: 4.9, 5.0, 4.8 (sd 0.1) vs 5.6, 5.7, 5.6 (sd 0.06)
   → training variation does not swallow the gap.
4. Report: "−0.7 abs. WER (95% CI [−0.9, −0.5], paired bootstrap;
   consistent across 3 seeds)."

Two randomness sources, two checks, one sentence in the paper. If step 2's CI

had straddled zero, the honest paper reports the trend and softens the verb —

and usually survives review better than the inflated version.

Negative results and regressions

Interspeech's mixed jury respects a disclosed regression far more than a

suspicious clean sweep. If the method loses on clean speech while winning on

noisy, print both numbers and make the trade-off the story — condition-dependent

behavior is a finding in a field about acoustic variability, and hiding it is

the reviewer-trust equivalent of a failed significance test.

Pre-submission experiment audit

[ ] Primary metric matches task convention; ruler disclosed
[ ] Baseline set includes a public-recipe or challenge anchor
[ ] Each A>B claim carries a CI or matched-pairs test
[ ] Seeds: n stated; variance reported or single-run admitted
[ ] Speaker-disjoint splits verified where required
[ ] Condition/language breakdown present; regressions named
[ ] Dev-only tuning stated; test touched once
[ ] One analysis finding, not just deltas

Output format

[Claim inventory] each claim → metric → evidence status
[Metric-law check] conventions met / violations
[Baseline verdict] anchored / self-referential / stale
[Statistics] randomness covered (test-set / seeds / raters) per claim
[Coverage gaps] speaker / condition / language / leakage
[Cheapest decisive fix] <one experiment that most raises conviction>

Metric conventions are community law rather than CFP text and move slowly, but

challenge editions and recipe baselines roll every year — re-anchor at design

time (sources logged in resources/official-source-map.md, 2026-07-08).