lang-data-analysis

Data Analysis (lang-data-analysis)

At Language the analysis exists to make the theoretical claim credible — not to display

technique or notation. A cross-subfield, double-anonymous reviewer will ask whether the evidence

actually warrants the generalization and whether uncertainty is handled honestly. Where the work is

quantitative, Language now expects properly specified models (typically mixed-effects models in

R) rather than by-subject t-tests or raw counts; where it is analytic, it expects the pattern to be

demonstrable from the glossed data. This skill stress-tests the analysis chain in the idiom of your work.

When to trigger

• Planning the analysis, or auditing it before writing up
• A reader doubts the statistics, the evidence-to-claim link, or the treatment of variability
• Reconciling multiple data sources (corpus + experiment, judgments + text) into one argument
• Deciding which analyses are confirmatory vs. exploratory

Analysis norms (by mode)

Quantitative (experiment / corpus)

• Fit mixed-effects models with the random-effects structure the design justifies (crossed

by-subject and by-item random effects; random slopes for within-cluster predictors). Report the

model, not just p-values.

• Report effect sizes and intervals, not stars alone; state the coding/contrasts and the

convergence status; keep seeds and pinned package versions.

• Distinguish preregistered/confirmatory from exploratory analyses where applicable.

Phonetic

• State measurement settings (windowing, formant ceilings, alignment) and how outliers/mis-tracks were

handled; show the effect is not an artifact of the measurement pipeline.

Analytic (formal)

• Demonstrate the generalization directly from numbered, glossed examples; show the analysis

derives the attested cases and blocks the unattested ones.

Historical / typological

• Make the inferential logic explicit (implicational universals, reconstruction, statistical

tendencies); guard against non-independence of the sample.

Convergent evidence (a Language strength)

Language rewards a generalization shown through more than one window — e.g., an experimental

effect corroborated by a corpus trend, or judgments backed by text frequencies. When windows disagree,

say so and explain the discrepancy rather than hiding the inconvenient one.

Referee-pushback patterns on the evidence chain (Language fixes)

| Referee writes… | The Language-specific fix |

|-----------------|---------------------------|

| "No random effects / pseudoreplication." | fit the justified mixed model; cluster by subject and item |

| "Significance without effect size." | report estimates + intervals in interpretable units |

| "The stat model doesn't match the design." | align random-effects structure with the sampling |

| "Analysis doesn't rule out the alternative." | show it derives attested and blocks unattested cases |

Calibration (Language appetite, hedged)

Orienting heuristics; confirm against the current author pages. Language increasingly expects that a

quantitative claim rests on a model appropriate to the clustered, repeated-measures nature of

linguistic data — the modal avoidable failure is pseudoreplication (ignoring by-speaker or by-item

structure). Illustrative: a paper claims a durational contrast "is significant (p < .01)" from 1,200

tokens produced by 8 speakers, analyzed as if independent. A referee writes "pseudoreplication." The fix

refits a mixed-effects model with by-speaker and by-word random intercepts and slopes, reports the

estimate (an illustrative 12 ms, 95% CI ~4–20), and notes two speakers who show no effect — turning a

fragile claim into a credible, bounded one.

Execution bridge (StatsPAI / Stata MCP)

Language asks for a properly specified model, which is a claim about a fitted object,

not about a paragraph. Fit it and report from it. Full map:

[execution-with-mcp](../../../shared-resources/empirical-methods/execution-with-mcp.md).

• Mixed-effects: mixed (continuous responses) and melogit / meglm (binary and

categorical ones) carry the crossed by-subject and by-item structure the design justifies;

icc states how much clustering there actually is, which is the number a reviewer needs

when the sample's non-independence is the objection.

• Uncertainty: bootstrap for intervals where the asymptotics are thin — small

fieldwork samples and unbalanced cells, both routine here.

• Multiple comparisons: holm or benjamini_hochberg across a family of contrasts.

A typological or corpus paper testing many predictors at once needs this stated, not

assumed.

• Reporting: etable / margins so the effect sizes and intervals in the prose are

the fitted ones.

Where a server is not connected, adapt the ../../resources/code/ skeleton and say so —

never report a number you did not compute. This bridge touches the quantitative strand

only; analytic and historical arguments are made from the glossed examples themselves.

Anti-patterns

• Treating repeated measures as independent (pseudoreplication); stars-only reporting
• A statistical model whose random structure ignores the sampling design
• Phonetic effects that are artifacts of measurement settings, not language
• Cherry-picked examples that ignore counterexamples in the same corpus/elicitation
• Presenting exploratory results as if confirmatory
• Notation or technique foregrounded over the generalization it is meant to support

Evidence pass for Language

Treat this skill as an executable review pass, not a prose hint. First lock the empirical

generalization, evidence base, warrant, and theoretical payoff; then judge whether the manuscript

answers the venue's real reader: linguists across subfields who value grounded analysis, transparent and

checkable evidence, and careful, appropriately scoped generalizations.

• Do the pass: audit the analysis before polishing prose — unit of analysis, random-effects

structure, effect sizes, measurement pipeline, exclusions, and reproducibility must be visible.

• Return a ledger: give claim / evidence / risk / manuscript location rows so the next agent can

edit rather than rediscover the issue.

• Sibling guard: compare against Laboratory Phonology, Journal of Memory and Language,

Language Variation and Change; if a sibling owns the contribution, recommend re-routing before

polishing.

• Stop condition: do not give submission-ready advice until resources/official-source-map.md has

been checked and the manuscript has one concrete fix for the largest venue-specific risk.

Output format

【Claim under test】from theory-building
【Primary evidence】the analysis that carries the claim
【Model】mixed-effects structure matches the design? [Y/N/NA]
【Uncertainty】effect sizes + intervals reported? [Y/N]
【Convergence】corroborated across windows? [Y/N/NA]
【Confirmatory vs. exploratory】labeled where relevant? [Y/N]
【Next】lang-data-and-transparency

Supplementary resources

• [../../resources/external_tools.md](../../resources/external_tools.md) — R / lme4 / brms, Praat, corpus tooling
• [../../resources/official-source-map.md](../../resources/official-source-map.md) — Language quantitative and evidence expectations