cogpsych-data-analysis

Data Analysis & Model Fitting (cogpsych-data-analysis)

Cognitive Psychology holds analyses to a model-based standard: fit the formal model, **compare it

to rivals with principled criteria, demonstrate that parameters and models are recoverable**, use

mixed models or hierarchical Bayesian estimation where the design demands it, and report **effect

sizes with uncertainty** for behavioral results — all regenerable from deposited code. This is the

experiment-to-model-fit loop that defines the venue.

When to trigger

• Fitting the formal model and comparing it to rival accounts
• Running the behavioral analyses (mixed models, hierarchical Bayesian, contrasts)
• A reviewer asked for model comparison, recovery, robustness, or fuller disclosure
• Preparing analysis/model code and a data dictionary for deposit

Reporting norms Cognitive Psychology expects

1. Fit and compare models, don't just fit one. Report fit for your model and the rival(s) under

matched flexibility; compare with AIC/BIC, cross-validation, or Bayes factors as

appropriate, and say what the comparison licenses.

1. Show recovery. Demonstrate parameter recovery (can the fitting procedure recover known

parameters from simulated data) and model recovery (does the comparison criterion pick the

generating model) — without these, a fit edge is not interpretable.

1. Use the right hierarchical structure. Crossed random effects over subjects and items call for

(generalized) linear mixed models; for cognitive models, hierarchical Bayesian estimation

pools strength across participants. Justify the structure; don't aggregate away the variance.

1. Effect sizes + uncertainty for behavior. Report standardized/unstandardized effect sizes with

confidence/credible intervals for key behavioral results, not just p-values and stars.

1. Confirmatory vs. exploratory. Separate pre-committed model comparisons and tests from

exploratory model exploration; do not present a post hoc winning model as predicted.

1. Reproducible. Model and analysis code, with seeds and pinned versions, regenerate every reported

fit, figure, and table in a fresh session (see cogpsych-open-science-and-transparency).

Robustness

• Show the conclusion survives reasonable alternative model specifications, priors (for Bayesian fits),

and exclusion choices; report sensitivity, not a single fragile fit. Report convergence diagnostics

(e.g., R-hat, ESS) for Bayesian models.

Worked micro-example (illustrative numbers)

A preregistered three-experiment recognition-memory program fitting UVSD vs. DPSD to confidence-ROC data.

Model comparison (preregistered) — pooled across Exps 1-3
  Fit (hierarchical Bayesian, matched flexibility):
    UVSD favored: dBIC = 14 vs. DPSD; Bayes factor ~ 30 in favor of UVSD
  Recovery (required): parameter recovery good (recovered d', sigma within
    credible intervals); model recovery ~ 92% correct at the design's N/trials
  Diagnostic signature: z-ROC slope 0.78, 95% CrI [0.72, 0.84], and linear
    (no reliable curvature) — the qualitative pattern UVSD predicts and DPSD
    forbids, consistent across all three experiments
Behavioral effect (mixed model)
  List-strength manipulation on d': b = 0.31, 95% CI [0.18, 0.44]
Exploratory (labeled)
  A small response-bias drift surfaced post hoc; reported as exploratory

Why this passes Cognitive Psychology scrutiny: the model is compared (not just fit), recovery

makes the comparison interpretable, the qualitative signature corroborates the fit index, hierarchy

respects subject/item variance, and the exploratory drift is honestly demoted.

Analysis-stage reviewer pushback and the venue fix

| Reviewer pushback | What it signals here | Cognitive Psychology fix |

|-------------------|----------------------|--------------------------|

| "You only fit your model" | one-model storytelling | fit the rival under matched flexibility; report AIC/BIC/BF and what it licenses |

| "Better fit may be overfitting" | flexibility imbalance | add model recovery + cross-validation; penalize complexity |

| "Can you recover these parameters?" | identifiability doubt | run and report parameter + model recovery simulations |

| "Aggregated means hide variance" | wrong error structure | refit with crossed-random-effects mixed model / hierarchical Bayesian |

| "Is this the model you predicted?" | post hoc selection | pre-commit the comparison; relabel post hoc fits exploratory |

| "I can't rerun your fits" | reproducibility gate | ship seeded model code + a fresh-session run log |

Calibration anchors

• A model that is fit, compared, and recovered is the unit of evidence here — a single fit with a

good index but no rival and no recovery is not persuasive.

• Trust a crossed qualitative prediction over a marginal fit advantage; report both and lead with

the signature the rival forbids.

• Respect the data's hierarchy: aggregating over subjects or items inflates false positives and can

bias parameter estimates; use mixed/hierarchical models and justify the random-effects structure.

• For Bayesian fits, report priors, convergence, and sensitivity — a fit without diagnostics is not

reproducible evidence.

Execution bridge (StatsPAI / Stata MCP)

Run the battery, don't just enumerate it. Full map:

[execution-with-mcp](../../../shared-resources/empirical-methods/execution-with-mcp.md). Cognitive Psychology is experimental — within-subject designs and mixed models dominate; report the model, the effect size, and multiple-comparison control.

• Many outcomes / specifications: romano_wolf (step-down FWER) or

benjamini_hochberg — report the adjusted threshold.

• OVB sensitivity: oster_delta / sensemakr.
• Inference: wild_cluster_bootstrap (few clusters), twoway_cluster / conley;

multilevel data → cluster at the right level.

• Re-fit off one handle: audit_result(result_id) lists the missing checks and the

exact suggest_function for each.

• Exhibits: etable / did_summary_to_latex from the handle — no retyped numbers.

Keep the decisive checks in the body and the exhaustive battery in the supplement. See

the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).

Anti-patterns

• Fitting only your model with no rival and no comparison criterion
• Claiming a fit advantage without matched flexibility, recovery, or cross-validation
• Aggregating to cell means and ignoring crossed subject/item variance
• p-values and stars with no effect size or interval for behavioral results
• Presenting a post hoc winning model as a predicted result
• Model/analysis code that does not regenerate the reported fits

Output format

【Model comparison】rivals fit under matched flexibility + criterion (AIC/BIC/BF)? [Y/N]
【Recovery】parameter + model recovery reported? [Y/N]
【Hierarchy】mixed model / hierarchical Bayesian where apt + diagnostics? [Y/N]
【Behavioral effects】effect sizes + intervals? [Y/N]
【Confirmatory vs exploratory】separated? [Y/N]
【Reproducible】seeded code + data dictionary + fresh-session check? [Y/N]
【Next】cogpsych-tables-figures

Supplementary resources

• [../../resources/external_tools.md](../../resources/external_tools.md) — modeling, model-comparison, lme4/brms/Stan, JAGS, recovery simulation
• [../../resources/official-source-map.md](../../resources/official-source-map.md) — statistical and modeling expectations