joap-data-analysis

Data Analysis (joap-data-analysis)

JAP analyses must be model-appropriate, fully reported, and reproducible. The house toolkit is

SEM/CFA, multilevel (HLM) models, mediation and moderation with proper inference, and

meta-analysis. The journal expects effect sizes with confidence intervals, fit indices,

bootstrap CIs for indirect effects, full disclosure of how data were handled, and a clean

confirmatory vs. exploratory separation, with data and code shareable under TOP.

When to trigger

• Specifying and reporting the main and supporting analyses
• A reviewer asked for fit indices, indirect-effect CIs, robustness, or disclosure
• Reconciling preregistered analyses with exploratory follow-ups
• Preparing analysis scripts and a data/codebook for deposit

Reporting norms JAP expects

1. Measurement before structure. Report the measurement model (CFA fit: χ²/df, CFI, TLI, RMSEA,

SRMR) and reliability/AVE before interpreting the structural model; report measurement invariance

when comparing groups or waves.

1. Effect sizes + uncertainty. Give standardized and/or unstandardized estimates **with confidence

intervals** for key paths — not just p-values and stars.

1. Mediation done right. Report the indirect effect with a bootstrap (or Monte Carlo) CI (not

only the Baron-Kenny steps or a Sobel z); for multilevel mediation use the appropriate (e.g.,

1-1-1, 2-1-1, 2-2-1) decomposition and within/between separation.

1. Multilevel correctly. Report ICC(1)/ICC(2), center predictors appropriately (group-mean vs

grand-mean), model random effects, and justify the estimator; do not run OLS on nested data.

1. Full disclosure. Report how sample size was determined, all exclusions (careless responding,

attrition) and reasons, all conditions, and all measures. Label confirmatory vs. exploratory.

1. Meta-analysis (if applicable). Report the model (random vs fixed), heterogeneity (Q, I²),

artifact corrections, publication-bias checks, and a transparent coding protocol.

1. Reproducibility. Provide scripts and a codebook; results should regenerate from shared data in a

fresh session (see joap-open-science-and-transparency).

Worked micro-example (illustrative numbers)

The servant-leadership package: a multilevel mediation in the field and a causal test in the lab.

Measurement model (field, N = 612 in 74 teams)
  CFA: χ²/df = 2.1, CFI = .96, TLI = .95, RMSEA = .045, SRMR = .04
  Reliabilities ω: leadership .91, safety .88, performance .87
  Aggregation: ICC(1) = .16, ICC(2) = .77, r_wg(j) = .85 → team-level OK
Confirmatory (preregistered) — multilevel mediation (2-2-2)
  a (leadership→safety) = .42 [.27, .57]; b (safety→performance) = .31 [.14, .48]
  Indirect = .13, 95% Monte Carlo CI [.05, .23] → mediation supported
  Direct (leadership→performance | safety) = .09 [-.06, .24], ns
Confirmatory (preregistered) — lab experiment (causal leg)
  Servant vs control on safety: d = 0.46, 95% CI [0.21, 0.71]
  H3 boundary: interaction with interdependence, ΔR² = .03, CI excludes 0
Exploratory (labeled): voice as a serial L1 mediator surfaced post hoc;
  reported as exploratory, flagged for confirmation in future work.

Why this passes JAP scrutiny: the measurement model is reported before structure; the indirect effect

carries a bootstrap/Monte Carlo CI; nesting is modeled and aggregation justified; the experimental leg

supplies causal warrant; and the post hoc serial path is honestly demoted to exploratory.

Analysis-stage reviewer pushback and the venue fix

| Reviewer pushback | What it signals | JAP fix |

|-------------------|-----------------|---------|

| "Mediation by Sobel/steps only" | outdated inference | report indirect effect + bootstrap/Monte Carlo CI |

| "OLS on nested data" | dependence ignored | multilevel model; report ICC, centering, random effects |

| "No fit indices / measurement model" | construct validity unchecked | report CFA fit + reliability before structure |

| "Which exclusions were preregistered?" | forking-paths concern | disclosure table: rule, count, preregistered vs post hoc, result with/without |

| "Is this confirmatory?" | HARKing concern | point to the preregistration; relabel post hoc analyses exploratory |

| "Reviewer 2 couldn't rerun your code" | reproducibility gate | ship a fresh-session run log (see open-science skill) |

Calibration anchors

• Report measurement before structure: a beautiful path model on a misfitting measurement model

convinces no JAP reviewer.

• Prefer estimation language with CIs to dichotomous "significant/not"; bare p-value sentences read as

pre-reform.

• For mediation and cross-level effects, the interval on the carrying effect is the result — design

and report so that interval is defensible.

Execution bridge (StatsPAI / Stata MCP)

Run the battery, don't just enumerate it. Full map:

[execution-with-mcp](../../../shared-resources/empirical-methods/execution-with-mcp.md). JAP is organizational psychology — multilevel survey/field data and experiments; cluster at the right level and apply mediation/moderation discipline.

• Many outcomes / specifications: romano_wolf (step-down FWER) or

benjamini_hochberg — report the adjusted threshold.

• OVB sensitivity: oster_delta / sensemakr.
• Inference: wild_cluster_bootstrap (few clusters), twoway_cluster / conley;

multilevel data → cluster at the right level.

• Re-fit off one handle: audit_result(result_id) lists the missing checks and the

exact suggest_function for each.

• Exhibits: etable / did_summary_to_latex from the handle — no retyped numbers.

Keep the decisive checks in the body and the exhaustive battery in the supplement. See

the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).

Anti-patterns

• p-values and stars with no effect size, CI, or fit indices
• Mediation claimed via Baron-Kenny steps or Sobel z without a bootstrap/Monte Carlo CI
• Ignoring nesting (OLS on multilevel data) or aggregating without ICC/r_wg justification
• Selectively reporting conditions, measures, or exclusions (undisclosed flexibility)
• HARKing exploratory findings into confirmatory hypotheses
• Code that does not reproduce the reported numbers

Output format

【Measurement model】CFA fit + reliability + invariance reported? [Y/N]
【Main result】effect size(s) + CI(s) + meaning
【Mediation/multilevel】indirect-effect bootstrap/Monte Carlo CI; nesting modeled? [Y/N/NA]
【Disclosure】N-determination + all exclusions + all conditions + all measures? [Y/N]
【Confirmatory vs exploratory】clearly separated? [Y/N]
【Reproducible】scripts + codebook + fresh-session check? [Y/N]
【Next】joap-tables-figures

Supplementary resources

• [../../resources/external_tools.md](../../resources/external_tools.md) — Mplus, lavaan, lme4/nlme, psych, metafor/metaSEM, bootstrap/Monte Carlo tools
• [../../resources/official-source-map.md](../../resources/official-source-map.md) — statistical and disclosure requirements