jop-data-analysis

Data Analysis (jop-data-analysis)

At JOP, analysis and reproducibility are the same task: acceptance is **contingent on

replicability, and a JOP replication analyst** re-runs your code at conditional acceptance. Write the

analysis so that every number in the paper is regenerated by a script — and reported with honest

uncertainty within the page budget.

When to trigger

• Setting up the estimation/analysis pipeline
• Deciding which robustness checks belong in the main text vs the Online Appendix
• A reviewer asked for additional specifications, uncertainty, or sensitivity
• Preparing numbers that must match the deposited replication package exactly

Analysis norms

• Report uncertainty, not just point estimates: CIs, SEs (clustered appropriately), and

substantive effect sizes a general reader can interpret.

• Specification transparency: show the primary specification clearly; relegate the grid of

alternatives to the Online Appendix, but reference it.

• Robustness that targets the threat: each check should answer a specific objection (confounding,

functional form, sample, measurement), not pad the count.

• Multiple comparisons: adjust or pre-specify when testing many implications.
• Substantive interpretation: translate coefficients into quantities of interest (predicted

probabilities, marginal effects) — general-interest readers want magnitudes, not just stars.

Reproducible-from-line-one (the JOP analyst will re-run this)

• One master script runs everything in order and sets the working directory once.
• Set a seed for every stochastic step (bootstrap, simulation, MCMC, jitter, sampling).
• Record software and package versions for the readme (e.g., "R 4.3.1", "Stata/MP 18.0").
• Build a codebook naming and defining every variable used in the analysis.
• Tables and figures are generated by code, never hand-edited — numbers in the text must match.

Fit the analysis to the page budget

• Lead with the result that carries the argument; do not narrate every regression.
• Move the robustness grid, balance tables, and diagnostics to the Online Appendix (≤ 25 pp).
• A Short Article (≤ 10 pp) should show one clean, decisive analysis, not a buffet.

Execution bridge (StatsPAI / Stata MCP)

Run the battery, don't just enumerate it. Full map:

[execution-with-mcp](../../../shared-resources/empirical-methods/execution-with-mcp.md). Journal of Politics spans observational and experimental political science; report the identifying assumption and the magnitude, not just stars.

• Many outcomes / specifications: romano_wolf (step-down FWER) or

benjamini_hochberg — report the adjusted threshold.

• OVB sensitivity: oster_delta / sensemakr.
• Inference: wild_cluster_bootstrap (few clusters), twoway_cluster / conley;

multilevel data → cluster at the right level.

• Re-fit off one handle: audit_result(result_id) lists the missing checks and the

exact suggest_function for each.

• Exhibits: etable / did_summary_to_latex from the handle — no retyped numbers.

Keep the decisive checks in the body and the exhaustive battery in the supplement. See

the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).

Anti-patterns

• Numbers in the manuscript that the deposited code cannot reproduce (fails the analyst check)
• Unseeded randomness or unpinned versions ("works on my machine")
• Star-gazing with no effect sizes or uncertainty a general reader can use
• Robustness checks chosen to inflate the count rather than rebut a threat
• Cramming every specification into the main text and blowing the page budget

What a JOP analysis referee is looking for

The reviewer pool spans subfields, so an analysis only a specialist can audit reads as fragile. Map each

demand to the move that satisfies it before the page count forces an ugly cut.

| Referee demand | Pass move | Fail signal |

|----------------|-----------|-------------|

| Usable magnitude | Marginal effect or predicted probability with CI | Coefficient stars, no magnitude in prose |

| Correct uncertainty | Cluster at assignment level; randomization inference | Default SEs on clustered or experimental data |

| Targeted robustness | Each check named to the threat it rebuts | A grid with no mapping to objections |

| Multiplicity honesty | Pre-specified families; adjusted p-values | One mined "significant" interaction |

| Reproducibility | Master script regenerates every number | "Available on request"; drifting numbers |

Worked micro-example (illustrative figures)

A hypothetical Short Article asks whether a state's adoption of automatic voter registration (AVR) raised

turnout, using a staggered difference-in-differences across states. The first pass runs naive two-way

fixed effects and reports a +3.1-point effect (illustrative). Because adoption is staggered, already-treated

states act as forbidden controls and the estimate carries negative-weight comparisons. The JOP-credible

re-analysis uses a heterogeneity-robust estimator (Callaway–Sant'Anna or Sun–Abraham), reports the

group-time average as +1.8 points, 95% CI [0.4, 3.2] (illustrative), shows flat pre-trends, and clusters

by state. The robustness grid goes to the Online Appendix, cited in one line of main text.

Referee pushback patterns and the JOP fix

• "Your DID uses naive TWFE on staggered adoption." Re-estimate with a heterogeneity-robust estimator,

show the event-study plot, and decompose the two-way estimate so the negative-weight problem is resolved.

• "Standard errors do not reflect the design." Cluster at the assignment level — the state in the AVR

example — with wild-cluster bootstrap when states are few.

• "This interaction looks fished." Show the pre-registered family and the adjusted p-value; concede a

null openly.

Output format

【Primary result】estimand + magnitude + uncertainty
【Robustness】each check ↔ the threat it answers (main vs appendix)
【Reproducible】master script + seeds + pinned versions + codebook? [Y/N]
【Numbers match】text == deposited output? [Y/N]
【Page discipline】main text lean, overflow in appendix? [Y/N]
【Next】jop-tables-figures

Supplementary resources

• [../../resources/external_tools.md](../../resources/external_tools.md) — estimation packages and reproducibility tooling (renv, seeds, version pinning)
• [../../resources/official-source-map.md](../../resources/official-source-map.md) — JOP replicability-contingent acceptance and replication-analyst check