empirical-playbook

Applied Micro Toolkit

Reference for applied micro research design: method selection, diagnostics, inference, pitfalls, reporting standards, and power analysis.

When to Use This Skill

Use when the user is:

• Choosing between empirical methods for a causal question
• Evaluating which identification strategy fits their data and setting
• Running standard diagnostic tests and unsure which ones apply
• Designing a study and needs to calculate statistical power
• Reviewing or critiquing an empirical strategy
• Preparing the "Empirical Strategy" section of a paper
• Downloading macroeconomic or cross-national data (see references/data-sources.md for FRED/World Bank API access)

Skip when:

• Implementation details for a specific method are needed (use causal-inference skill for IV, DiD, RDD, SC, matching)
• The task is structural estimation (use structural-modeling skill)
• The task is manuscript preparation or journal logistics (use submission-guide skill)
• The task is formal identification proof (use identification-proofs skill)
• The task is Bayesian model specification (use bayesian-estimation skill)

After selecting a method, the econometric-reviewer agent can review the implementation and the identification-critic agent can evaluate the identification argument.

Method Selection Decision Tree

Start with the fundamental question: What source of variation identifies the causal effect?

Step 1: What is your source of variation?

| Source of Variation | Method Family | Key Assumption |

|--------------------|---------------|----------------|

| Randomized assignment (with full compliance) | Experimental analysis (OLS on treatment indicator) | Random assignment |

| Randomized assignment (with imperfect compliance) | IV / 2SLS using random assignment as instrument | Exclusion restriction, monotonicity |

| Policy change at a sharp threshold | Sharp RDD | Continuity of potential outcomes at cutoff |

| Policy change at a threshold with imperfect compliance | Fuzzy RDD (= IV at the cutoff) | Continuity + monotonicity at cutoff |

| Policy change at a point in time, with affected and unaffected groups | Difference-in-differences | Parallel trends |

| Staggered policy adoption across units over time | Staggered DiD (Callaway-Sant'Anna, Sun-Abraham, etc.) | Parallel trends (conditional on group and time) |

| Rare event affecting a single unit, long pre-treatment data | Synthetic control | Pre-treatment fit implies post-treatment counterfactual |

| Exogenous shifter of treatment that does not affect outcome directly | IV / 2SLS / GMM | Exclusion restriction, relevance, monotonicity |

| Rich set of observables that plausibly captures all confounders | Matching, IPW, AIPW (selection on observables) | Conditional independence (no unobserved confounders) |

| No credible exogenous variation | Sensitivity analysis, bounds, partial identification | Depends on bounding assumptions |

Step 2: Refinements Within Method Families

Within DiD:

Is treatment timing staggered?
├── No → Classic 2x2 DiD (TWFE is fine)
└── Yes
    ├── Can treatment turn off (reversals)?
    │   ├── Yes → de Chaisemartin-D'Haultfoeuille (2020)
    │   └── No
    │       ├── Do you have never-treated units?
    │       │   ├── Yes → Callaway-Sant'Anna (2021) with never-treated controls
    │       │   └── No → Callaway-Sant'Anna with not-yet-treated controls
    │       │           or Sun-Abraham (2021)
    │       └── Are effects likely heterogeneous across cohorts?
    │           ├── Yes → Callaway-Sant'Anna or Sun-Abraham (NOT TWFE)
    │           └── No → TWFE is OK, but report Bacon decomposition

Within IV:

How many instruments for how many endogenous regressors?
├── Exactly identified (K instruments = K endogenous)
│   └── 2SLS (= IV = Wald estimator for single instrument)
├── Over-identified (K instruments > K endogenous)
│   ├── 2SLS (default)
│   ├── GMM (efficient, use if heteroskedasticity suspected)
│   └── LIML (less biased with weak instruments)
└── Under-identified (K instruments < K endogenous)
    └── Cannot identify all parameters — need more instruments or fewer endogenous regressors

Within RDD:

Does crossing the threshold guarantee treatment?
├── Yes → Sharp RDD
└── No → Fuzzy RDD
    └── Is the running variable continuous?
        ├── Yes → Standard rdrobust
        └── No (discrete / few mass points)
            └── Cattaneo-Idrobo-Titiunik (2019) discrete RD methods

Within Matching / Selection on Observables:

Is the selection-on-observables assumption plausible?
├── No → Need a different identification strategy
└── Yes
    ├── Do you need ATE or ATT?
    │   ├── ATE → IPW or AIPW
    │   └── ATT → Matching or IPW with ATT weights
    ├── Is the propensity score model well-specified?
    │   ├── Uncertain → Use AIPW (doubly robust)
    │   └── Confident → IPW or regression adjustment
    └── Many covariates or nonlinear confounding?
        ├── Yes → ML-based methods (causal forests, DML)
        └── No → Parametric PS model + AIPW

Standard Diagnostics by Method

Key diagnostics to run for each method family. For full reporting checklists and minimum standards, see references/reporting-standards.md.

| Method | Must-Run Diagnostics | Key Concern |

|--------|---------------------|-------------|

| IV / 2SLS | First-stage F (KP), reduced form, overid test | Weak instruments (F < 10), exclusion restriction |

| DiD (classic) | Pre-trend F-test, event study plot, raw means by group/period | Parallel trends violation |

| Staggered DiD | Bacon decomposition, Callaway-Sant'Anna group-time ATTs | Negative TWFE weights with heterogeneous effects |

| RDD | McCrary density test, covariate balance at cutoff, bandwidth sensitivity | Manipulation of running variable, extrapolation bias |

| Synthetic Control | Pre-fit RMSPE, permutation p-value, leave-one-out | Pre-period fit quality, donor pool sensitivity |

| Matching / AIPW | Overlap plots, Love plot (SMD before/after), Oster/Rosenbaum bounds | Lack of overlap, unobserved confounders |

| Structural | Convergence, identification rank condition, robustness to starting values | Global vs local optimum, identification failure |

For implementation details and diagnostic code by method, see the causal-inference skill.

Inference Frameworks

Clustering Decision Rule

1. Identify the level at which treatment is assigned → cluster at that level (minimum)
2. If there are within-cluster correlations beyond treatment (e.g., spatial), consider multi-way clustering
3. If the number of clusters is small (< 30–40), use wild cluster bootstrap (Cameron-Gelbach-Miller 2008)
4. If the number of clusters is very small (< 10), cluster-robust methods may not work at all — consider randomization inference or aggregate to the cluster level

| Mistake | Consequence | Fix |

|---------|------------|-----|

| Clustering too fine (individual when treatment is at state level) | SEs too small; over-rejection | Cluster at the level of treatment assignment |

| Few clusters (< 30–40) with standard cluster-robust SEs | Poor finite-sample properties | Wild cluster bootstrap |

| Not clustering when treatment varies at group level | SEs dramatically understated | Always cluster at level of treatment assignment |

Design-Based vs Model-Based Inference

| Dimension | Design-Based | Model-Based |

|-----------|-------------|-------------|

| Source of randomness | Treatment assignment mechanism | Outcome draws from a superpopulation |

| Key assumption | Known or modeled treatment assignment | Correct outcome model specification |

| Examples | Experiments, RCTs, RDD, DiD, natural experiments | Structural models, matching, cross-sectional surveys |

| Advantages | Transparent; does not require outcome model | More powerful; extends to complex settings |

Design-based is appropriate when the assignment mechanism is known (experiments, lotteries, cutoffs). Model-based when random sampling is reasonable. The standard in applied micro is hybrid: design-based identification + model-based inference. Doubly robust methods (AIPW) combine both.

Power Analysis

The key quantity is the Minimum Detectable Effect (MDE) — the smallest effect detectable with 80% power at alpha = 0.05.

Quick MDE formula (equal groups, two-sided test):

MDE = 2.8 × sigma / sqrt(N)

Required N = (2.8 × sigma / MDE)²

For IV designs, the effective MDE is inflated by the inverse of the first-stage coefficient: MDE_IV ≈ MDE_OLS / |pi|. A weak first stage (small pi) dramatically reduces power.

For DiD designs, effective power increases with more post-treatment periods and higher within-group correlation (absorbed by FEs). For RDD, use effective N (observations within bandwidth), not total N.

For cluster-randomized designs, the design effect (1 + (m-1) × ICC) inflates variance — with ICC = 0.05 and cluster size m = 50, you need 3.45x as many observations.

For full MDE formulas (DiD, IV, RDD, cluster-randomized), power simulation code, and MDE interpretation tables, see references/reporting-standards.md.

Research Design Checklist

Before Touching Data

• [ ] Research question: What causal parameter are you trying to estimate? Write it as a formal estimand.
• [ ] Identification strategy: What source of variation identifies the effect? Draw the DAG.
• [ ] Assumptions: List all identification assumptions explicitly. Which are testable?
• [ ] Threats: For each assumption, what is the most plausible violation? How would you detect it?
• [ ] Power: Given your expected sample size, what is the MDE? Is it policy-relevant?
• [ ] Pre-analysis plan: For prospective studies, register the plan before seeing outcomes.

During Analysis

• [ ] Data cleaning documented: Every sample restriction justified and recorded.
• [ ] Summary statistics: Know your data before running regressions.
• [ ] Main specification: Run the main spec first. Resist the urge to search for significance.
• [ ] Diagnostics: Run all standard diagnostics for your method (see table above).
• [ ] Robustness: Vary specification choices systematically.
• [ ] Magnitude interpretation: Can you explain the coefficient in plain language?

Before Submission

• [ ] All diagnostics reported: See method-specific standards in references/reporting-standards.md.
• [ ] Replication package: Code runs from raw data to all tables and figures.
• [ ] Seeds set: All random number generators seeded for reproducibility.
• [ ] Limitations discussed: What are the strongest objections? Address them in the paper.
• [ ] Literature positioned: Have you cited and compared to the 5 closest papers?

Common Pitfalls

Bad Controls

A "bad control" is a variable that is itself an outcome of treatment. Conditioning on it introduces selection bias.

| Variable Type | Example | Why It Is Bad |

|--------------|---------|---------------|

| Post-treatment outcome | Controlling for occupation when estimating returns to education | Education affects occupation; conditioning selects on an outcome of treatment |

| Mediator | Controlling for wages when estimating effect of training on employment | Blocks part of the causal effect |

| Collider | Conditioning on "survived" when estimating health effects | Opens a non-causal path |

Rule of thumb: If you cannot be sure a variable is determined before treatment, do not include it as a control. When in doubt, draw the DAG.

Staggered DiD with Heterogeneous Effects

| Mistake | Consequence | Fix |

|---------|------------|-----|

| Running TWFE with staggered timing | Already-treated units used as controls; negative weights; estimate can have wrong sign | Use Callaway-Sant'Anna, Sun-Abraham, or other modern DiD estimator |

| Using single post-treatment indicator for all cohorts | Masks heterogeneity in treatment effects across cohorts | Estimate group-time ATTs separately, then aggregate |

| Not reporting the Bacon decomposition | Reader cannot assess how much of the TWFE estimate comes from problematic comparisons | Report bacondecomp output |

Forbidden Regressions

Never plug a manual first-stage into an OLS second stage (SEs are wrong — use proper 2SLS). Never use a nonlinear first stage with linear second stage (not consistent — use control function). Never include generated regressors without bootstrapping the full two-step procedure.

Integration

For full minimum reporting standards (method-specific checklists for IV, DiD, RDD, SC, Matching) and complete power analysis code, see references/reporting-standards.md. For sensitivity analysis procedures (Oster bounds, Conley bounds, breakdown frontiers, specification curves), see references/sensitivity-analysis.md.

Agents:

• econometric-reviewer: Reviews identification strategy, standard errors, and diagnostic results
• identification-critic: Evaluates identification argument completeness and exclusion restrictions
• numerical-auditor: Designs power simulations for nonstandard study designs
• journal-referee: Reviews whether the empirical strategy meets journal standards

Cross-references:

• identification-proofs skill: Formalize an identification argument for the chosen method
• references/diagnostic-battery.md: Run the full diagnostic battery for the estimated specification
• references/sensitivity-analysis.md: Run sensitivity analysis (Oster bounds, specification curve, breakdown frontier)
• publication-output skill: Format regression tables and diagnostic output for publication