robustness

Robustness Test Checklist

Systematic checklist of robustness tests for empirical research. Use this to ensure comprehensive testing before submission.

---

Core Robustness Categories

1. Alternative Samples

| Test | Description | When to Use |

|------|-------------|-------------|

| Exclude outliers | Winsorize/trim at different levels (0.5%, 2%, 5%) | Always |

| Drop financial firms | Exclude SIC 6000-6999 | If not already excluded |

| Drop regulated industries | Exclude utilities, telecoms | Industry-specific effects |

| Different time periods | Split sample pre/post crisis, early/late | Results may be period-specific |

| Geographic subsamples | By region, state, country | External validity |

| Size subsamples | Small vs. large firms | Heterogeneous effects |

| Balanced panel | Require continuous observations | Survivorship concerns |

2. Alternative Specifications

| Test | Description | When to Use |

|------|-------------|-------------|

| Different fixed effects | Firm, industry×year, state×year | Control for unobservables |

| Additional controls | Add variables referees might suggest | Omitted variable concerns |

| Drop controls | Verify not over-controlling | Mediator concerns |

| Different clustering | Firm, industry, state, two-way | Inference robustness |

| Different standard errors | Bootstrap, Newey-West, Driscoll-Kraay | Serial/cross-sectional correlation |

| Nonlinear specifications | Quadratic terms, splines | Linearity assumption |

| Log vs. level | Transform dependent variable | Skewed distributions |

3. Alternative Measures

| Test | Description | When to Use |

|------|-------------|-------------|

| Alternative dependent variable | Different proxy for same concept | Measurement concerns |

| Alternative treatment measure | Continuous vs. binary, different threshold | Treatment definition |

| Alternative control measures | Different proxies for size, leverage, etc. | Standard practice |

| Scaled differently | By assets, sales, employees | Scaling choice matters |

4. Identification Tests

| Test | Description | When to Use |

|------|-------------|-------------|

| Placebo/Falsification | | |

| Placebo timing | Fake treatment 1-3 years before actual | DiD parallel trends |

| Placebo outcome | Effect on outcome that shouldn't be affected | Specificity of mechanism |

| Placebo treatment | Random assignment of treatment | Rule out spurious correlation |

| Pre-trends | | |

| Event study plot | Coefficient for each pre/post period | Visual parallel trends |

| Joint F-test | Test pre-period coefficients = 0 | Statistical parallel trends |

| Endogeneity | | |

| Instrumental variables | Find exogenous variation | Selection concerns |

| Heckman selection | Model selection explicitly | Sample selection |

| Propensity score matching | Match treated/control | Observable selection |

| Entropy balancing | Reweight to balance covariates | Covariate imbalance |

| Regression discontinuity | If threshold exists | Sharp identification |

5. Inference Robustness

| Test | Description | When to Use |

|------|-------------|-------------|

| Wild cluster bootstrap | Small number of clusters | <50 clusters |

| Randomization inference | Permutation-based p-values | Few treated units |

| Conley standard errors | Spatial correlation | Geographic data |

| Multiple hypothesis correction | Bonferroni, FDR | Many outcomes tested |

---

DiD-Specific Tests

For difference-in-differences designs:

• [ ] Event study with pre-treatment coefficients
• [ ] Parallel trends test (formal)
• [ ] Bacon decomposition (staggered treatment)
• [ ] Callaway-Sant'Anna or Sun-Abraham estimator (heterogeneous treatment effects)
• [ ] Placebo treatment timing
• [ ] Vary treatment window
• [ ] Triple-difference if possible
• [ ] Exclude always-treated or never-treated

---

IV-Specific Tests

For instrumental variables:

• [ ] First-stage F-statistic (>10, prefer >100)
• [ ] Weak instrument robust inference (Anderson-Rubin)
• [ ] Overidentification test (if multiple IVs)
• [ ] Exclusion restriction discussion
• [ ] Reduced form results
• [ ] Compare OLS vs. IV magnitudes

---

Quick Commands

• "robustness checklist" - Full checklist for current paper
• "DiD robustness" - DiD-specific tests only
• "what tests for [method]" - Tests for specific identification strategy
• "referee-proof" - Most commonly requested tests
• "prioritize tests" - Rank by importance for your setting

---

Referee-Proof Minimum

At minimum, most papers should include:

1. Sample robustness: Exclude outliers, alternative time periods
2. Specification robustness: Alternative fixed effects, controls
3. Measurement robustness: Alternative variable definitions
4. Inference robustness: Alternative clustering
5. Identification test: At least one placebo or pre-trend test

---

Reporting Template

For robustness tables:

Table X: Robustness Tests
Panel A: Alternative Samples
  (1) Baseline
  (2) Exclude financial firms
  (3) Exclude 2008-2009
  (4) Winsorize at 5%

Panel B: Alternative Specifications
  (5) Add industry×year FE
  (6) Control for firm age
  (7) Cluster by industry

Panel C: Alternative Measures
  (8) Alternative dependent variable
  (9) Continuous treatment measure