smr-simulation-studies

SMR Simulation Studies

Use this to build the Monte Carlo that an SMR reviewer will trust. At a methods journal the

simulation is not a formality — it is the primary evidence that the analytical properties hold in

finite samples and that the method beats real competitors. A weak or self-serving simulation sinks

otherwise sound papers.

Design the DGP space deliberately

Reviewers attack the data-generating process first. Specify it as a designed experiment, not a

convenient example:

• Factors and levels: sample size (and, for panels/networks, the relevant dimensions), the

parameter that controls the difficulty (effect size, dependence, missingness rate, sparsity), and

any nuisance complications. State why each level is realistic for sociological data.

• Coverage of the assumption boundary: include cells where your own assumptions fail, so the

paper shows the method's limits, not just its triumphs. SMR rewards honesty about breakdown.

• Calibration to the application: at least one DGP should be calibrated to the real dataset in

smr-empirical-illustration, so the simulation speaks to a setting readers care about.

• Replications and seeds: enough Monte Carlo replications for stable estimates of the metrics, with

seeds fixed and reported for reproducibility.

The competitor set (non-negotiable)

A simulation that compares the new method only to a naive baseline is the classic reject. Include:

• The current default practitioners actually use.
• The strongest existing alternative for the same problem (often from a neighboring discipline —

see smr-literature-positioning).

• Where relevant, an oracle / infeasible benchmark to show the gap your method closes.

If your method loses to a competitor in some cell, report it and explain when each method is

preferable — conditional recommendations are more credible than universal victory.

Metrics that match the claim

| Claim type | Report | Common SMR pitfall |

|---|---|---|

| Point estimation | bias, RMSE, relative efficiency | reporting bias but hiding variance |

| Inference / testing | empirical size, power, CI coverage and width | "performs well" with no coverage number |

| Selection / classification | accuracy + the costs of each error | accuracy only, ignoring imbalance |

| Computation | runtime, scaling, convergence rate | feasibility claim with no timing |

Coverage and size near the nominal level are the metrics SMR reviewers scrutinize most for inference

methods — report the actual numbers, not adjectives.

Presenting the study compactly

• Summarize the full grid in a table or a small-multiples figure; do not narrate every cell.
• Lead with the cell that makes the contribution's point (where the incumbent breaks and the method

holds), then show the boundary where the method itself degrades.

• Hand the exhibit design to smr-tables-figures so the grid is self-contained and readable in print.

Checklist

• [ ] The DGP is specified as a factorial design with realistic levels, each justified.
• [ ] Cells where the method's own assumptions fail are included.
• [ ] At least one DGP is calibrated to the empirical illustration's data.
• [ ] The competitor set includes the current default and the strongest alternative.
• [ ] Metrics match each claim (coverage/size for inference, bias+variance for estimation).
• [ ] Replication count and seeds are reported.
• [ ] Cells where the method loses are reported with a conditional recommendation.

Anti-patterns

• Strawman comparison: only a naive baseline, never the real competitor.
• Sunny-cell selection: showing only regimes that favor the method.
• Adjective metrics: "good size control" with no rejection rates.
• Cherry-picked n: one favorable sample size with no scaling pattern.
• Uncalibrated fantasy DGP: a design unrelated to any sociological data.
• Hidden seeds / replication count: results that cannot be reproduced.

Output format

[Simulation status] convincing / needs repair / not ready
[DGP factors] <factor : levels, with realism note>
[Competitor set] <default + strongest alternative (+ oracle)>
[Metrics] <bias/RMSE/coverage/size/power/runtime as claimed>
[Boundary cell] <where the method degrades and why that is honest>
[Next SMR skill] smr-empirical-illustration