designing-experiments

Designing Experiments

Helps choose and specify a research design before data analysis starts. This skill owns study-design decisions: what is treated, what is compared, what outcome is measured, which assumptions are required, which validation or recovery experiment should follow a failed scientific experiment, and which design is defensible.

It does not fit causal models, estimate treatment effects, interpret fitted model output from existing data, or debug software/build failures.

Decision Framework

1. Control Group?

• Yes: Go to Step 2.
• No: Consider Interrupted Time Series (ITS).

1. Unit Structure?

• Single Treated Unit:
• With multiple controls: Synthetic Control (SC).
• No controls: ITS.
• Multiple Treated Units:
• With control group: Difference-in-Differences (DiD).

1. Time Structure?

• Panel Data (Multiple units over time): Required for DiD and SC.
• Time Series (Single unit over time): Required for ITS.

Method Quick Reference

• Difference-in-Differences (DiD): Compares trend changes between treated and control groups. Assumes Parallel Trends.
• Interrupted Time Series (ITS): Analyzes trend/level change for a single unit after intervention. Assumes Trend Continuity.
• Synthetic Control (SC): Constructs a synthetic counterfactual from weighted control units. Assumes Convex Hull (treated unit within range of controls).

Failed Experiment Recovery

When a scientific experiment or optimization plan produces weak or contradictory results, use the same design surface to:

• Separate implementation or measurement errors from design-assumption failures.
• Identify which assumption should be tested next.
• Define a minimal validation experiment before abandoning the approach.
• State the decision rule for continuing, revising, or stopping the line of work.