experiment-type-selection

Experiment Type Selection

Use this skill when the experiment question determines the test type. Not every

experiment should be a simple superiority test; some decisions need evidence

that a change is not worse, roughly equivalent, or durable over time.

Source Traceability

Primary source: Practical A/B Testing by Leemay Nassery. Guidance is

transformed and paraphrased from chapter 3, especially lines 2013-2870. Related

variant design context comes from chapter 1 lines 539-571 and chapter 2 lines

1564-1735.

Related Advanced Skills

• experimentation-throughput-strategy: use when the choice is isolated versus

overlapping testing or when testing availability constrains the design.

• adaptive-experimentation-strategy: use when fixed-horizon A/B testing may be

replaced by sequential testing, bandits, or contextual bandits.

• ml-experiment-evaluation: use when the experiment is evaluating ML models,

rankers, offline metrics, interleaving, or model filtering.

• long-term-impact-evaluation: use when the test type question is really

about delayed or sustained impact measurement.

Reference Routing

| Need | Read |

|------|------|

| Test type concepts | references/core/knowledge.md |

| Selection rules | references/core/rules.md |

| Scenario examples | references/core/examples.md |

| Step-by-step selection | workflows/choose-experiment-type.md |

Workflow

1. State the decision question in plain language.
2. Identify whether the team wants to prove improvement, avoid degradation, or

show practical similarity.

1. Check whether the metric movement must persist after launch.
2. Decide whether multiple variants are necessary and interpretable.
3. Choose the simplest test type that answers the decision question.
4. Document assumptions, risk, and follow-up analysis.

Output Format

# Experiment Type Recommendation

## Decision Question
[What the team needs to learn.]

## Recommended Type
[Superiority | Non-inferiority | Equivalence | A/B/n | Holdback]

## Why This Type Fits
- Goal:
- Metric behavior needed:
- Risk tolerance:
- Time horizon:

## Design Notes
- Primary metric:
- Guardrails:
- Variants:
- Population:
- Follow-up analysis:

## Do Not Use
[Types that would answer the wrong question and why.]

Quality Bar

• Do not default to superiority when the real question is safety or sameness.
• Do not use equivalence unless the team can define an acceptable equivalence

band.

• Do not recommend many variants unless the user has traffic and the variants

preserve interpretable learning.

• Use holdback-experiment-design for detailed long-term holdback planning.