experiment-readout

Experiment Readout Skill

A test result is only a decision if the statistics are sound — and "variant looks higher" is not a

result. This skill computes the lift, the p-value, and a confidence interval from the raw counts, checks

the guardrails, and writes an honest readout with a clear ship/no-ship call — flagging the traps

(peeking, underpowered, novelty, a significant but tiny effect) that make teams ship noise.

Required Inputs

Ask for these only if they aren't already provided:

• The metric & data — for a conversion test: users and conversions per variant (control vs. treatment). For a continuous metric: mean, SD, and n per variant.
• The hypothesis — what you expected and the minimum effect that matters.
• Guardrail metrics — what shouldn't get worse (revenue, latency, retention).
• Test setup — planned sample size/duration, and whether it ran to plan (for the peeking check).

Output Format

Experiment Readout: [test name]

1. Result — computed (use the helper): control vs. treatment rate, absolute & relative lift, p-value, and the confidence interval on the difference.

| Variant | N | Conversions | Rate |

|---|---|---|---|

| Control | | | |

| Treatment | | | |

→ Lift: X% (CI: [a%, b%]) · p = 0.0xx

2. Verdict — significant at the stated bar or not, and whether the effect is big enough to matter (a significant +0.2% may not be worth the complexity). Distinguish statistical from practical significance.

3. Guardrails — did anything you promised not to harm move? A win that tanks a guardrail isn't a win.

4. Validity checks — was it run to the planned sample (no peeking/early-stopping)? Sample-ratio mismatch? Novelty/seasonality? Call out anything that undermines the result.

5. Recommendation — ship / no-ship / iterate / re-run, with the reason. If inconclusive, say so — "no significant difference" is a valid, useful result, not a failure to spin.

Programmatic Helper

scripts/ab_significance.py (stdlib only) computes the two-proportion z-test, p-value, lift, and CI:

# python3 ab_significance.py <control_n> <control_conv> <treat_n> <treat_conv>
python3 scripts/ab_significance.py 10000 800 10000 880
python3 scripts/ab_significance.py 10000 800 10000 880 --json

Quality Checks

• [ ] Lift, p-value, and a confidence interval are computed (not just "higher")
• [ ] Statistical significance AND practical significance are both assessed
• [ ] Guardrail metrics are checked, not just the primary
• [ ] Validity is checked: ran to planned n, no peeking, no sample-ratio mismatch
• [ ] An inconclusive result is reported honestly, not spun into a win
• [ ] The recommendation is explicit (ship/no-ship/iterate/re-run)

Anti-Patterns

• [ ] Do not call significance by eye — compute the p-value and CI; a higher number isn't a result
• [ ] Do not ignore the confidence interval — a CI spanning zero (or huge) means you don't actually know the effect
• [ ] Do not confuse statistical with practical significance — a tiny significant lift may not be worth shipping
• [ ] Do not trust a peeked/early-stopped test — stopping when it looks good inflates false positives massively
• [ ] Do not spin a null result — "no detectable difference" is honest and often the right call

Based On

Frequentist A/B analysis — two-proportion z-test, confidence intervals, guardrails, and the peeking/practical-significance pitfalls.