trustworthy-experiment-insights

Trustworthy Experiment Insights

Use this skill to decide whether an experiment result is believable enough to

shape a product or engineering decision. It focuses on false positives, false

negatives, power, replication, meta-analysis, stratified sampling, covariate

adjustment, and suspicious result review.

Source Traceability

Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is

transformed and paraphrased from Chapter 6 on false positives and negatives,

meta-analysis, metric sensitivity, stratified random sampling, covariate

adjustments, replication, longer runs, and statistical power.

Related skills:

• ab-test-results-readout for standard experiment reporting.
• experiment-sensitivity-optimization for improving precision before or

during experiment design.

• experiment-verification-monitoring for operational validity checks.

Reference Routing

| Need | Read |

|------|------|

| Insight-quality concepts | references/core/knowledge.md |

| Credibility and follow-up rules | references/core/rules.md |

| Result-review scenarios | references/core/examples.md |

| Step-by-step credibility review | workflows/review-experiment-credibility.md |

Workflow

1. Confirm the experiment was operationally valid enough to interpret.
2. Check power, practical significance, and whether metrics were underpowered.
3. Look for false positive risk: suspicious lift, many comparisons, early stop,

weak prior, or contradiction with prior experiments.

1. Look for false negative risk: noisy metrics, small sample, low sensitivity,

or over-broad metric choice.

1. Compare with similar experiments or run meta-analysis when available.
2. Recommend launch, replicate, extend, investigate, or reject the result.

Output Format

# Experiment Insight Credibility Review

## Result Under Review
[Experiment, metric, observed result, and proposed decision.]

## Credibility Assessment
[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]

## Evidence
| Check | Finding | Risk |
|-------|---------|------|

## Follow-Up
- Replication needed:
- Longer run needed:
- Meta-analysis/comparison:
- Variance reduction opportunity:

## Decision Guidance
[What decision can be made now, and what should wait.]

Quality Bar

• Do not celebrate a result before checking whether it could be a false positive.
• Do not dismiss a flat result before checking power and sensitivity.
• Do not compare against prior experiments without noting differences in

population, metric, design, and timing.

• Do not use statistical checks to hide operational failures; verify experiment

health first.