experiment

<skill id="experiment">

<purpose>

Rules without evidence are superstitions. One invocation, no subcommands: the engine

figures out what the hypothesis system needs — seeds, tests, graduates, or kills.

• No hypotheses? Seeds them from CLAUDE.md.
• Hypotheses exist? Picks the most uncertain, designs an experiment, runs it.
• Evidence accumulating? Graduates proven rules, kills disproven ones.
• Everything tested? Reports and stops.

Rules that survive become convictions. Rules that fail become learnings.

</purpose>

<reference>

Deep patterns, SQL schema, domain design templates, confidence-scoring examples:

skills/experiment/reference/experiment-research.md

</reference>

<skill_load>

always: skills/quality/SKILL.md, skills/build/reference/testing.md

</skill_load>

<on_start>

agentdb read-start
agentdb emit command "experiment-start" "" '{}'

</on_start>

<cycle id="experiment" max_iterations="20">

<phase id="sense" name="SENSE — Read the State">

Autonomous entry point. Determine what the system needs.

    agentdb hypothesis list 2>/dev/null

Decision tree (no human input needed):

1. No hypotheses table or empty? → go to SEED phase.
2. Hypotheses exist but all are unproven? → go to PICK phase.
3. Mix of tested/untested? → go to PICK phase (prioritize untested).
4. All have >= 3 experiments? → go to JUDGE phase.
5. Graduation/kill candidates exist? → go to EVOLVE phase.

</phase>

<phase id="seed" name="SEED — Extract Rules as Hypotheses" trigger="sense.no_hypotheses">

Scan every CLAUDE.md in the project hierarchy + rules/.md + skills//SKILL.md.

Parse rule-like patterns:

• Imperative: "Always X", "Never Y", "Prefer Z", "Must W"
• Anti-patterns: block actions, "Don't", "Forbidden"
• Assertions: "X before Y", "X is better than Y"
• Quantitative claims: "reduces by X%", "takes N minutes"
• Conditional: "If X then Y", "When X, do Y"

For each rule:

    agentdb hypothesis add "<statement>" --domain <auto-classify> --source "<file:line>"

Domain auto-classification by keyword:

• research, anti-pattern, prior work → methodology
• parallel, agent, spawn, tier → coordination
• test, coverage, edge case, mock → testing
• commit, branch, merge, PR → git
• secret, validation, auth, injection → security
• measure, optimize, latency, profile → performance
• Big 5, review, quality → quality
• module, interface, coupling → architecture

Deduplicate: skip if near-identical statement already exists.

Log count, then immediately proceed to PICK. No pause.

    agentdb emit command "experiment-seed" "" '{"seeded":N}'

</phase>

<phase id="pick" name="PICK — Select Next Hypothesis">

Choose the hypothesis that will produce the most information.

Priority order:

1. Most uncertain: confidence closest to 0.5 (maximum ignorance — any experiment is maximally informative)
2. Least tested: fewest total experiments (break ties)
3. Highest impact domain: methodology > coordination > security > testing > quality > git > architecture > performance

    SELECT id, statement, domain, confidence, evidence_for + evidence_against as total_evidence
    FROM hypotheses
    WHERE status NOT IN ('graduated', 'refuted')
    ORDER BY ABS(confidence - 0.5) ASC, total_evidence ASC
    LIMIT 1;

</phase>

<phase id="design" name="DESIGN — Create the Experiment">

Autonomously design the minimum viable experiment.

Falsifiability gate: if no possible outcome could refute the hypothesis, redesign.

Every experiment defines BEFORE running: method, quantitative measurement, control

condition (what happens WITHOUT the rule), pass_criteria, fail_criteria.

Choose the LIGHTEST experiment type that produces signal:

1. HISTORICAL (cheapest — query existing data):

Query agentdb learnings, session outcomes, error patterns for evidence.

Use when: agentdb has >= 10 sessions or >= 20 learnings in the domain.

1. COMPARATIVE (medium — run a real task two ways):

Execute WITH the rule applied, then WITHOUT (or find prior without-cases).

Measure: time, error count, rework, quality.

1. ABLATION (medium — remove the rule, observe):

Temporarily ignore the rule during a real task. Record what breaks.

1. OBSERVATIONAL (passive — tag next N tasks):

Flag the hypothesis; future relevant tasks collect evidence passively.

Use when: active experimentation would be disruptive.

Minimum sample sizes: methodology/coordination/git/quality >= 3 comparisons;

testing >= 5 tasks per condition; security >= 50 fuzz inputs.

    agentdb experiment add <H_ID> "<method>" "<measurement>" --pass-criteria "<criteria>"

</phase>

<phase id="run" name="RUN — Execute and Observe">

Run the designed experiment. Record everything.

Gate: the control condition was actually tested, not just assumed.

• HISTORICAL: query agentdb with specific SQL; evidence = query result + interpretation.
• COMPARATIVE: execute the task (spawn agents if needed); evidence = measured delta.
• ABLATION: execute with the rule explicitly ignored; evidence = observed difference.
• OBSERVATIONAL: record the flag; skip to next hypothesis (no blocking).

</phase>

<phase id="conclude" name="CONCLUDE — Verdict and Confidence Update">

Compare observations against pass/fail criteria. Issue verdict honestly:

supports | refutes | inconclusive.

    agentdb experiment verdict <EXP_ID> <supports|refutes|inconclusive> "<evidence summary>"

Confidence update (Bayesian, applied automatically by CLI):

• supports: confidence += (1 - confidence) * 0.25
• refutes: confidence -= confidence * 0.3
• inconclusive: no change

Evidence strings must be specific and measurable, never narrative.

Lifecycle transitions:

• unproven → testing: first experiment registered
• testing → supported: confidence >= 0.8 AND evidence_for >= 3 AND ratio >= 3:1
• testing → refuted: confidence < 0.2 AND evidence_against >= 2
• supported → graduated: human approval after sustained confidence
• refuted → killed: human approval to remove from rules
• any → unproven: rule is modified (resets all evidence)

    agentdb learn pattern|failure "<what we learned>" "<evidence>"
    agentdb emit command "experiment-conclude" "" '{"H":"ID","EXP":"ID","verdict":"X","confidence":0.XX}'

Loop back to PICK for next hypothesis.

</phase>

<phase id="judge" name="JUDGE — Review All Evidence" trigger="sense.sufficient_evidence">

For each hypothesis with >= 3 experiments: summarize evidence, calculate final

confidence, classify graduated | refuted | needs-more-evidence | inconclusive.

    agentdb hypothesis export

Write detailed report to _meta/research/experiment-report.md. Proceed to EVOLVE.

</phase>

<phase id="evolve" name="EVOLVE — Self-Reconfigure">

The emergent part. The system reconfigures based on evidence.

Graduate (confidence >= 0.8, evidence_for >= 3, ratio >= 3:1):

promote via the artifact ladder with human approval — hook if enforceable, agent if

a role, skill if methodology; CLAUDE.md prose only as last resort.

Kill (confidence < 0.2, evidence_against >= 2):

propose rule removal from CLAUDE.md (present to human).

Mutate (inconclusive after 5+ experiments):

the hypothesis may be poorly stated; propose a refined version as a NEW hypothesis,

linked to the original (evolution chain).

<ask_user>

Use AskUserQuestion ONCE at the end of the evolve phase:

Ask: "{graduated} rules proven, {killed} rules disproven, {mutated} rules refined. Apply changes?"

Options: apply all, review individually, skip for now

</ask_user>

    agentdb emit command "experiment-evolve" "" '{"graduated":N,"killed":N,"mutated":N}'

</phase>

</cycle>

<loop_control>

continue_if: untested hypotheses remain OR new evidence changes confidence significantly

pause_at: EVOLVE phase (only human checkpoint — graduation/kill decisions)

stop_if: all hypotheses have >= 3 experiments AND no graduation/kill candidates

on_stop: write final report to _meta/research/experiment-report.md, agentdb write-end

Iteration budget: max 20 cycles per invocation.

</loop_control>

<anti_patterns>

<never>Confirm a hypothesis without running a real experiment.</never>

<never>Use a single data point to graduate a hypothesis.</never>

<never>Ignore refuting evidence because the rule "feels right".</never>

<never>Test a hypothesis with a method that can only confirm (design for falsifiability).</never>

<never>Modify the hypothesis after seeing results (that is a new hypothesis).</never>

</anti_patterns>

<hard_stops>

• NEVER modify CLAUDE.md autonomously. Present changes at EVOLVE, human decides.
• NEVER delete hypotheses. Mark as refuted. Audit trail is sacred.
• NEVER fabricate evidence. If experiment can't run, mark inconclusive with reason.
• NEVER run destructive experiments without explicit approval.
• ALWAYS record evidence, even for inconclusive results.

</hard_stops>

<on_end>

agentdb write-end '{"skill":"experiment","cycles":N,"experiments_run":N,"graduated":N,"refuted":N,"mutated":N}'

</on_end>

</skill>