hill-climb

Hill Climb

The toolkit's metric-driven optimization loop. One number moves; everything else

stays fixed. Each iteration states one hypothesis, makes one change, runs the

correctness floor, re-measures, and either accepts the change or reverts it. The

ledger of what was tried and what failed ships with the code.

Sibling to objective-loop: that loop verifies boolean criteria and reschedules;

this loop optimizes a continuous metric against variance. Route here whenever the

goal is a number moving in a direction.

Phase 1: SPEC

Fill these fields from the request. Interview only for what is missing.

| Field | Meaning | Required | Default |

|---|---|---|---|

| METRIC | One number, with units and direction (lower or higher is better) | yes | — |

| MEASURE | A deterministic command that prints that number, repeatable | yes | — |

| TARGET | The value that ends the loop | yes | — |

| FLOOR | Correctness gate command(s) that must exit 0 every iteration | yes | — |

| FIXTURE | Dataset, workload, or input identity, pinned to a commit or checksum | yes | — |

| Variance tolerance | Spread below which a delta means nothing | no | 2x the baseline spread |

| Iteration budget | Iterations before a forced stop | no | 8 |

| Plateau threshold K | Consecutive non-improving iterations that stop the loop | no | 3 |

Rules:

• One METRIC per loop. Two numbers with a trade-off need one of them promoted to the FLOOR (for example: "p99 latency drops, memory stays under 500 MB").
• MEASURE prints the number and nothing that requires interpretation. Wrap noisy tools in a script that emits one value.
• A hill climb against a varying dataset measures nothing. Pin FIXTURE before Phase 2 — same input rows, same seed, same machine class, same warm/cold state.
• FLOOR is executed, not asserted. Name the command.
• A domain playbook fills this table fast: references/domain-playbooks.md carries pre-filled blocks for frame rate, API latency, CI time, test runtime, bundle size, memory, token cost, and game-design quality.
• A judgment score can be the METRIC only under the frozen-rubric contract in that reference: rubric frozen at SPEC time, graded in a fresh context by an agent that did not author the change, wider accept threshold. Without a freezable rubric the request is not hill-climbable — that reference names where it goes instead.

Gate: all required fields hold concrete values. Proceed to Phase 2.

Phase 2: BASELINE

Run MEASURE N times (N ≥ 5, N ≥ 10 for wall-clock metrics) before changing any

code. Record every sample, the median, and the spread (max − min, or p95 − p5).

| Condition | Action |

|---|---|

| Spread < target improvement | Proceed. Set the variance tolerance from the spread. |

| Spread ≥ target improvement | Stop. Report that the harness is too noisy to hill-climb on. |

A noisy-harness stop is a correct outcome, not a failure. Report the measured

spread, name the likely noise sources (shared CI runners, thermal throttling,

network calls, unpinned data, garbage-collection timing), and offer to stabilize

the harness first. Never proceed by averaging harder and hoping.

Gate: baseline median and spread written to the ledger. Proceed to Phase 3.

Phase 3: PROFILE

Locate the cost before changing anything. Guessing at hot spots is the dominant

failure mode of optimization work.

| Domain | Tooling |

|---|---|

| Python CPU | py-spy record, cProfile + snakeviz, pyinstrument |

| Python memory | memray, tracemalloc |

| Go | pprof (-cpuprofile, -memprofile), go test -bench -benchmem, benchstat |

| Browser runtime and frame rate | Chrome DevTools performance trace; in this harness mcp__chrome-devtools__performance_start_trace, performance_stop_trace, performance_analyze_insight, take_heapsnapshot |

| Bundle size | webpack-bundle-analyzer, rollup-plugin-visualizer, source-map-explorer |

| Test runtime | pytest --durations=25, vitest --reporter=verbose, go test -json timings |

| CI wall-clock | Per-job and per-step durations from the CI API; critical-path analysis across the job graph |

| Token cost | Per-call token counts by prompt component; context-size attribution |

Write one hypothesis before the edit, in this shape: *"X consumes N% of METRIC

because Y; doing Z should recover about M."* Hypothesis first, edit second.

Gate: profile output captured, one hypothesis written to the ledger. Proceed to Phase 4.

Phase 4: ITERATE

Per iteration, in order:

1. State one hypothesis (from Phase 3, or re-profiled after the last accept).
2. Make one change. One change per iteration — bundled changes cannot be attributed.
3. Run FLOOR. Red floor ends the iteration: revert, log, next.
4. Run MEASURE the same N times as the baseline, same fixture, same conditions.
5. Compare the new median to the current best.

| Floor | Metric vs best | Action |

|---|---|---|

| Green | Improved beyond variance tolerance | Accept. New best. Log. |

| Green | Improved within variance tolerance | Revert. Noise, not a win. Log as inconclusive. |

| Green | Worse or unchanged | Revert. Log the negative result — it is the valuable part. |

| Red | Any | Revert. Log the floor failure. |

Guardrails (hard rules). Never move the number by moving the goalposts:

• Never edit, shrink, reseed, or re-sample the FIXTURE.
• Never relax, skip, mark-xfail, or delete a FLOOR test.
• Never shrink the workload, lower the iteration count, or cut the input size.
• Never change MEASURE mid-loop; a changed measure invalidates the baseline and restarts the loop.
• Never cache away work the metric is supposed to include, or move it outside the measured region, unless the user accepts that as the intended change.
• Never accept on a single sample.

Hitting a guardrail is a stop-and-report event, not a judgment call. If the only

visible path to TARGET crosses one, stop and put the conflict to the user.

Gate: iteration logged with accept/revert and evidence. Loop until a Phase 5 stop condition fires.

Phase 5: LEDGER

.hillclimb/<slug>/ledger.md is append-only and written every iteration.

Entries are never edited retroactively. Template and field rules:

references/ledger.md.

Per iteration record: hypothesis, change summary (files touched), metric median

and spread, delta vs baseline, delta vs best, floor status, accepted or reverted,

and why.

The ledger ships with the code. It tells the next person what was tried and what

did not work, which is what stops the same dead end being re-walked. Keep

.hillclimb/ unstaged unless the user asks for the ledger in the repo.

Phase 6: STOP

| Condition | Report |

|---|---|

| TARGET hit | Baseline → final value, accepted changes in order with their deltas, floor green, ledger path |

| Iteration budget exhausted | Best achieved value, remaining gap to TARGET, next hypothesis in the queue |

| Plateau: K consecutive non-improving iterations | Best achieved value, the hot spots the profiler still shows, and what a next attempt would need |

| Noisy harness (Phase 2) | Measured spread, noise sources, harness-stabilization proposal |

| Guardrail conflict (Phase 4) | The guardrail, the change that would have crossed it, and the decision put to the user |

On plateau, name the shape of the next attempt honestly: a different algorithm,

a different data structure, an accepted architectural change, or "the remaining

cost is irreducible at this design." Do not silently keep grinding past K.

Error Handling

| Error | Cause | Solution |

|---|---|---|

| MEASURE prints varying numbers for identical code | Unpinned fixture, shared machine, network in the measured path | Return to Phase 2; pin the fixture or stabilize the harness before iterating |

| FLOOR was already red at baseline | Pre-existing failure | Stop. Fix correctness first — a hill climb over a red floor accepts nothing |

| Metric improves but the floor flakes intermittently | Flaky test in FLOOR | Treat as red; stabilize or replace that gate. Never downgrade a flaky gate to make an accept stick |

| Large win from one change looks too good | Work was cached, skipped, or moved out of the measured region | Verify the workload still executes: assert output equality against the baseline run before accepting |

| Ledger missing on resume | .hillclimb/ removed mid-loop | Re-run Phase 2 baseline; do not resume from memory |

Reference Loading Table

| Signal | Load These Files | Why |

|---|---|---|

| Writing or resuming the ledger | references/ledger.md | Append-only template, slug rules, resume protocol |

| Choosing a profiler, or reading its output | references/profiling-tools.md | Per-domain commands, output reading, common traps |

| Phase 1 in a named domain — frame rate, API latency, CI time, test runtime, bundle size, memory, token cost, game-design quality | references/domain-playbooks.md | Pre-filled METRIC/MEASURE/FIXTURE/FLOOR blocks and typical hot spots |

| The metric is a judgment score, or the request may not be measurable at all | references/domain-playbooks.md | Frozen-rubric contract and the not-hill-climbable boundary |

References

• ${CLAUDE_SKILL_DIR}/references/ledger.md — ledger template and resume protocol
• ${CLAUDE_SKILL_DIR}/references/profiling-tools.md — per-domain profiling commands and traps
• ${CLAUDE_SKILL_DIR}/references/domain-playbooks.md — pre-filled SPEC blocks per domain, rubric-metric contract, not-hill-climbable boundary