eval

Eval — Quality & Regression Gate

CRITICAL: Run HARNESS_DIR=$(epic path) first. Never use .harness/ in the project directory.

When to Trigger

• Before /ship creates a PR (automatic if eval.yaml exists)
• After /go completes a feature
• On explicit /eval command
• When user mentions "regression", "baseline", "eval suite", "quality check"
• CI: make eval or epic eval --json

Execution Modes

4 dimensions run in parallel where possible:

1. eval:correctness — Test pass rate, mutation score, assertion density
2. eval:performance — Throughput, latency, memory (opt-in)
3. eval:quality — Lint, code quality, LLM-as-judge
4. eval:regression — Baseline comparison, score deltas

---

Process

Step 0: Prerequisites

HARNESS_DIR=$(epic path)

If $HARNESS_DIR/eval/eval.yaml does not exist, run scaffold:

epic eval --init

Read the config:

cat $HARNESS_DIR/eval/eval.yaml

Step 0.5: Scaffold benchmarks (when no benchmark infrastructure exists)

If eval.yaml has benchmarks: [] and no benchmark files are found in the project:

1. Generate stub files using the CLI:

   epic eval --scaffold

Supported stacks (auto-detected from project markers):

| Stack | Detected by | Generated file | Output format |

|-------|-------------|----------------|---------------|

| Rust | Cargo.toml | benches/eval_harness.rs | criterion (exit code) |

| Python | pyproject.toml / setup.py | benchmarks/eval_runner.py | JSON composite |

| TypeScript | tsconfig.json | benchmarks/eval.ts | JSON composite |

| Node.js | package.json | benchmarks/eval.mjs | JSON composite |

| Go | go.mod | benchmarks/eval_test.go | JSON composite |

| Java | pom.xml / build.gradle | benchmarks/EvalBenchmark.java | exit code |

| Kotlin | build.gradle.kts | benchmarks/EvalBenchmark.kt | exit code |

| Ruby | Gemfile | benchmarks/eval_benchmark.rb | JSON composite |

| PHP | composer.json | benchmarks/eval_benchmark.php | JSON composite |

| C# | .csproj / .sln | Benchmarks/EvalBenchmark.cs | JSON composite |

| Swift | Package.swift | benchmarks/EvalBenchmark.swift | JSON composite |

| Elixir | mix.exs | benchmarks/eval_benchmark.exs | JSON composite |

| C++ | CMakeLists.txt | benchmarks/eval_benchmark.cpp | exit code |

1. Customize the generated file — every file has # TODO / // TODO markers:

• Replace placeholder logic with calls to your actual domain functions
• Adjust the composite score weights to reflect your domain priorities
• For precision/recall benchmarks: wire in your real test set and model

1. If --scaffold can't generate a useful stub (domain too complex, custom evaluation logic needed), generate a custom benchmark with LLM assistance:

• Read the project's main source files to understand the domain
• Identify the 2–3 most critical quality signals (latency, accuracy, throughput, precision/recall)
• Write a benchmark that measures those signals and outputs {"composite": 0.0–1.0, ...}
• Save to benchmarks/eval_runner.{ext} matching the project language

1. Wire into eval.yaml:

   benchmarks:
     - name: eval_runner
       command: python3 benchmarks/eval_runner.py full
       result_type: composite   # parse composite field from JSON stdout

Use result_type: exit_code for frameworks (criterion, JMH, BenchmarkDotNet) that manage their own output.

Step 1: Run Rust CLI

Execute the structured evaluation via the Rust binary:

epic eval --json

This runs all enabled dimensions and outputs a JSON result. Capture the output.

If the CLI reports llm_judge: SKIPPED (no LLM available in CLI mode), proceed to Step 2 for LLM-as-judge. Otherwise, skip to Step 3.

Step 2: LLM-as-Judge (when llm_judge enabled)

If the quality dimension has llm_judge: true and CLI marked it SKIPPED:

1. Sample 3–5 changed files from the current branch:

   git diff --name-only $(git merge-base HEAD main)

1. For each sampled file, evaluate on a 1-10 rubric:

• Readability (naming, structure, flow)
• Correctness (logic, edge cases, error handling)
• DRY (no unjustified duplication)
• Security (no obvious vulnerabilities)

1. Average scores across files. Map to 0.0–1.0 scale.

1. Record results alongside CLI output.

Step 3: Load Baseline

cat $HARNESS_DIR/eval/baselines/latest.json

If no baseline exists, the current run BECOMES the first baseline. Save it:

epic eval --baseline-update

Report: "First baseline established. Future runs will compare against this."

Step 4: Synthesize Report

Combine CLI output + LLM-as-judge results into a single report:

## Eval Report
- Branch: {branch}
- Commit: {commit_short}

### Correctness: [PASS/WARN/FAIL] — score: {score}
- Tests: {passed}/{total} passing ({pass_rate}%)
- Mutation score: {mutation_score}% (if enabled)
- Delta vs baseline: {+/-delta}

### Performance: [PASS/WARN/FAIL] — score: {score} (if enabled)
- Avg latency: {latency}ms (delta: {+/-delta})
- Throughput: {throughput} (delta: {+/-delta})

### Quality: [PASS/WARN/FAIL] — score: {score}
- Lint errors: {count}
- LLM judge: {score}/10 (if enabled)

### Regression: [PASS/FAIL]
| Dimension | Baseline | Current | Delta | Verdict |
|-----------|----------|---------|-------|---------|
| correctness | {prev} | {cur} | {delta} | {pass/fail} |
| quality | {prev} | {cur} | {delta} | {pass/fail} |

### Overall: [PASS/WARN/FAIL] — {overall_score}

Step 5: Act

• All PASS + no regression: "Eval passed. Run /ship to create a PR."
• WARN: Show warnings. Ask whether to fix before shipping.
• FAIL or regression detected: List each failure with fix hint. "Fix with /go, then re-run /eval."

Step 6: Save Results

epic eval --baseline-update  # if user approves this as new baseline

Results auto-saved to $HARNESS_DIR/eval/results/EVAL-{timestamp}.json.

---

Anti-Rationalization

| Excuse | Rebuttal | What to do instead |

|--------|----------|-------------------|

| "Tests pass, no need for eval" | Tests pass today but regress tomorrow without baselines | Run eval and establish a baseline |

| "Performance testing is premature" | Latency regressions are invisible until users complain | Enable performance dimension, run benchmarks now |

| "Mutation testing is too slow" | Slow mutation catches bugs fast tests miss | Run on changed modules only (--dimension correctness) |

| "LLM-as-judge is subjective" | Subjective beats absent — fixed rubric + averaging reduces variance | Use the 4-axis rubric, average across 3+ files |

| "We can add eval later" | Later never comes; regressions accumulate silently | Start with correctness+quality, add dimensions incrementally |

| "CI will catch regressions" | CI only catches build/test failures, not quality drift | Eval measures what CI misses: mutation score, LLM quality |

Evidence Required

• [ ] epic eval --json output captured (all enabled dimensions scored)
• [ ] Baseline comparison performed (or first baseline established)
• [ ] Each dimension has PASS/WARN/FAIL verdict
• [ ] No dimension regressed beyond threshold (or explicit user override)
• [ ] Results saved to $HARNESS_DIR/eval/results/
• [ ] LLM-as-judge scores recorded (if enabled)

Red Flags

• Reporting PASS without actual epic eval output
• Skipping regression comparison "because it's the first run" (first run should ESTABLISH baseline)
• Reporting PASS with 0 test coverage
• Ignoring mutation score drops >5%
• Marking eval PASS when any dimension below minimum threshold
• Running eval on main branch instead of feature branch