ml-pipeline

ML Pipeline Discipline

**Hard rule: no model is trained until every earlier step in the pipeline is done and the

user has explicitly approved the phase gates before it.** "Train a model on this data" is a

request to start the pipeline at step 1, not at step 10.

The pipeline (strict order — never reorder, never skip silently)

RAW DATA
  → 1. Data inspection
  → 2. Exploratory data analysis (EDA)
  → 3. Define the prediction problem
  ──────────── GATE A: user approval ────────────
  → 4. Data cleaning
  → 5. Data engineering
  → 6. Train / validation / test split
  → 7. Feature engineering
  → 8. Preprocessing
  ──────────── GATE B: user approval ────────────
  → 9. Baseline model
  → 10. Model training
  → 11. Hyperparameter tuning
  → 12. Model evaluation
  ──────────── GATE C: user approval ────────────
  → 13. Error analysis
  → 14. Final test (test set touched ONCE)
  → 15. Deployment (only if user asks)
  → 16. Monitoring + retraining plan
  ──────────── GATE D: wrap-up report ───────────

Phase gates — explicit permission, every time

At each gate, STOP and give the user, in plain non-jargon language:

1. What was done — the steps completed, 1–2 sentences each.
2. What was found — key findings, with the visuals that show them.
3. Decisions made and why — e.g. "dropped 312 duplicate rows", "chose time-based split

because the data has dates".

1. What comes next — the next phase's steps, in one short list.
2. The question — ask for explicit permission to continue. Wait for a clear yes.

Silence, ambiguity, or "hmm" is not a yes. If the user redirects, incorporate it.

If the user says "skip ahead" or "just train it": explain in 2–3 sentences which steps are

missing and the concrete risk (usually leakage or garbage-in), then ask once for explicit

override confirmation. If they confirm, proceed and record the override in PIPELINE.md.

Progress tracking (survives across sessions)

On first use in a project, create ml_pipeline/PIPELINE.md — a checklist of the 16 steps

with status (todo / in progress / done / approved-gate / overridden), one line of results

per finished step, and dated gate approvals. Update it after every step. On any new session,

read it first and resume from the first unfinished step — never restart, never skip ahead

of it.

Record each gate approval on its own line in exactly this shape — the enforcement hook

parses it:

- Gate A: approved 2026-09-16
- Gate B: approved 2026-09-17
- Override: Gate B - user approved skipping to training 2026-09-17 - reason: <why>

A line that says a gate is todo, pending, or not yet approved never counts as approval.

After recording a gate approval in a git repository, checkpoint it:

git add -A && git commit -m "ml-pipeline: Gate X approved" && git tag -f gate-X. Every approved

phase is then a reproducible point to return to.

Tools: marimo notebooks + matplotlib visuals

• The workbench is a marimo notebook, not loose scripts. Keep notebooks in

ml_pipeline/: 01_eda.py (steps 1–3), 02_prep.py (steps 4–8), 03_model.py

(steps 9–12), 04_eval.py (steps 13–16). In Claude Code, drive them live with the

marimo-pair skill so the user watches the work happen. In other harnesses, write the

notebook files and tell the user to open them with marimo edit <file>.

• Every step that looks at data produces matplotlib figures (seaborn on top is fine).

Also save each figure to ml_pipeline/figures/<step>_<name>.png so gates can reference

them even without a live notebook.

• Explain every figure in 1–2 plain sentences: what it shows and why it matters for

the next decision. A figure without an explanation is not done.

What each step must produce

1. Data inspection — load raw data read-only. Report: rows × columns, column types,

first rows, memory size, unique counts, obvious junk. No modification yet.

1. EDA — distributions of every variable, missing-value map, correlations,

target balance, time trends if temporal, group structure (repeated entities?).

Output: figures + a short list of hypotheses and problems spotted.

1. Define the prediction problem — write a short contract: target (exact definition,

units), prediction unit and population, prediction time/horizon, information actually

available at prediction time, objective, evaluation metric, constraints. **The user must

approve this contract at Gate A** — it controls everything after.

1. Data cleaning — missing values, duplicates, invalid/impossible values, inconsistent

categories, unit/format issues. Report before/after counts for every rule. Document every

rule in PIPELINE.md. Prefer preserving data over deleting. Never use information from the

future or from the test rows to decide a cleaning rule.

1. Data engineering — joins/integration, aggregation, time alignment to an index date,

business rules, one-row-per-prediction-unit feature table, data-quality checks

(row counts, uniqueness, ranges).

1. Split before any fitting — copy the plugin's guard library to ml_pipeline/guard.py

(its path is given at session start; in Codex/Kimi it is skills/ml-pipeline/lib/mlpipeline_guard.py

next to this file) and split with it:

train, val, test = guard.split(df, target=..., time_col=<col> if the data is temporal, group_col=<col> if the same entity appears in multiple rows).

It drops exact duplicates, refuses a random split when a datetime column exists, keeps groups

together, checks that no row lands in two splits, and freezes a fingerprint of the test set.

The test set is touched exactly once, at step 14, through guard.final_test().

1. Feature engineering — design features on the training set's statistics only, then

apply the same transformations to validation/test.

1. Preprocessing — scalers, encoders, imputers fit on train only, wrapped in a pipeline

object so train and inference can never diverge.

1. Baseline first — a dummy predictor (majority class / mean) AND one simple model

(logistic/linear regression or small tree). Record their metrics. Every later model is

judged against this line; a complex model that can't beat it gets reported as such.

1. Model training — train candidate models on train, compare on validation. Log every

run's config and score.

1. Hyperparameter tuning — on validation/cross-validation only. The test set is never

part of tuning.

1. Model evaluation — the contract's metric plus supporting views: confusion matrix and

ROC/PR curves for classification, residual plots for regression, always compared to the

baseline. Figures required.

1. Error analysis — worst predictions, performance by slice/subgroup, calibration,

where the model fails and a hypothesis for why.

1. Final test — score = guard.final_test(model.predict, test, target=..., metric_fn=...).

It verifies the frame is the frozen test set, refuses a second call, and logs the result to

PIPELINE.md. Report the number honestly, even if it is worse than validation. No going back to

tune on it — if the result forces changes, agree a new test strategy with the user and record

- Override: final test - user approved a second evaluation YYYY-MM-DD - reason: <why>.

1. Deployment — only when the user asks. Save the full pipeline artifact

(preprocessing + model together), verify a reloaded artifact reproduces predictions,

document the inference input contract.

1. Monitoring + retraining — write down: what drift to watch (input and prediction

distributions), what metric threshold triggers retraining, and how retraining reuses

this same pipeline from step 1.

Enforcement in Claude Code (hook)

Installed as a Claude Code plugin, hooks/guard_training.py runs before every Bash, Write,

Edit, and notebook tool call, scans the code for fitting and training calls, and denies:

• model training until PIPELINE.md has Gate B: approved … or an Override: Gate B …

line. With no ml_pipeline/PIPELINE.md at all, training is denied with a pointer to step 1.

• any fitting — scalers, encoders, imputers included — until Gate A: approved ….
• fitting on the test split (X_test, df_test, test_*) — always, at every gate.
• evaluating on the test split — .predict/.score or a metric function whose arguments name

X_test, df_test, test_* — until Gate C: approved … (or Override: Gate C …).

A PostToolUse hook also warns (never blocks) when a result looks too good to be honest

(accuracy/AUC/F1 ≥ 0.98) or when train_test_split() is called without stratify= or on data that

mentions dates. To prove a pipeline does not leak, run it on examples/canary/canary.csv: honest

test accuracy cannot exceed 0.80, and mlpipeline_canary.verdict(score, n_test) says whether a

number is above the ceiling.

A denial is not an obstacle to route around: do the missing steps, get the user's explicit

approval, record the gate line, and retry. The hook fails open on its own errors, and

ML_PIPELINE_ENFORCE=0 switches it off for projects that are not ML pipelines. Codex and Kimi

have no hook support, so there the same rules apply as instructions only.

Non-negotiables

• Selection leakage is the one that matters most: the test set is never used to pick a model, a

seed, a feature, or a threshold. Validation only.

• Test set is used exactly once. No tuning, no peeking, no "just checking".
• All fitting (cleaning statistics, features, preprocessing, models) uses training data only.
• Temporal data gets temporal splits; repeated entities get group splits.
• Baseline before any complex model; every result is reported relative to it.
• Failures and disappointing numbers are reported plainly — never hidden or reframed.
• Chat explanations stay beginner-friendly; the code stays production-grade.