run-parity

Run pull request parity

Run the pull request base and head against the same repository commits. Write deterministic newline-delimited JSON (NDJSON) artifacts and compare them.

Prepare the run

Require DAYTONA_API_KEY, authenticated gh, and a pushed pull request head. Do not push changes without permission. Use ni and nr in this repository.

Resolve the pull request:

gh pr view <pr-number-or-url> \
  --json number,url,baseRefOid,headRefOid,headRepository,headRepositoryOwner

Derive the base repository from the pull request URL. Derive the head repository from headRepositoryOwner.login and headRepository.name. Use the returned commit hashes, not branch names.

Create tmp/parity-pr-<number>-<head-short-sha> and preserve it after the run. Run ni before evaluation.

Run both revisions

Run from packages/evals. The default corpus contains the 2,000 highest-ranked repositories, and the initial concurrency is 200. Sandbox creation is capped at 20 to avoid overloading Daytona. The evaluator cleans up resources and retries failed projects at concurrency 50, then 10.

nr --silent eval \
  --react-doctor-repository <base-repository-url> \
  --react-doctor-ref <baseRefOid> \
  > <absolute-run-directory>/baseline.ndjson

nr --silent eval \
  --repositories <absolute-run-directory>/baseline.ndjson \
  --react-doctor-repository <head-repository-url> \
  --react-doctor-ref <headRefOid> \
  > <absolute-run-directory>/candidate.ndjson

Cache a successful baseline only under its exact React Doctor commit, corpus

manifest hash, evaluator schema, and ruleset/config hash. A cached baseline must

still pass the streaming validator before reuse. PRs may share that immutable

baseline, but never combine their candidate heads or deltas.

The evaluator stamps every record with the exact detector commit, revision-local

rule/config hash, and evaluator source hash. Never cache an older unstamped run.

After the normal input validator passes, create the adjacent provenance file:

node .agents/skills/run-parity/scripts/baseline-cache-provenance.mjs create \
  --baseline <baseline.ndjson> \
  --corpus-manifest <corpus-manifest.json> \
  --base-commit <full-base-commit> \
  --repository <react-doctor-repository-url> \
  --evaluator-source-hash <packages-evals-source-hash> \
  --config-contract revision-local-rule-config-v1 \
  --rule-set-hash <stamped-full-ruleset-hash>

Get the expected evaluator hash with nr --silent source-hash from

packages/evals. Before every reuse, run the same command with verify.

Verification streams the raw NDJSON bytes, requires full-baseline ruleKeys: [],

checks every record's producer, and independently requires the exact pinned

corpus project set to match its manifest. Any mismatch is a cache miss.

On a cache hit, keep the base out of Daytona: run the normal candidate-only

command above against the validated cached baseline and do not pass any

--paired-* option.

On a cache miss, or for a required full-versus-scoped shadow run, evaluate both

detectors in the same Daytona sandbox:

nr --silent eval \
  --repositories <corpus-manifest-or-validated-input> \
  --paired-baseline-output <absolute-new-baseline-path> \
  --paired-base-react-doctor-repository <base-repository-url> \
  --paired-base-react-doctor-ref <base-or-full-shadow-commit> \
  --paired-base-rule <treatment-plugin/rule> \
  --react-doctor-repository <treatment-repository-url> \
  --react-doctor-ref <treatment-commit> \
  --paired-execution sequential \
  --rule <treatment-plugin/rule> \
  > <absolute-run-directory>/candidate.ndjson

The baseline output path is created exclusively and never overwrites an

existing artifact. Paired sandboxes fetch each target repository once into one

object store, then scan isolated base/treatment target worktrees with isolated

detector installs, config files, and report paths. A project pair is emitted

only after both sides succeed, so retries cannot leave a partial successful

pair in either output. Paired writes use one single-writer queue and roll the

baseline back if either sink fails. Any nonzero paired evaluator run still

makes both output artifacts invalid and non-reusable; discard them instead of

feeding them to the comparator or cache.

Use the same --paired-base-rule and --rule values for performance

comparison, or omit both for a full-rule comparison. A full-versus-scoped

shadow run remains valid diagnostic evidence but not performance evidence.

Paired sandboxes request four CPU cores, eight GiB of memory, and twenty GiB of

disk. --paired-execution auto is the default and runs the scans in parallel

only when the sandbox has at least four CPU cores. Use

--paired-execution sequential for the controlled sequential benchmark and

the performance comparison below. Use auto or parallel only for

diagnostic-only parity where timing evidence will be discarded. Paired

evaluations default to 50 sandboxes, below the observed capacity ceiling for

four-core sandboxes; pass --concurrency only when the Daytona allocation

supports a different envelope.

Both modes share the same hard attempt deadline and exact evaluation-label

cleanup. Each paired scan has a five-minute command cap so a small number of

stuck sandboxes cannot consume the whole attempt budget; the ordinary evaluator

retains its existing timeout.

For several pull requests with the same immutable base and corpus, use repeatable

matrix treatment descriptors. Get the exact corpus identity and evaluator hash first:

cd packages/evals
nr --silent matrix-corpus-identity <absolute-corpus-manifest-path>
nr --silent source-hash

Each descriptor is an immutable JSON file with this exact shape:

{
  "schemaVersion": 1,
  "id": "pr-1234",
  "artifactDirectory": "/absolute/path/pr-1234",
  "reactDoctorRepository": "https://github.com/millionco/react-doctor.git",
  "reactDoctorCommit": "<40-character-head-commit>",
  "impactManifestPath": "/absolute/path/pr-1234-impact.json",
  "impactManifestSha256": "<sha256>",
  "group": {
    "baseReactDoctorRepository": "https://github.com/millionco/react-doctor.git",
    "baseReactDoctorCommit": "<40-character-base-commit>",
    "baseFullRuleSetHash": "<full-base-rule-set-sha256>",
    "baseArtifactPath": "/absolute/path/base-union-scoped.ndjson",
    "baselineOutputPath": "/absolute/cache/full-baseline.ndjson",
    "baselineProvenancePath": "/absolute/cache/full-baseline.provenance.json",
    "corpusManifestPath": "/absolute/path/corpus.json",
    "corpusManifestSha256": "<matrix-corpus-identity manifestSha256>",
    "corpusProjectSetSha256": "<matrix-corpus-identity projectSetSha256>",
    "evaluatorSourceHash": "<source-hash>",
    "configContract": "revision-local-rule-config-v1",
    "scanContract": "react-doctor-json-full-v1",
    "reportContract": "react-doctor-complete-report-v1",
    "projectRootPolicy": "manifest-root-dir-v1"
  }
}

The referenced impact manifest must be the exact output from

find-impacted-rules.mjs; its hash, base commit, head commit, mode, and candidate

rule keys are revalidated. Before Daytona starts, the matrix runner fetches the

pinned base and head commits, reruns the current generator, and requires

byte-identical manifest output. Every repeated descriptor must have the exact

same group object, a unique safe id, and a distinct artifact directory.

nr --silent eval \
  --matrix-treatment /absolute/path/pr-1234.json \
  --matrix-treatment /absolute/path/pr-1235.json \
  --matrix-wave-width 2

A validated full cache hit keeps the base out of Daytona. Otherwise, the matrix

scans one full base when any treatment requires full parity, or one sorted union

of incremental rule scopes. One target bare clone feeds isolated lane worktrees.

The default two-lane wave uses four CPU cores and eight GiB per sandbox; the

runner derives concurrency under the 400-CPU envelope and keeps sandbox creation

at 20. Retries retain successful (lane, project) results and retry only failed

work at 50, 10, then 2 concurrency. Each treatment is atomically published with

its candidate NDJSON, exact corpus manifest, descriptor, impact manifest, rules,

hashes, counts, and provenance. A missing base marks successful treatments

blocked rather than making an independent merge decision.

Treat each published treatment directory as self-contained evidence. Verify its

status, canonical relative paths, producer binding, byte lengths, hashes, exact

record counts, complete reports, and corpus project tuples before comparison.

Never follow a shared cache or scoped-base source path from provenance:

node .agents/skills/run-parity/scripts/verify-matrix-artifact.mjs \
  <treatment-artifact-directory>

# Incremental treatment
node .agents/skills/run-parity/scripts/compare-parity.mjs \
  --rules <treatment-artifact-directory>/rules.json \
  <treatment-artifact-directory>/base.ndjson \
  <treatment-artifact-directory>/candidate.ndjson \
  > <treatment-artifact-directory>/parity.json

# Full treatment
node .agents/skills/run-parity/scripts/compare-parity.mjs \
  <treatment-artifact-directory>/base.ndjson \
  <treatment-artifact-directory>/candidate.ndjson \
  > <treatment-artifact-directory>/parity.json

For rule-only pull requests, build the conservative impact manifest before the

candidate run:

node .agents/skills/run-parity/scripts/find-impacted-rules.mjs \
  <repository-root> <base-ref> <head-ref> <impact.json>

Use incremental mode only when the manifest reports "mode": "incremental".

Its candidateRuleKeys already includes known diagnostic-interaction closure.

Write those keys unchanged to the rules JSON and pass each as a repeatable

candidate argument:

nr --silent eval \
  --repositories <validated-baseline-or-corpus-manifest> \
  --react-doctor-repository <head-repository-url> \
  --react-doctor-ref <headRefOid> \
  --rule <plugin/rule> \
  > <absolute-run-directory>/candidate-scoped.ndjson

The evaluator stamps every unselected revision-local rule off. When the

scope contains no security-scan rule it also skips that whole-tree pass.

Security-scan rule changes, global runner/config/report/registry changes,

removed or renamed rule IDs, unresolved runtime edges, parse failures, and

other uncertain dependency surfaces fall back to full parity. Plugin utilities

that reach a host module without first crossing a mapped rule boundary also

fall back to full parity. A manifest with no runtime rule impact uses full mode;

never construct an empty incremental rule scope.

Every run keeps its hard outer timeout. Short evaluator budgets cap aggregate

retry reserve at 25% of the time remaining when an attempt starts, leaving at

least 75% for active work instead of letting snapshot build consume the whole

first-attempt deadline.

The baseline records resolved repository hashes. Reusing the baseline as the candidate corpus prevents default branches from moving between runs.

Before the candidate run, stream-validate every baseline record. This rejects unpinned repositories, evaluation errors, malformed reports, and incomplete projects without loading the NDJSON corpus into memory:

jq -e -n \
  -f <repository-root>/.agents/skills/run-parity/scripts/validate-parity-input.jq \
  <absolute-run-directory>/baseline.ndjson >/dev/null

If the baseline command exits non-zero or the check fails, inspect its failed records and stop. Candidate runs reject unpinned evaluation NDJSON.

Require both commands to exit zero and report 100% completion. Otherwise, report the failed projects and stop the comparison.

Compare results

Run from the repository root:

node .agents/skills/run-parity/scripts/compare-parity.mjs \
  <run-directory>/baseline.ndjson \
  <run-directory>/candidate.ndjson \
  > <run-directory>/parity.json

node .agents/skills/run-parity/scripts/compare-parity-performance.mjs \
  <run-directory>/baseline.ndjson \
  <run-directory>/candidate.ndjson \
  > <run-directory>/performance-parity.json

Interpret exit codes:

• 0: diagnostics match
• 1: comparison succeeded with diagnostic changes
• 2: inputs are incomplete or invalid

The evaluator retries incomplete reports instead of recording them as successful. The comparator validates both inputs again and exits with invalid-input status if either side contains an evaluation error, a malformed report, a missing completion marker, or a partial legacy report. It canonicalizes diagnostics to report-relative identities across legacy and v3 report schemas, so overlapping workspace scans do not inflate counts and schema upgrades do not appear as diagnostic churn.

The comparator streams both NDJSON inputs, stages baseline records in the system temporary directory, and writes large detail arrays incrementally. It retains changed diagnostic entries only long enough to sort them deterministically, so leave temporary-disk and output capacity proportional to the run size.

The performance comparator requires complete v3 reports, identical project

coverage, rule scope, config contract, and evaluator source. Run it only on a

paired sequential evaluation so base and candidate share the same Daytona

sandbox without competing for CPU. Exit 0 means timings stayed within the

fixed noise-tolerant thresholds, 1 means a material project or aggregate

regression, and 2 means the evidence is invalid. It ignores projects whose

base scan is under one second, flags an individual project only when it is at

least 50% and two seconds slower, and flags a corpus-wide regression only with

at least ten eligible projects, a 20% ratio, and ten seconds of aggregate

added latency. Inspect the emitted measurements before attributing a cause.

Matrix lanes run concurrently and remain diagnostic evidence; do not feed

matrix artifacts to the performance comparator.

For exit code 1, inspect affected source locations before classifying changes.

For a scoped comparison, filter the full baseline to the same rule and

always-on invariant scope:

node .agents/skills/run-parity/scripts/compare-parity.mjs \
  --rules <rules.json> \
  <baseline.ndjson> \
  <candidate-scoped.ndjson> \
  > <run-directory>/parity-scoped.json

The scoped comparator rejects arbitrary out-of-scope candidate diagnostics,

requires exact project/framework/analyzed-file coverage, preserves duplicate

multiplicity, and compares every semantic diagnostic field including canonical

primary and related paths.

Build compact Merkle indexes when the same baseline or candidate will be

compared more than once:

node .agents/skills/run-parity/scripts/build-parity-index.mjs \
  --rules <rules.json> <baseline.ndjson> \
  > <baseline.index.json>

node .agents/skills/run-parity/scripts/build-parity-index.mjs \
  --candidate --rules <rules.json> <candidate-scoped.ndjson> \
  > <candidate.index.json>

node .agents/skills/run-parity/scripts/compare-parity-indexes.mjs \
  <baseline.index.json> <candidate.index.json> \
  > <index-diff.json>

Equal whole-run roots stop immediately. Differing roots descend through rule

hashes and then project buckets. Empty project/rule buckets are explicit, and

scope or coverage metadata drift fails closed.

Until incremental parity has enough shadow history to become a required gate,

run one full candidate at the exact same head/corpus/concurrency policy and

require its rule-filtered output to match the scoped candidate exactly. Report

measured wall time and project latency; do not project a speedup from samples.

Validate comparator changes from the repository root:

node --test .agents/skills/run-parity/scripts/compare-parity.test.mjs
node --test .agents/skills/run-parity/scripts/compare-parity-performance.test.mjs
node --test .agents/skills/run-parity/scripts/find-impacted-rules.test.mjs
node --test .agents/skills/run-parity/scripts/parity-index.test.mjs
node --test .agents/skills/run-parity/scripts/validate-parity-input.test.mjs

Report results

Report the pull request URL, commit hashes, compared and skipped project counts, diagnostic totals, added and removed counts, largest rule deltas, and artifact paths.