palace-index-curator

Palace Index Curator

Overview

The web-research hooks auto-capture every WebFetch and WebSearch into

hooks/memory-palace-index.yaml, storing each as a markdown file and an

index entry. Captures land at the defaults routing_type: pending,

maturity: seedling, importance_score: 50, and nothing advances them.

Left alone, the index becomes a write-only graveyard: the majority of

entries are never incorporated, analyzed, or surfaced.

This skill drains that backlog and keeps it drained. It wires the

capture index to the corpus tooling the plugin already ships

(decay_model, keyword_index, marginal_value) through three

commands: a read-only report, a dry-run-first promotion engine, and a

SessionStart surfacing hook.

When To Use

• A commit was blocked because the index still carries pending

entries the drain held back.

• The capture backlog has grown and most entries are still pending.
• You want a corpus health report (inert ratio, orphans, topic clusters).
• You want stored research surfaced automatically during sessions.

When NOT to Use

• Ingesting a single new resource: use knowledge-intake.
• Searching stored knowledge ad hoc: use knowledge-locator.
• Tending a digital garden file: use digital-garden-cultivator.

Workflow

1. Analyze (read-only)

uv run python scripts/memory_palace_cli.py index report

Reports total entries, the inert ratio, orphaned captures (entries whose

backing file is gone), the largest topic clusters by domain, and the

top promotion candidates. Writes nothing.

2. Incorporate (dry-run, then apply)

# Dry run: prints promote/archive proposals, writes nothing.
uv run python scripts/memory_palace_cli.py index promote

# Apply: backs up the index under data/backups/, then persists.
uv run python scripts/memory_palace_cli.py index promote --apply

Each pending entry is classified into one action:

• promote: recent, authoritative, or clustered. Gets a real

importance score, a routing type, and maturity seedling -> growing.

• archive: orphaned or older than the archive horizon and never

revisited. Marked archived rather than promoted, following the

principle that unused captures should drain, not accumulate.

• hold: everything else stays pending with no change.

Applying is idempotent: promoted and archived entries are no longer

pending, so a second run proposes nothing new. The dry-run diff is

always shown before --apply writes.

Running these by hand is the exception. --apply runs on every commit

from scripts/precommit_palace_maintenance.sh, so the backlog drains

continuously rather than in occasional sweeps. Reach for the commands

above when a commit is blocked, or when you want the dry-run diff before

the hook decides for you.

3. Committed state must be drained

The commit that carries the index must carry it with zero pending

entries. scripts/check_capture_index_drained.py runs at the end of the

maintenance hook and fails the commit otherwise, and

tests/test_capture_index_artifact.py re-checks the same invariant in

CI so a bypassed hook does not land a backlog.

Two things make that gate reachable rather than a standing block:

• The capture write stages the index

(hooks/shared/deduplication._stage_index). Without it, pre-commit

reverts the unstaged write before any hook runs, so the drain reads a

tree the fresh capture is missing from and converges on a fixed point

that excludes exactly the entries it exists to process. That is how 47

captures accumulated behind a drain that reported nothing to do.

• The drain resolves promote and archive by itself. Only hold survives

it, so a blocked commit means a specific capture needs a person to

score or archive it. The gate names the keys.

4. Surface (learn)

A SessionStart hook (hooks/index_surfacer.py) names the highest-value

promoted captures at the start of a session. It is disabled by default.

Enable it in memory-palace-config.yaml:

feature_flags:
  context_injection: true

The hook only speaks when promoted entries clear the importance floor,

and it exits silently on any error so it can never block a session.

The corpus keyword index is a separate artifact

The three steps above all operate on the capture index at

hooks/memory-palace-index.yaml. Retrieval reads a different file:

data/indexes/keyword-index.yaml, built from the staging captures and

consumed by cache_lookup. Curating one does nothing to the other.

That keyword index is derived data and is not tracked in git, so a

fresh checkout has none at all. Rebuild it with:

# Report what would be indexed, writing nothing.
uv run python scripts/build_indexes.py --dry-run

# Write data/indexes/keyword-index.yaml.
uv run python scripts/build_indexes.py

The builder refuses to write an empty index over a populated one. An

empty corpus is reported with "wrote": false and any existing index

is left untouched. Writing entries: {} over real data is how the

corpus went dark in 1.5.0, and it stayed dark because the regeneration

script named in that stub file had never been written.

When a capture is missing from search

A capture whose frontmatter will not parse contributes nothing to the

keyword index. Body extraction is gated on that parse, so the whole

document drops out rather than just its topic, and the drain above

reports nothing wrong because the index entry itself looks ordinary.

The usual cause is a page title or search query holding a double

quote, which closed the YAML scalar early when the capture was

written. The capture hooks escape their scalars now, so this reaches

captures written before that fix and no others.

# Report which captures cannot be parsed, writing nothing.
uv run python scripts/repair_capture_frontmatter.py

# Re-quote them, backing the originals up under data/backups/.
uv run python scripts/repair_capture_frontmatter.py --apply

The repair rewrites the broken scalar and nothing else, and refuses

any file it cannot re-parse afterward: turning an invisible capture

into a subtly wrong one is worse than leaving it alone. Rebuild the

keyword index once it has run, since retrieval reads the separate

artifact described above.

Design Notes

• Promotion uses only structural signals (recency, domain authority,

cluster size). The decision logic is deterministic. No model call

gates a transition.

• The decay half-lives (14/30/90 days) are tunable priors, not retention

constants. Wixted & Ebbesen (1997) and Murre & Dros (2015) show

forgetting follows a power law. FSRS (Ye, Su & Cao, 2022) validates

exponential decay only with a learned per-item half-life. Calibrate

against reopen logs if usage data accrues.

• Retrieval stays keyword-first (cache_lookup / keyword_index), and

embeddings are not required at the current corpus scale. BM25 is the

workhorse up to ~5000 documents. Embeddings add value only for

vocabulary-mismatch discovery.

• Near-duplicate detection layers SHA-256 exact match (present via

content_hash) then MinHash with k-shingling for near-duplicates

(Broder, 1997). SimHash is preferable only at tens of thousands of

documents.

• Importance formula: `relevance = w1 centrality + w2 decay(t) +

w3 * usage. The plugin ships all three terms (graph_analyzer`

PageRank, decay_model, usage_tracker).

Archiving Completed Work

Triage decides whether a single capture drains or accumulates. When a

whole body of work finishes, freeze it behind an index instead of

deleting it or leaving it in the active listing.

See modules/archive-pattern.md for the structure, the closing-note

requirement, and the two discoverability layers.

Exit Criteria

• [ ] build_indexes.py --dry-run reports a non-zero entry count and

leaves data/indexes/keyword-index.yaml byte-identical.

• [ ] build_indexes.py against an empty corpus reports `"wrote":

false` and leaves an existing populated index untouched.

• [ ] index report runs and prints the inert ratio and orphan count

for the live index.

• [ ] index promote (no flag) prints proposals and writes nothing

(the index file is byte-identical afterward).

• [ ] index promote --apply creates a timestamped backup under

data/backups/ before persisting, and a re-run proposes nothing.

• [ ] With context_injection: true, a SessionStart event surfaces the

top promoted captures, and with the flag off it stays silent.

• [ ] Failure modes (missing index, corrupt YAML, missing backing files)

are handled without raising: report degrades, promote holds, hook

exits silently.

• [ ] check_capture_index_drained.py exits 0 against the committed

index and exits 1 naming the keys when one is left pending.

• [ ] repair_capture_frontmatter.py reports zero repairable

captures against the committed corpus, and refuses a file it

cannot re-parse after repair.

• [ ] A capture written by update_index appears in git diff --cached

without anyone staging it by hand.