github-sensitive-data-cleanup

GitHub Sensitive Data Cleanup

Overview

This skill guides you through safely removing sensitive data from a Git

repository's history and pushing the cleaned history to GitHub. It encodes the

hard-won lessons from real incidents: scan first, backup before rewriting,

verify after rewriting, and never force-push to a public repo without checking

its visibility and fork count.

The bundled scripts automate the mechanical parts:

• scripts/scan_repo.py — scan the repo for secrets and private context.
• scripts/rewrite_history.py — create a backup and rewrite history with

git-filter-repo.

• scripts/verify_cleanup.py — confirm the sensitive content is gone.
• scripts/safe_push.py — verify repo visibility and push safely.

This skill is conservative by design. If any safety check fails, it stops

and asks for human confirmation rather than continuing.

When to Use This Skill

Trigger this skill when the user:

• Says "scan sensitive data", "扫描敏感信息", "看看仓库有没有泄露".
• Wants to "clean git history", "sanitize history", "rewrite history",

"remove secrets from history".

• Has accidentally pushed a secret, private domain, internal IP, or PII to a

public repository.

• Is about to force-push to a public repository (even without sensitive data).
• Mentions git filter-repo, BFG, git-filter-branch, or history rewrite.

Prerequisites

Install these tools once per machine:

# git-filter-repo (modern replacement for git-filter-branch)
brew install git-filter-repo

# gitleaks (secret scanner)
brew install gitleaks

# GitHub CLI
brew install gh

The scripts assume git-filter-repo and gitleaks are on PATH. The skill

will check this before running destructive operations.

Safety Rules (Non-Negotiable)

1. Scan before you decide. Never rewrite history based on a hunch.
2. Create a backup before rewriting. Use git bundle or a fresh bare clone.
3. Verify repo visibility with gh repo view before any push. Do not infer

public/private from the URL or directory name.

1. Never use --no-verify to bypass hooks. If the PII Guard hook fails,

fix the underlying issue or add an allowlist; do not bypass.

1. Use --force-with-lease first. Fall back to --force only if the

remote ref is stale because of the rewrite itself.

1. Verify after rewriting. A clean git log is not enough; re-run the

scanner and do an AI semantic review.

1. Public repos with forks need extra care. Every fork keeps a copy of the

old history. Coordinate with fork owners if the leaked data is high-risk.

Workflow

Step 0: Confirm the repo path and current branch

cd /path/to/repo
git status --short
git remote -v

Step 1: Scan for sensitive data

Run the scanner to find what needs to be removed:

uv run --with gitpython scripts/scan_repo.py --repo /path/to/repo --output /tmp/scan-report.json

The scanner auto-loads repo-specific patterns from .pii-patterns in the repo

root. If that file contains real private domains, do not commit it — add it

to .gitignore or keep it outside the repo. rewrite_history.py will abort if

the working tree has untracked files.

To enable Layer 3 (private infrastructure context from your gitleaks config and

an optional identities file):

uv run --with gitpython scripts/scan_repo.py \
  --repo /path/to/repo \
  --gitleaks-config ~/scripts/git-pii-guard/gitleaks.toml \
  --identities-file ~/.config/github-sensitive-data-cleanup/identities.txt \
  --output /tmp/scan-report.json

The --gitleaks-config flag reads private-domain-context and

private-ip-context rules from your private gitleaks config. The real patterns

stay in your private config; nothing is copied into this public skill.

Review /tmp/scan-report.json. It includes:

• gitleaks findings (secrets, API keys, tokens).
• Custom pattern matches (internal IPs, phone numbers, PII).
• Layer 3 context matches (private domains, IPs, identities from your config).
• A reminder to do an AI semantic review for content that regex cannot catch.

If nothing sensitive is found, stop. Do not rewrite history.

Step 1.5: AI semantic review (Layer 4)

Regex scanners (Layers 1-3) cannot catch novel private context: real names,

project codenames, transcript snippets, internal meeting references, or

architecture descriptions. You must do an AI semantic review.

Use the prompt in references/ai_semantic_review_prompt.md on the flagged

commits. Re-run the review until no new private context is found.

If you skip this step, you may push private context that gitleaks never knew to

look for.

Step 2: Classify findings and choose a remediation

For each finding, decide:

• Rotate the credential (always do this for live secrets first).
• Remove from history (for private domains/IPs, PII, or already-rotated

secrets that still reveal internal context).

• Add to .gitignore or allowlist (for false positives only).

Live secrets must be rotated before history cleanup. Removing history does

not invalidate a secret that has already been exposed.

Step 3: Prepare a replacements file

Create a text file with one replacement per line in git-filter-repo

--replace-text format:

literal:internal.example.com==>example.com
literal:private.example.org==>example.org
literal:sk-example-aaaaaaaaaaaaaaaa==>sk-example-REDACTED

Replace these with your actual sensitive strings. Do not commit the real

values; keep the replacements file outside the repository.

Use literal: for exact string matches. For regex replacements, use

regex: (only if you are confident in the pattern).

Save this file outside the repo, e.g. /tmp/sensitive-replacements.txt.

Step 4: Create a backup

uv run scripts/rewrite_history.py --repo /path/to/repo \
  --replacements /tmp/sensitive-replacements.txt \
  --backup /tmp/repo-backup.bundle \
  --yes

# Entity leaks live in commit MESSAGES too, not just file content. Cover both:
uv run scripts/rewrite_history.py --repo /path/to/repo \
  --replacements /tmp/sensitive-replacements.txt \
  --message-replacements /tmp/sensitive-replacements.txt \
  --backup /tmp/repo-backup.bundle \
  --yes

This script:

1. Verifies git-filter-repo is installed and executable.
2. Checks that the working tree is clean (no uncommitted changes or untracked

files). If not, aborts.

1. Creates a git bundle backup of the current state.
2. Verifies the backup bundle with git bundle verify.
3. Runs git filter-repo --replace-text. When --message-replacements is

given, it also runs --replace-message so commit messages are rewritten,

not just file blobs — a cleanup that only covers blobs can leave the

entity naming itself in a commit message.

1. Reports the old and new commit hashes.

If the backup or verification step fails, the script stops. Do not proceed

manually.

Step 5: Verify the cleanup

uv run scripts/verify_cleanup.py --repo /path/to/repo --replacements /tmp/sensitive-replacements.txt

This re-runs the scanner and also checks that none of the original sensitive

strings remain in any commit. If it finds anything, go back to Step 3.

Step 6: Check visibility and push

uv run scripts/safe_push.py --repo /path/to/repo --remote origin --branch main

This script:

1. Runs gh repo view to confirm visibility, isPrivate, and forks.
2. Warns loudly if the repo is public and has forks.
3. Uses --force-with-lease first.
4. Falls back to --force only if the remote ref is stale because of the

local rewrite.

1. Refuses to add --no-verify.

If the PII Guard hook fails, fix the issue and re-run. Do not bypass.

Step 7: Post-push verification

After the push succeeds:

1. Open the repo on GitHub and confirm the sensitive strings are gone from

commit history.

1. Check that open PRs still target valid commits. Rewriting history may break

existing PR branches.

1. Notify any fork owners for high-risk leaks.

What the Bundled Scripts Do

scripts/scan_repo.py

Runs gitleaks and a custom bash/grep layer for patterns that gitleaks does

not cover (private domains, internal IPs, Chinese phone numbers, certain PII).

Outputs a JSON report.

uv run --with gitpython scripts/scan_repo.py --repo /path/to/repo --output /tmp/report.json

scripts/rewrite_history.py

Creates a backup bundle and runs git filter-repo --replace-text. Pass

--message-replacements <file> to also rewrite commit messages via

--replace-message (the same replacements file usually covers both).

uv run --with gitpython scripts/rewrite_history.py \
  --repo /path/to/repo \
  --replacements /tmp/sensitive-replacements.txt \
  --message-replacements /tmp/sensitive-replacements.txt \
  --backup /tmp/repo-backup.bundle \
  --yes

scripts/verify_cleanup.py

Re-runs the scanner and greps all commits for the original sensitive strings,

covering both blob content (git grep over every commit) and commit messages

(git log over all refs with a hash-annotated record format), so a rewrite

that missed --replace-message still fails verification.

uv run --with gitpython scripts/verify_cleanup.py \
  --repo /path/to/repo \
  --replacements /tmp/sensitive-replacements.txt

scripts/safe_push.py

Checks visibility and pushes safely.

uv run --with gitpython scripts/safe_push.py --repo /path/to/repo --remote origin --branch main

Handling Special Cases

The repo has open PRs

Rewriting history invalidates commit refs in open PRs. After push:

1. Ask PR authors to rebase their branches onto the new main.
2. If the PR is yours, delete the local branch, fetch the rewritten main,

and cherry-pick the changes as new commits.

The repo has forks

Public forks retain the old history until their owners sync. For high-risk

leaks (live secrets, production credentials), consider:

1. Rotating the credential immediately (mandatory).
2. Asking GitHub Support to remove cached views of the sensitive data.
3. Notifying fork owners with a brief, factual message.

For lower-risk leaks (internal domain names, placeholder IPs), document the

rewrite and move on.

git filter-repo reports "need a fresh clone"

git-filter-repo refuses to run on repos with multiple remotes or non-origin

refs. To fix:

git clone --mirror /path/to/repo /tmp/repo-mirror.git
cd /tmp/repo-mirror.git
# run rewrite_history.py against the mirror

gitleaks false positives

If gitleaks flags documentation examples or test fixtures, add an allowlist

entry to the repo's .gitleaks.toml or .gitleaksignore (never use

--no-verify). See references/tooling_notes.md for allowlist patterns.

What This Skill Does NOT Do

• It does not rotate live credentials for you. Rotate first, clean history

second.

• It does not remove data from GitHub's own backups or forks. It only cleans

the upstream repository history.

• It does not bypass git hooks. If a hook fails, fix the root cause.
• It does not make secret leaks "safe." Once pushed, assume the data was seen.

References

• references/incident-lessons.md — what went wrong in real cleanups and how

this skill prevents those mistakes.

• references/tooling_notes.md — choosing between git-filter-repo and BFG,

allowlist patterns, and common errors.

• references/ai_semantic_review_prompt.md — Layer 4 AI semantic review prompt

for finding private context that regex cannot catch.