data-audit

Audit Data References

Scan all notebooks for data file references and verify they exist on disk.

Steps

1. Scan all .ipynb files in notebooks/ for data loading patterns:

• Python: pd.read_csv(...), pd.read_stata(...), pd.read_excel(...), pd.read_parquet(...), open(...), np.loadtxt(...)
• R: read.csv(...), read_csv(...), read.dta(...), haven::read_dta(...), readxl::read_excel(...), load(...)
• Stata: use "...", import delimited "...", import excel "...", insheet using "..."
• Also check the .md Jupytext pairs for the same patterns

1. Extract every referenced file path and normalize it:

• Resolve relative paths from the notebook's directory (notebooks/)
• Resolve paths using DATA_DIR, RAW_DATA_DIR from config.py / config.R

1. Check that each referenced file exists in data/rawData/ or data/

1. Scan data/rawData/ and data/ for all data files present on disk

1. Report three categories:

Resolved — referenced and found:

• File path, which notebook references it, line/cell number

Broken — referenced but not found:

• File path as written in code, which notebook, suggested fix (closest matching file, or note that it may need to be downloaded)

Undocumented — on disk but never referenced by any notebook:

• File path in data/rawData/ or data/ that no notebook loads

1. Print a summary: total references, resolved, broken, undocumented files

Error handling

• If no notebooks exist, report "No notebooks found" and stop.
• If data/rawData/ does not exist, warn but continue checking data/.