uai-reproducibility

UAI Reproducibility

UAI's 2026 CFP did not impose a formal reproducibility checklist (one may appear later —

待核实 each cycle), but it strongly encouraged code and data availability and instructed

reviewers to judge whether claims are backed up convincingly. At this venue "convincing"

has a technical meaning: stochastic-inference results carry diagnostics, not just point

estimates. This skill turns that norm into an audit.

The UAI-specific bar

Reproducibility questions at a probabilistic-inference venue go one level deeper than

"can I rerun the script":

• A sampler that reproduces the same posterior mean but different tail quantiles has not

reproduced the paper — report and check the diagnostics that detect this (R-hat, ESS,

divergent transitions where relevant).

• A variational result is reproducible when the ELBO trajectory and the selected

restart are recoverable, not merely the final metric; multi-restart selection rules

must be stated.

• Calibration claims reproduce only if the binning scheme, the split used for

recalibration, and the α grid are all pinned down; empirical coverage moves with all

three.

• Causal-discovery results depend on graph generation as much as on the algorithm:

publish the SCM sampler, noise families, and intervention protocol.

Determinism ledger

Record every randomness source once, in code, and cite it from the paper:

# repro/ledger.py — imported by every experiment entry point
import json, platform, random, numpy as np

def fix_and_log(seed: int, path: str = "run_manifest.json"):
    random.seed(seed)
    np.random.seed(seed)
    manifest = {
        "seed": seed,
        "python": platform.python_version(),
        "numpy": np.__version__,
        "chains": 4, "warmup": 1000, "draws": 2000,   # sampler config lives here
        "elbo_restarts": 10, "restart_rule": "best final ELBO",
    }
    json.dump(manifest, open(path, "w"), indent=2)
    return manifest

The manifest style matters more than the specific fields: one machine-readable file per

run, checked into the artifact, lets a reviewer reconcile the paper's Table 3 with an

actual execution.

Disclosure map

| What must be recoverable | Where it lives at UAI | Common omission |

|---|---|---|

| Model and assumption set | Main part, stated with each theorem/method | Assumptions distributed across three sections |

| Sampler / optimizer settings | Appendix (unlimited, same PDF) | "Default settings" without library version |

| Hyperparameter search space and selection rule | Appendix table | Only the winning configuration reported |

| Seeds and number of repeats | Appendix + artifact manifest | Single-run results with no variance |

| Dataset versions, splits, preprocessing | Appendix + loader script in ZIP | Preprocessing "as in [12]" where [12] is ambiguous |

| Compute (hardware, runtime, memory) | Appendix | Runtime reported only for the proposed method, not baselines |

| Code/data availability statement | Main part or appendix | Silence, which reviewers read as "unavailable" |

Diagnostic quick reference

What "reported convincingly" tends to mean per inference family — as conventions of

the field, not venue mandates:

| Family | Minimum reported | Stronger version |

|---|---|---|

| MCMC | R-hat per parameter block, ESS, chain count/length | Rank plots; comparison against a long-run gold standard |

| Variational | Final ELBO, restart count and rule | ELBO traces; posterior-quality check on a tractable case |

| SMC / particle | Particle count, resampling scheme, ESS trajectory | Variance of the marginal-likelihood estimate over repeats |

| Conformal / intervals | Split sizes, α grid, empirical coverage | Conditional coverage slices; width distribution |

| Calibration | Binning scheme, ECE definition used | Reliability diagrams with confidence bands over seeds |

| Causal discovery | Graph generator, noise family, SHD/SID per seed | Sensitivity to assumption violations (unfaithfulness, confounding) |

If a row's "minimum" column is missing for your method family, expect the backing

criterion to absorb the damage.

Honesty over completeness

• If code cannot be released (industrial constraints, licensed data), say so in the

paper and compensate: fuller pseudocode, exact hyperparameters, synthetic surrogates

for private datasets. The encouraged-not-mandatory wording gives room for honesty, not

for vagueness.

• Report failure modes you observed — initializations that collapse, chains that need

longer warmup on one dataset. Probabilistic-ML reviewers trust papers that know where

their method breaks.

• Never let variance disappear in the retelling: if three of ten seeds underperform,

the aggregate table must reflect it (mean ± sd over all ten, or a stated, principled

selection rule).

Where reproducibility evidence lives

Split by tier deliberately: the availability statement and diagnostic summaries in

the reviewed PDF (body or appendix), where they count toward backing; manifests,

loaders, and per-run logs in the ZIP, where they support spot-checks. Never leave the

only mention of seeds or repeat counts inside the optional archive — reviewers grade

what the PDF says.

Pre-submission reproducibility drill

1. Clone your own artifact onto a machine that never ran the project; follow only the

README.

1. Reproduce the smallest headline number end to end, including the diagnostic that

validates it.

1. Diff the regenerated figure against the paper's; investigate any visible deviation

before a reviewer does.

1. Grep the paper for every "we observe/we find/consistently" and confirm each maps to a

logged run.

1. Write the availability statement last, describing what is actually in the ZIP — not

what was planned in January.

Output format

[Repro grade] turnkey / recoverable with effort / under-specified
[Diagnostics reported] <R-hat/ESS/ELBO/coverage/SHD... as applicable>
[Determinism ledger] present / partial / absent
[Disclosure gaps] <items from the map still missing>
[Availability statement] drafted / needs honesty pass / missing