LLM evaluation harness with real statistical power analysis
Bootstrap confidence intervals, paired comparison tests, minimum-detectable-effect calculation, and sequential testing with alpha spending, with loud warnings when a reported model difference is within noise.
Most eval tooling reports a bare percentage on 200 examples and calls it a result. Building the version that reports uncertainty properly is a sharp, defensible differentiator for a portfolio, and it is the same calibration theme running through H4: a number without a confidence interval is a confident assertion the tool itself hasn't earned.
An eval harness that reports bootstrap confidence intervals and paired comparison tests instead of a bare accuracy percentage, computes minimum-detectable-effect sizes so users know how many eval examples they'd need to trust a claimed difference, and supports sequential testing with alpha spending so a run can stop early once a result is statistically conclusive rather than running a fixed, possibly under- or over-powered sample size. Warns loudly, not quietly, when a reported difference between two models is within noise.
All four pieces are implemented and validated against the ground-truth layer described below:
| deliverable | module | validated against |
|---|---|---|
| Bootstrap CI (single model, paired difference) | src/eval_power/bootstrap.py |
Oracle coverage checks; percentile method chosen as default from measured coverage across configurations, not textbook assumption |
| Paired comparison tests (McNemar, bootstrap) | src/eval_power/paired_test.py, src/eval_power/exact_power.py |
Exact enumeration of the discordant-pair distribution + Monte Carlo oracle cross-check |
| Minimum detectable effect | src/eval_power/mde.py |
Bisection over the exact power engine, closed-loop verified against the Monte Carlo oracle |
| Sequential testing with alpha spending | src/eval_power/alpha_spending.py, src/eval_power/sequential_test.py |
True false-positive rate verified by simulating the full repeated-peeking procedure, not just the spending function in isolation |
Several of these validation passes overturned an initial assumption rather than just confirming it — e.g. the bootstrap CI's default method, McNemar's continuity-correction default, and a real parametrization bug in the alpha-spending boundary derivation that only the end-to-end simulation caught. See each module's docstring and the corresponding test file in tests/ for the numbers behind those calls.
Python, SciPy, statsmodels, pandas, Typer
Every statistical procedure is validated by simulation: a nominal 95% confidence interval must achieve 95% empirical coverage across thousands of synthetic evaluation runs with a known true effect size, or the implementation is wrong.
See GROUND_TRUTH.md for the full account of the oracle
and which build checkpoints are worth auditing closely.
This project is part of Project Bathtub's
Phase 1 data collection. It is specifically scoped to stress H4 (calibration / overconfident assertion) and H1 (false verification claims),
and every build session run against it is logged by a Stop-hook harness
(.claude/settings.json -> _harness/log_event.py) so failure-mode
occurrence rates can eventually be measured empirically rather than assumed.
See bathtub.manifest.json for the research metadata and
GROUND_TRUTH.md for the correctness oracle this project's failure-mode
data depends on.
implemented — the ground-truth verification layer and all four planned
deliverables (bootstrap CI, paired comparison tests, minimum-detectable-effect
calculation, sequential testing with alpha spending) are built and validated;
41 tests pass under pytest. See bathtub.manifest.json for machine-readable
status.
Part of Project Bathtub — applying FMEA and survival analysis to AI coding agent failure modes. Phase 0 paper: 10.5281/zenodo.21607993.