---

name: foundations-statistical-inference description: "Tests whether data supports a claim: A/B significance, peeking, sample size, eval error bars, multiple comparisons. Use when deciding if results are significant." version: "1.1" last_validated: 2026-09-17

Statistical Inference Foundations

Turn observations into appropriately bounded claims. A numerical estimate is incomplete without its population, estimand, sampling mechanism, independent unit, uncertainty, and assumptions.

Triggers: "is this A/B test uplift significant or noise", "can we stop the experiment early / we peeked daily", "how many users or eval items do we need", "error bars on eval or benchmark scores", "is model B really better than A", "CI for pass rate graded by an LLM judge", "we tested 20 metrics", sample-ratio mismatch, CUPED, ratio metrics, confidence or credible intervals, calibration, conformal prediction.

Use measurement theory for whether the instrument measures the intended construct, causal inference for identification of intervention effects, and decision theory for choosing actions given uncertainty. This skill owns inference contracts; applied evaluation workflows remain with ai-evals. Do not create causal claims from statistical significance.

Quick Reference

Need	Load
Population, estimator, confidence or credible interval	Sampling and estimation
Precision, selection, repeated looks	Design and sequential inference
Forecast uncertainty or conformal threshold	Predictive calibration
A/B test validity (SRM), CUPED, ratio metrics, peeking, sequential stopping	A/B test craft
Holm vs BH vs BY; FWER vs FDR	Multiplicity procedures
Eval scores: clustered SE, paired model comparison, eval size	Eval error bars
LLM-judge labels plus a small human gold set	Judge-assisted inference (PPI/PPI++)
CQR, per-group, covariate-shift, drift, or risk-control conformal	Conformal variants

Workflow

1. Define the target population, estimand, units, time window, data provenance, selection/missingness mechanism, and independent sampling or randomization unit. Count users or clusters when runs are dependent; repeated outputs are not automatically independent evidence.
2. Read sampling and estimation for descriptive, frequentist, or Bayesian inference. Select a method that matches dependence, outcome support, and available sample size. State modeling choices before interpreting results.
   • Most labels come from an LLM judge or classifier: read judge-assisted inference; a judge-only mean is not an estimate of the human-label quantity.
   • Eval or benchmark scores, or model A vs B on shared items: read eval error bars; pair, and cluster related items.
3. Read design and sequential inference when power, multiple hypotheses, stopping, or repeated comparisons matter. Declare the hypothesis family, minimum meaningful effect, and stopping rule. Separate exploration from confirmatory claims.
   • Online experiment: run the SRM and unit checks in A/B test craft before reading any effect, and pick fixed-horizon, group-sequential, or always-valid stopping before launch.
   • More than one hypothesis: pick FWER or FDR and the procedure in multiplicity procedures.
4. Read predictive calibration for predictive intervals, probabilistic forecasts, or conformal sets. Check training/calibration separation and exchangeability before claiming marginal coverage. For per-group coverage, drift, covariate shift, or an error-rate target (abstention), choose from conformal variants.
5. Produce an inference report, adapting its detail to the request. Report effect magnitude, uncertainty, assumptions, diagnostics, and what the analysis cannot establish. A null result can be imprecise; equivalence needs an explicit margin and appropriate test or interval.

Interpretation constraints

• A p-value is not the probability a hypothesis is true; significance does not measure effect magnitude or practical importance [ASA2016].
• A frequentist confidence interval has repeated-sampling coverage under its assumptions. A Bayesian credible interval is posterior probability conditional on the model and prior; neither automatically covers future observations.
• More samples reduce sampling error under the model; they do not repair selection bias, invalid measurement, leakage, or dependence.
• Ordinary fixed-horizon intervals are not generally valid after data-dependent repeated stopping. Anytime-valid methods require their own null, filtration, and process assumptions [Ramdas2023].
• Conformal coverage is marginal over calibration and future observations under exchangeability. It does not ensure every group or individual has nominal coverage [AngelopoulosBates2022].
• Do not translate a standardized effect size into a conversion multiplier without the outcome model and baseline needed for that conversion.
• Correlated voters are fewer votes than they count. For a panel of judges, reviewers, or refuters, estimate the effective number of independent votes from their pairwise error correlation (the design-effect logic of [Kish1965]) before crediting agreement as confirmation. An abstract-level 2026 report on LLM judge panels found several judges from several model families worth far fewer effective votes, with the best single judge matching the panel [JudgePanelEffectiveVotes2026, unreplicated]. Stop adding voters when the effective number stops rising; change the evidence source instead.

Helper

split_conformal_quantile.py computes only the exact finite-sample split-conformal threshold; it cannot check exchangeability or certify coverage. Interface in predictive calibration; tests: python3 scripts/test_split_conformal_quantile.py.

Completion criteria

The report names population, estimand, independent unit, method, assumptions, uncertainty, multiplicity/stopping policy, missing evidence, and supported interpretation. Any numerical helper result is reproducible from supplied inputs. Disclose exploratory or model-conditional conclusions and avoid treating structural validation as demonstrated agent performance.

Verification Notes

Verify the precise theorem and assumptions against the primary source before asserting a guarantee. Worked values in the references are illustrative re-derivations, not empirical effect estimates; re-check package versions (ppi-python, MAPIE) before quoting them.

Navigation

• Sampling and estimation: inferential targets and interval interpretation.
• Design and sequential inference: precision, multiplicity, and stopping.
• Predictive calibration: coverage assumptions and full helper interface.
• A/B test craft: SRM, CUPED, delta method, heavy tails, stopping designs, peeking remediation.
• Multiplicity procedures: Holm, BH, BY, e-BH, gatekeeping.
• Eval error bars: clustered and paired eval inference, eval sizing.
• Judge-assisted inference: PPI/PPI++ for LLM-judge-labelled data.
• Judge, prompt and run variance (facets) that an item-only interval omits: see measurement theory.
• Conformal variants: CQR, Mondrian, weighted, adaptive, risk control.
• Inference report: output template.

Source IDs above resolve to dated primary records in sources.json. No claim of exhaustive coverage of the research literature.

Related: causal inference, decision theory, AI evaluations.