---

name: foundations-measurement-theory description: Audits whether scores, benchmarks, LLM judges, or surveys measure what we think. Use when a benchmark is saturated, a score unreliable, or judge agreement untrustworthy. version: "1.1" last_validated: 2026-09-27

Measurement Theory Foundations

Establish whether an observation supports its intended interpretation and decision. A repeatable score can consistently measure the wrong thing.

When to use

Use when a task asks whether a metric, questionnaire, benchmark, sensor, rating, or composite score measures its intended target, or whether scores can be compared across groups, instruments, or time.

Do not use when the task is only sampling uncertainty, causal identification, choosing an action, or implementing an evaluation pipeline. Those belong respectively to statistical inference, causal inference, decision theory, and ai-evals.

Workflow

1. Define the intended interpretation, population, decision, and cost of measurement error. Identify what is observed and what remains latent.
2. Choose the physical-measurement or psychometric branch in measurement primitives. Do not translate psychometric reliability into metrological traceability, or treat benchmark scores as physical quantities without justification.
3. Audit the instrument with the eight primitives below. Use evidence actually available; mark missing evidence rather than supplying thresholds or validity claims.
4. For score comparisons, read comparability and drift. Preserve instrument versions, scoring changes, administration conditions, and population differences. For LLM benchmarks or LLM-judge scores, also read AI evaluation measurement.
5. Complete the measurement audit template, using the synthetic example only as a format example.

Quick Reference

Eight primitives:

Primitive	Required decision
Construct/measurand	Specify the target, domain, population, and intended use.
Operationalization	Identify observation, instrument, scoring, and construct underrepresentation or irrelevant variation.
Validity	Evaluate evidence for this interpretation and use; never infer validity from reliability alone.
Reliability/precision	Specify what is replicated: items, raters, occasions, tasks, or instruments; estimate the relevant uncertainty. For LLM judges, name the facets (judge, prompt template, order, seed).
Scale/transformations	Identify meaningful comparisons and operations; an arbitrary zero does not support ratio claims.
Error/uncertainty	Separate systematic bias, random error, missingness, and uncertainty in the measurement model.
Calibration/traceability	Identify reference and scope; calibration of a probability forecast is different from metrological calibration.
Invariance/drift	Determine whether the same interpretation survives group, time, setting, and instrument changes. A score that has become a target (a gate, a leaderboard, a training objective) is under selection pressure: treat it as a changed instrument (Goodhart's and Campbell's laws) and re-audit validity before reading later values as the original construct.

Completion criteria

Return a supported, conditional, or insufficient verdict for each intended interpretation, with evidence and restrictions. A verdict is an audit conclusion, not certification. Name the missing evidence and the smallest study or calibration needed to resolve it.

Do not use universal Cronbach alpha cutoffs. Internal consistency alone establishes neither unidimensionality, test-retest stability, agreement, nor validity. Keep item-level ordinal responses separate from assumptions used to analyze a composite.

Navigation

• Measurement primitives — definitions, assumptions, and counterexamples.
• Comparability and drift — cross-group and longitudinal checks.
• AI evaluation measurement: read for LLM benchmarks and judges. Covers facets and G-theory D-studies, linking after judge or benchmark changes, IRT/DIF, saturation, a construct-validity checklist, and judge validation rules.
• For the PPI correction that uses a human gold set to debias judge-labelled data, see statistical inference.
• Completed example — synthetic benchmark audit.
• Output template — deliverable structure.
• Dated primary sources — source scope and verification cutoff.