data-science-experimentationUse this skill when designing experiments, A/B tests, multi-armed bandits, randomized controlled trials, statistical hypothesis tests, sample size calculations, experiment design, power analysis, or causal inference for product changes. This skill enforces: rigorous experiment design with pre-registered hypotheses, proper sample size calculations, statistic…
Independent GitHub snapshot · Automated classification · Original documentation governs
Design and analyze rigorous A/B tests and experiments. Enforce proper sample size planning, pre-registration, statistical methods, guardrail monitoring, and decision frameworks.
Exact user phrases: "A/B test", "experiment", "randomized trial", "sample size", "power analysis", "hypothesis test", "p-value", "multiple testing", "guardrail metric", "experiment design", "treatment effect", "H0", "alternative hypothesis", "statistical significance", "practical significance", "MDE", "minimum detectable effect".
Experiment design with sample size calculation, analysis plan, and decision criteria.
## Experiment Design
Hypothesis: {H0 and H1}
Primary Metric: {name, baseline, MDE}
Sample Size: {N per variant, total N}
Duration: {days} at {traffic allocation}
Analysis: {test type, corrections, covariates}
## Analysis Results
Treatment Effect: {estimate, CI, p-value}
Practical Significance: {effect size, decision}
Guardrails: {√ or ✗ per metric}
Formulate clear, falsifiable hypotheses. H0 (null): no effect. H1 (alternative): effect exists in specified direction. A good hypothesis: "Changing the checkout button color from blue to green will increase purchase conversion rate by at least 0.5 percentage points." Pre-register on experiment platform or in documentation before launch. Include rationale, prior evidence, and expected mechanism of action.
Define one overall evaluation criterion. Must be: sensitive to the change, reliable (low measurement variance), timely (can be measured during experiment window), and directional (higher or lower is clearly better). Examples: purchase conversion rate, revenue per user, retention rate (day 7), task completion rate.
Related success indicators that might be affected. Examples: average order value, session duration, pages per session. Secondary metrics are reported but not used for the go/no-go decision.
Metrics that MUST NOT degrade even if primary metric improves. Examples: page load time, error rate, unsubscribes, support ticket volume, latency p99. Set acceptable thresholds upfront. A treatment that improves the primary metric but degrades a guardrail is usually not shipped.
Define metrics once in a central repository (SQL or Python). Store: metric name, SQL definition, owner, expected range, seasonality, guardrail assignment. Version control definitions. Consistent definitions across experiments enable comparability.
Baseline conversion rate (p0): current value of the metric. Minimum detectable effect (MDE): smallest effect worth detecting. Significance level (α): probability of false positive, typically 0.05. Statistical power (1-β): probability of detecting true effect, target ≥ 0.80. Allocation ratio: 1:1 (most efficient).
n = (Z_(α/2) + Z_β)² × (p1(1-p1) + p2(1-p2)) / (p1 - p2)²
For continuous metrics: n = 2 × (Z_(α/2) + Z_β)² × σ² / δ²
| Baseline | MDE (relative) | α | Power | N per variant |
|---|---|---|---|---|
| 5% | 10% (0.5pp) | 0.05 | 0.80 | ~13,500 |
| 5% | 20% (1.0pp) | 0.05 | 0.80 | ~3,600 |
| 10% | 10% (1.0pp) | 0.05 | 0.80 | ~3,400 |
| 10% | 5% (0.5pp) | 0.05 | 0.80 | ~13,800 |
| 20% | 10% (2.0pp) | 0.05 | 0.80 | ~2,500 |
Simple: each unit assigned independently. Stratified: block by important covariates (country, platform, segment) to reduce variance. Cluster: group-level randomization when network effects exist (markets, regions). Re-randomization: reject unbalanced allocations (if used, must adjust inference).
Monitor daily: chi-square test on observed vs expected allocation ratio. p < 0.05 indicates SRM. Common causes: assignment code bug, caching layers, bot filtering, ad blockers. SRM found → experiment is invalid. Do not analyze primary metric. Fix root cause, re-launch.
Two-sample t-test for continuous metrics. Chi-square test for proportions. Report: point estimate, confidence interval, p-value, effect size (Cohen's d or lift %). Pre-specified analysis plan prevents p-hacking.
Beta-Binomial for proportions (Beta prior + Binomial data → Beta posterior). Normal-Normal for continuous outcomes. Report: posterior mean, credible interval (HDI), probability of direction P(effect > 0), probability of practical significance P(effect > MDE).
FWER (Bonferroni, Holm): use when false positive is catastrophic. FDR (Benjamini-Hochberg): use when discovering true effects is important. Pre-register which corrections apply. Report all tests performed, not just significant ones.
Pre-specified subgroups only (region, platform, user segment). Test via interaction model: Y = β0 + β1×T + β2×S + β3×T×S. Post-hoc discovery uses causal forest or BART but acknowledge inflated false discovery rate.
Coordinate overlapping experiments. Namespace via MD5 hash partitions. Maximum simultaneous experiments per user: 5-10. Conflict resolution: last-writer-wins or explicit priority assignment.
What question are we answering?
├── Did conversion increase?
│ └── Primary: conversion rate (proportion)
├── Did revenue increase?
│ └── Primary: revenue per user (continuous)
├── Did retention improve?
│ └── Primary: D7 retention rate (proportion)
└── Did user engagement change?
└── Primary: sessions per user (count)
Data type?
├── Binary (conversion, click)
│ ├── Two groups → Chi-square or z-test
│ ├── Multiple groups → Chi-square or logistic regression
│ └── Paired → McNemar's test
├── Continuous (revenue, time)
│ ├── Normal, two groups → t-test
│ ├── Normal, multiple groups → ANOVA
│ ├── Non-normal, two groups → Mann-Whitney U
│ └── Non-normal, multiple groups → Kruskal-Wallis
├── Count (sessions, clicks)
│ ├── Low variance → Poisson test
│ └── Overdispersed → Negative binomial
└── Ordinal (rating, tier)
└── Mann-Whitney U or ordered logit
CUPED uses pre-experiment data to explain metric variance, reducing required sample size. Formula: Y_cv = Y - θ × (X - μ_X). θ = Cov(Y, X) / Var(X). Variance reduction = Corr(Y, X)². Typical reduction: 20-50% for metrics with strong pre-experiment correlation.
def cuped_adjust(treatment_metric, control_metric, pre_treatment, pre_control):
# Pool pre-experiment data (both variants before treatment)
pre_pooled = np.concatenate([pre_treatment, pre_control])
post_pooled = np.concatenate([treatment_metric, control_metric])
# Calculate theta
theta = np.cov(pre_pooled, post_pooled)[0, 1] / np.var(pre_pooled)
# Adjust metrics, variance_reduction = corr(pre, post)^2
mu_x = np.mean(pre_pooled)
treatment_adjusted = treatment_metric - theta * (pre_treatment - mu_x)
control_adjusted = control_metric - theta * (pre_control - mu_x)
return treatment_adjusted, control_adjusted
# CUPED with multiple covariates
# Y_cv = Y - Σ(θ_i × (X_i - μ_i))
# Use pre-period metric, user-level features, segment indicators
High correlation between pre/post metric (r > 0.5). Stable user behavior (don't use for completely new features). Metric measured at same unit level (user-level metric → user-level covariate). Less effective for: rare events (conversion < 1%), metrics with high individual volatility.
Allows continuous monitoring without inflating false positive rate. Uses a mixing distribution over effect sizes. Decision boundary widens over time, maintaining valid type I error at any stopping time.
# mSPRT for normal data (continuous metrics)
def msprt_normal(treatment_data, control_data, variance=None):
n = len(treatment_data)
m = len(control_data)
theta_hat = np.mean(treatment_data) - np.mean(control_data)
se = np.sqrt(np.var(treatment_data)/n + np.var(control_data)/m) if variance is None else variance
# Log-likelihood ratio integrated over normal prior N(0, tau²)
tau = 0.2 * se # Mixing variance (tune based on MDE)
z = theta_hat / se
lrt = (tau**2 / (se**2 + tau**2)) * np.exp(
(z**2 * tau**2) / (2 * (se**2 + tau**2))
)
# Reject H0 when LRT crosses threshold (typically 1/e ≈ 0.368)
return lrt > np.exp(-1)
Use sequential testing for: long-running experiments, high-traffic products, safety-critical changes. Configure: expected effect size, alpha spending function, maximum sample size. Tools: Google's CORE, Optimizely Sequential, custom mSPRT implementation.
def sample_size_proportion(p0, mde_relative, alpha=0.05, power=0.80, ratio=1.0):
p1 = p0 * (1 + mde_relative)
z_alpha = stats.norm.ppf(1 - alpha/2)
z_beta = stats.norm.ppf(power)
p_bar = (p0 + p1 * ratio) / (1 + ratio)
n = ((z_alpha * np.sqrt(p_bar * (1 - p_bar) * (1 + 1/ratio))
+ z_beta * np.sqrt(p0 * (1 - p0) / ratio + p1 * (1 - p1)))**2
/ (p1 - p0)**2)
return int(np.ceil(n))
def sample_size_continuous(mu, sigma, mde_absolute, alpha=0.05, power=0.80):
z_alpha = stats.norm.ppf(1 - alpha/2)
z_beta = stats.norm.ppf(power)
n = 2 * (z_alpha + z_beta)**2 * sigma**2 / mde_absolute**2
return int(np.ceil(n))
# Example: baseline 10%, MDE 5% relative (0.5pp), α=0.05, power=0.80
# sample_size_proportion(0.10, 0.05) → ~13,800 per variant
Ratio metrics (revenue per user, CTR = clicks/impressions) need special variance estimation. Delta method: Var(R) ≈ (μ_y/μ_x)² × (Var(y)/μ_y² + Var(x)/μ_x² - 2Cov(x,y)/(μ_x·μ_y))
def delta_method_variance(a, b):
# For ratio metric R = a / b (e.g., revenue per session)
mu_a, mu_b = np.mean(a), np.mean(b)
n = len(a)
var_a = np.var(a, ddof=1)
var_b = np.var(b, ddof=1)
cov_ab = np.cov(a, b, ddof=1)[0, 1]
r = mu_a / mu_b
var_r = (r**2) * (var_a/mu_a**2 + var_b/mu_b**2 - 2*cov_ab/(mu_a*mu_b))
se_r = np.sqrt(var_r / n)
ci_lower = r - 1.96 * se_r
ci_upper = r + 1.96 * se_r
return r, se_r, ci_lower, ci_upper
When delta method assumptions fail (small samples, skewed distributions): resample (user, sessions, revenue) units with replacement, compute ratio per resample, use percentile CI. Minimum 1000 bootstrap replicates for stable CI.
Explore/exploit when: opportunity cost of exploration is real (revenue per impression), many variants (>5), metric is immediate (clicks, not retention). Don't use MAB: small sample sizes, delayed metrics, need hypothesis testing.
def thompson_sample(clicks, impressions, n_samples=1000):
# Beta-Binomial: prior Beta(1,1), posterior Beta(1+clicks, 1+impressions-clicks)
posteriors = [stats.beta(1+c, 1+i-c).rvs(n_samples)
for c, i in zip(clicks, impressions)]
# Choose variant with highest sampled value
return np.argmax(np.mean(posteriors, axis=1))
experiment_platform:
components:
- assignment_service:
description: "Deterministic variant assignment via consistent hashing"
implementation: "MD5(user_id + experiment_id) % 1000 → variant"
features:
- "Sticky assignment across sessions"
- "Stratified randomization by segments"
- "Exclusion/inclusion rules"
- event_pipeline:
description: "Capture and process experiment events"
components:
- "Client-side SDK (web, mobile, server)"
- "Event ingestion (Kafka/Kinesis)"
- "Stream processing (Flink/Spark) for real-time metrics"
- "Batch processing (daily/hourly) for final metrics"
- metric_calculation:
description: "Compute experiment metrics from raw events"
requirements:
- "Standardize metric definitions in SQL"
- "Compute per-user metric deltas"
- "CUPED adjustment layer"
- "Ratio metric calculation with delta method"
- analysis_service:
description: "Statistical analysis and reporting"
features:
- "Frequentist + Bayesian analysis"
- "Sequential testing (mSPRT)"
- "SRM detection (chi-square)"
- "Multiple testing correction"
- "Heterogeneity analysis (CATE estimation)"
- results_dashboard:
description: "Self-serve experiment results"
features:
- "Primary metric: point estimate + CI + p-value"
- "Guardrail metrics: pass/warn/fail status"
- "Sample size tracker: actual vs target"
- "Experiment timeline and decision log"
- "Signals: SRM alerts, guardrail breaches"
experiment_events_schema:
# Core experiment assignment table
experiment_assignments:
columns:
- experiment_id: STRING
- variant_id: STRING
- user_id: STRING
- assignment_timestamp: TIMESTAMP
- stratification_factors: MAP<STRING, STRING>
- is_control: BOOLEAN
partition: "(event_date)"
clustering: "(experiment_id)"
# Metric events joined with assignments
experiment_metrics:
columns:
- event_id: STRING
- user_id: STRING
- experiment_id: STRING
- variant_id: STRING
- metric_name: STRING
- metric_value: FLOAT
- metric_timestamp: TIMESTAMP
partition: "(event_date)"
clustering: "(experiment_id, metric_name)"
# Analysis results table
experiment_results:
columns:
- experiment_id: STRING
- run_timestamp: TIMESTAMP
- metric_name: STRING
- variant_id: STRING
- treatment_effect: FLOAT
- confidence_interval: STRUCT<lower FLOAT, upper FLOAT>
- p_value: FLOAT
- effect_size: FLOAT
- sample_size: INT
- is_guardrail: BOOLEAN
- guardrail_passed: BOOLEAN
- srm_p_value: FLOAT
Metric variability too high to detect MDE?
├── Pre-experiment data available for same metric
│ └── CUPED (reduces variance by correlation^2)
├── Pre-experiment data available for covariates
│ └── CUPED with multiple covariates
├── Stratification feasible (country, platform, segment)
│ └── Stratified randomization + stratified analysis
├── ML model predicting the metric available
│ └── CUPED with model predictions as covariates
└── Sequential monitoring needed
└── mSPRT (always valid inference at any stopping time)
When can we stop the experiment?
├── Sample size reached AND minimum duration elapsed
│ └── If using fixed-horizon test → run full analysis
├── Sequential test boundary crossed
│ └── Stop early — result is valid (mSPRT)
├── Bayesian probability of success > 95%
│ └── Consider stopping if opportunity cost is high
├── Guardrail metric breached
│ └── Stop experiment — roll back treatment
└── SRM detected
└── Stop experiment — invalid data, investigate root cause
Experimentation Design
├── Traffic volume?
│ ├── High (> 100k users/day) → Classic A/B test (frequentist)
│ ├── Medium → Bayesian A/B test (prior-informed)
│ └── Low (< 10k users/day) → Sequential testing (mSPRT)
├── Number of variants?
│ ├── 2 → A/B test (simplest, highest power)
│ ├── 3-5 → A/B/n with Bonferroni correction
│ └── > 5 → Multi-armed bandit (MAB) with Thompson sampling
├── Metric type?
│ ├── Ratio (revenue/user) → Delta method or bootstrap
│ ├── Binary (conversion) → Chi-squared / Fisher's exact
│ └── Continuous (session time) → t-test / Welch's test
└── CUPED implementation?
├── Available pre-experiment data → CUPED (30%+ variance reduction)
└── No pre-experiment data → Classic frequentist
Decision criteria: Balance statistical power, traffic allocation, and operational complexity.
# experimentation/frequentist_ab_test.py
import numpy as np
from scipy import stats
class ABTestAnalyzer:
def __init__(self, alpha: float = 0.05):
self.alpha = alpha
def analyze_conversion(self, control: np.ndarray, treatment: np.ndarray) -> dict:
n_c, x_c = len(control), control.sum()
n_t, x_t = len(treatment), treatment.sum()
p_c, p_t = x_c / n_c, x_t / n_t
se = np.sqrt(p_c * (1 - p_c) / n_c + p_t * (1 - p_t) / n_t)
z = (p_t - p_c) / se
p_value = 2 * (1 - stats.norm.cdf(abs(z)))
ci = (p_t - p_c) + np.array([-1, 1]) * stats.norm.ppf(1 - self.alpha / 2) * se
return {
"lift": p_t - p_c,
"relative_lift": (p_t - p_c) / p_c,
"p_value": p_value,
"ci_95": ci.tolist(),
"significant": p_value < self.alpha
}
# experimentation/bayesian_ab_test.py
import pymc as pm
class BayesianABTest:
def run(self, control: list[int], treatment: list[int]):
with pm.Model():
p_c = pm.Beta("p_c", alpha=1, beta=1)
p_t = pm.Beta("p_t", alpha=1, beta=1)
obs_c = pm.Binomial("obs_c", n=len(control), p=p_c, observed=sum(control))
obs_t = pm.Binomial("obs_t", n=len(treatment), p=p_t, observed=sum(treatment))
lift = pm.Deterministic("lift", p_t - p_c)
trace = pm.sample(2000, tune=1000)
prob_lift = (trace["lift"] > 0).mean()
return {"prob_treatment_winning": float(prob_lift), "trace": trace}
| Anti-Pattern | Consequence | Solution |
|---|---|---|
| Peeking without sequential boundaries | Inflated false positive rate | Use mSPRT or fixed-horizon test |
| Ignoring SRM check | Invalid experiment results | Daily SRM check, stop experiment if failing |
| Analyzing all users (not just exposed) | Diluted treatment effect | Filter to exposed users by assignment |
| Multiple primary metrics | Cherry-picking significant ones | Pre-register single primary metric |
| Novelty effect bias | Early positive effect reverses | Run for minimum 2 full business cycles |
analytics-engineering for downstream metric pipeline
ml-modeling for model-based personalization experiments