experiment-designDesign product experiments to test hypotheses. Use when the user says "run an experiment", "A/B test this", "build-measure-learn", "validate this assumption", "test this hypothesis", "we don't know if this will work", "minimum viable test", "lean experiment", or wants to reduce risk before fully building a feature - even if they don't explicitly say "experi…
Independent GitHub snapshot · Automated classification · Original documentation governs
name: experiment-design description: > Design product experiments to test hypotheses. Use when the user says "run an experiment", "A/B test this", "build-measure-learn", "validate this assumption", "test this hypothesis", "we don't know if this will work", "minimum viable test", "lean experiment", or wants to reduce risk before fully building a feature
Based on The Lean Startup by Eric Ries. The core insight: startups (and product teams) don't fail from lack of execution - they fail from executing on the wrong plan. The build-measure-learn loop is how you find the right plan faster. Every feature is a hypothesis. Every release is an experiment. Validated learning - not shipped features - is the unit of progress.
Format: "We believe [action/feature] will cause [user behavior change], resulting in [metric change], because [assumption]."
Example: "We believe adding a progress bar to onboarding will cause more users to complete setup, resulting in a 15% increase in activation rate, because users abandon when they can't see how much is left."
The "because" is critical - it forces you to name the assumption you're testing.
Most hypotheses have multiple assumptions embedded. Find the one that, if wrong, makes the whole hypothesis collapse.
List assumptions - rank by risk - test the riskiest one first.
What's the cheapest, fastest way to test the riskiest assumption?
Ries's ladder of experiment types (least to most expensive):
Start as low on the ladder as possible. Only move up when the cheaper test is insufficient.
Set the threshold before you see results. This prevents moving the goalposts.
"We will consider this experiment successful if [metric] changes by [amount] within [time period] with [sample size]."
For A/B tests: use a significance calculator. Undersized tests produce false results.
Whatever the result:
Ries: a failed experiment that teaches you something is more valuable than a successful one that teaches you nothing.
1. HiPPO-driven decisions instead of experiments Bad: "The CEO thinks we should add X, so we're adding X." Good: "The CEO thinks X will increase retention. Let's run a 2-week experiment with 10% of users."
2. Moving the goalposts Bad: Setting a 20% lift as success, seeing 8%, calling it a win anyway. Good: Success criteria set before the experiment runs. Results are binary: met or not met.
3. Testing too many variables at once Bad: Redesigning the entire onboarding and A/B testing it. Good: Change one variable per experiment. Otherwise you can't attribute the result.
4. Skipping the concierge step Bad: Building a full automated feature to test an unvalidated assumption. Good: Do it manually for 10 users first. If the manual version doesn't work, the automated one won't either.