Why it matters
Growth teams move fast by testing ideas instead of arguing about them. A test only helps when it is designed well, though, and a badly run test is worse than none because it gives you false confidence.
This station covers writing hypotheses, prioritising ideas, designing A/B tests, the statistics you actually need, the classic mistakes, and how to keep what you learn.
After this station you can
- Write a testable hypothesis with a clear success metric
- Score an idea backlog with ICE or RICE
- Estimate the sample size a test needs before you run it
- Spot peeking, novelty effects and other common errors
Every resource below is free to read or watch online. Links open in a new tab.
4.1 Hypotheses & ideas
| Resource | Type | Level | Time |
|---|---|---|---|
| Optimization Glossary — OptimizelyPlain definitions of the testing terms you will meet everywhere else on this station. | Docs | Beginner | Reference |
| Design an effective hypothesis — Optimizely DocsHow to turn a customer problem into a testable hypothesis with a measurable predicted result. | Docs | Beginner | 10 min |
4.2 Prioritization (ICE, RICE)
| Resource | Type | Level | Time |
|---|---|---|---|
| Growth Snack: Prioritizing High Impact Growth Experiments with ICE — Sean Ellis, The Breakout Growth PodcastThe person who coined ICE scoring explains how to rank experiment ideas quickly. | Podcast | Beginner | 10 min |
| RICE: Simple prioritization for product managers — Sean McBride, IntercomThe original RICE write-up: score ideas by reach, impact, confidence and effort. | Article | Beginner | 10 min |
4.3 A/B test design
| Resource | Type | Level | Time |
|---|---|---|---|
| Decision Making at Netflix — Netflix Technology BlogWhy a company decides with experiments instead of opinions, as the opener of a clear series. | Article | Beginner | 8 min |
| What is an A/B Test? — Netflix Technology BlogControl, treatment, randomization and metrics explained with simple product examples. | Article | Beginner | 8 min |
| The Open Guide to A/B Testing — GrowthBookA full free guide covering test setup, metrics, analysis and running an experimentation program. | Guide | Intermediate | 1.5 h |
4.4 Statistics & sample size
| Resource | Type | Level | Time |
|---|---|---|---|
| Seeing Theory — Brown UniversityInteractive visuals that make probability, distributions and inference intuitive. | Tool | Beginner | 1 h |
| Sample Size Calculator (Evan's Awesome A/B Tools) — Evan MillerWork out how many visitors you need before a test can detect your expected lift. | Tool | Beginner | 5 min |
| Simple Sequential A/B Testing — Evan MillerA simple method that lets you stop a test early without breaking the statistics. | Article | Advanced | 20 min |
4.5 Common pitfalls
| Resource | Type | Level | Time |
|---|---|---|---|
| How Not To Run an A/B Test — Evan MillerShows why peeking at results and stopping at significance produces false winners. | Essay | Intermediate | 10 min |
| Experiments at Airbnb — Airbnb Tech BlogReal cases where context, timing and early stopping misled Airbnb's test readings. | Article | Intermediate | 15 min |
| Seven Pitfalls to Avoid when Running Controlled Experiments on the Web — Ron Kohavi et al., exp-platform.comMicrosoft's experimentation team lists the mistakes that most often invalidate tests. | Article | Advanced | 45 min |
| A/B Testing Intuition Busters — Ron Kohavi, Alex Deng, Lukas VermeerCommon misreadings of p-values, surprising results and test math, with corrections. | Article | Advanced | 1 h |
4.6 Learning from experiments
| Resource | Type | Level | Time |
|---|---|---|---|
| The ultimate guide to A/B testing | Ronny Kohavi — Lenny's PodcastHow most ideas fail, how to build a testing culture, and how to trust what you learn. | Podcast | Intermediate | 1 h 23 min |
| Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained — Ron Kohavi et al., exp-platform.comFive real Bing results that looked wrong, and what digging into them taught the team. | Article | Advanced | 45 min |
Exercise.txt — Design one A/B test
- Pick a page or flow and one change you believe will help.
- Write the hypothesis: because of X, changing Y will improve Z.
- Use a sample size calculator to find how many users and days you need.
- Write down, before launch, what result means ship, what means stop, and what you will do either way.
Not done yet
Installing Experiments.exe
0 of 6 items done
Setup complete. Experiments.exe is installed. Next stop: Station 5.
Saved in this browser only.
Terms in this station
HypothesisICERICEStatistical significanceSample sizePeeking
On the bookshelf
Paid books that go deeper on this station: Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang & Ya Xu. See the full bookshelf.