A/B Sensei

Welcome grasshopper.

Today we begin your journey to experimentation enlightenment.

Cast off your preconceptions and assumptions. There is no room for complacency or ego in the experimentation dojo.

The art of experimentation is learning to learn.

The Way of the Product Warrior

Hypothesis-driven experimentation is a systematic way of exploring how to get to your vision, via measurable, reproducible results. It can be applied Growth experiments equally as well as it can to UI feature experiments. It is similar to the Scientific Method and shares the same goal: determining cause and effect relationships in a rigorous, objective manner. It is a powerful framework to guide your thinking.

There are 4 stages of a high-quality, controlled experiment:

Designing the experiment
Running the experiment
Analysing the experiment
Rapid iteration

All of these stem from your vision, strategy, and objectives. Once you’ve completed the 3rd stage, your newfound knowledge and insights will inform your next experiments. The cycle of experimentation continues, moving you closer to your goal with every step. Hypothesis-driven roadmaps and rapid iteration are a powerful way of reaching your vision.

Mastering the art of experimentation

Why should we A/B test ideas? And why should we use a rigorous scientific method to A/B test ideas? Read more

How to craft a good hypothesis (while avoiding hippos), and why a hypothesis can never be proven. Read more

How to design a rigorous experiment using the Hypothesis Kit and the "design like you're right, test like you're wrong" principle. We cover the null hypothesis, power analysis, the Minimum Detectable Effect, and how the length of experiment and the number of people you want to expose to it will affect this. Read more

Ramping up and why you shouldn't stop an experiment before it's finished. Read more

How to use Statistical Significance and the p-value to decide if your product change caused a change in your key metric. There's also an introduction to bell curves and conversion rate distributions to give you some background to the p-value (plus some maths, if you're interested). Finally, some tips on how to analyse an experiment objectively, and think like a scientist! Read more

Why your experiment is just the begining, not a means to an end. How should you decide what metrics to use (especially if you want to do no harm). Read more