4 Examples of A/B Testing

John Spacey, updated on March 15, 2024

A/B testing is a hypothesis testing method that compares the results of two experiment configurations. In many cases, a control is compared against an experiment. This is popularly used to test media such as mobile apps, advertisements or films. A/B testing can be conducted manually or can be highly automated. The following are illustrative examples.

Marketing

A digital advertising platform automatically creates different versions of an ad to test which generates higher conversions. Variations are continually run in competition with each other to optimize the ad.

Films

A film develops 7 different trailers. As part of their marketing efforts, they bring in focus groups and randomly assign them to watch one of the trailers. They then survey each person to gauge how likely they are to pay to see the movie.

Products

A consumer goods company sets up a blind taste test for a breakfast cereal that compares their existing product to a potential new and improved formulation.

Process

An artist experiments with painting at night or painting early in the morning to see which produces more works they deem worthy of an upcoming gallery exhibition. This is an example of a qualitative A/B test that doesn't produce an objective measurement but may have value nonetheless.

Overview

A/B testing is often used as an informal type of experiment that simply compares two versions of something. This is analogous to more formal experiments that randomly assign participants into control and experimental groups.

Summary: A/B Testing
Area	Experiments Problem Solving Decision Making
Definition	A hypothesis test that compares two alternatives, often an experiment and a control.
Value	Making decisions and solving problems with data as opposed to guesses.
Risks	A focus on low level optimization can cause a firm to miss big-picture opportunities.
Synonyms	Split Testing Bucket Testing
Related Concepts	Test and Learn