Marketing
Saved & recent

A/B testing

Also called: A/B test, split testing, randomized experiment

An A/B test randomly assigns eligible units to different experiences and compares prespecified outcomes. Concurrent groups help distinguish the change from time trends when assignment, measurement and implementation are sound.

A hypothetical test might record 50 purchases among 1,000 visitors in A and 60 among 1,000 in B. The observed one-percentage-point gap is an estimate; uncertainty and stopping rules still matter.

Read first
Conversion rate
Related
Conversion rate, Incrementality
Further reading
Microsoft Research — Online Experimentation at Microsoft
2,000 randomly assigned visitors 0% 2% 4% 6% purchase rate A · 1,000 visitors 5% 50 purchases B · 1,000 visitors 6% 60 purchases observed · +1 percentage point +20% relative · uncertainty not shown
Hypothetical randomized groups with the same follow-up and a prespecified purchase outcome: 50/1,000 = 5% versus 60/1,000 = 6%. The observed difference is one percentage point, or 20% relative. Uncertainty and stopping rules still matter; no winner is declared.

View in the full Marketing collection