A/B test

An A/B test shows two versions of a page to different visitors at the same time, and measures which version leads to more of a chosen result, such as sign-ups or purchases. Visitors are split at random, so the only systematic difference between the groups is the version they saw.

What it's good for

An A/B test measures what visitors actually do, not what they say. That makes it the most direct way to find out whether a change improves your conversion rate.

The catch: traffic

To see a real difference between two versions, you need many visitors, and many conversions in both groups. Small improvements need more. A site with a few hundred visitors a month can run a test for months and still not get a clear answer. The same statistics apply as in any test: see how many votes you need for how sample size and margin of error relate.

A/B test or first-impression test?

The two answer different questions:

• An A/B test tells you which version makes more visitors act, on your live site, with your real traffic.
• A first-impression test tells you which version strangers understand and trust better, and why. It works before launch and without traffic.

They work well together. Use a first-impression test to find the versions worth testing, and an A/B test to confirm the effect once you have the traffic. Read did your redesign work? for how to compare two versions on StayOrBounce.