A/B Test: Definition, Process, & Example

A/B Testing – What Does This Term Mean?

Ads, a landing page, or simply a CTA—you often have multiple ideas and approaches, not to mention target audiences that expect different ways of being addressed. An A/B test can help determine which one works best by testing two or even more versions.

For example, visitors to a website are randomly directed to Variant A or Variant B. The predefined goal—more inquiries, more purchases, more sign-ups—is then measured. The variant that achieves the better metric is the clear winner of the A/B test.

To ensure the results are reliable, two basic rules apply:

Only one thing is changed per test; otherwise, no one will know which change caused which effect.

The test runs until clear data is available. A lead after two days could simply be a coincidence, whereas after two months it’s a clear result. Technically, this is referred to as statistical significance.

Almost anything that visitors see or can interact with can be tested, including:

Headings
Images
Call-to-action
Forms
Order of arguments
Email subject lines

Even the color of the website or its structure can be put to the test, because there’s always something to optimize.

Where did A/B testing originate?

The principle had already proven its worth long before the internet. Even when skipping stones across a lake, people quickly figured out which types of stones skimmed better. Even cave dwellers recognized what helped against the rain: standing under a tree or in a cave instead of out in the open. The principle is logical, but it wasn’t used extensively in marketing until the 20th century.

In direct marketing, for example, companies would send two versions of a promotional letter to separate recipient groups and count the responses. Such split-run tests were also used for newspaper ads, among other things. Online, the method became standard in the 2000s, as variations could be tested without additional printing costs and results could be measured in real time. The principle received particular attention due to a case at Google. The search engine once tested 41 different shades of blue for links against one another, rather than deciding on the color in a meeting.

Why is A/B testing important?

A/B testing replaces opinions with measurement. Which headline, image, or form works better is a matter of personal preference for management or whoever shouts the loudest. A/B testing makes such decisions verifiable and reduces the risk associated with changes. Instead of making changes on a hunch, the change is first validated on a small scale.

The benefit lies in the cumulative effect: Several small, test-validated improvements add up to a significant increase toward the defined goal while maintaining the same visitor traffic.

Advantages of A/B testing

Decisions are based on objective evidence.
Internal debates over personal preferences lose their weight.
The risk is low, since changes are first tested on a small scale and rolled out on a large scale only if they show visible success.
With each test, knowledge grows about what convinces your target audience.

Challenges of A/B Testing

Test only one variable per test, and don’t stop at the first interim result. Additionally, the process requires a sufficient volume of visitors. With only a few hundred visitors per month, it takes longer to achieve a statistically reliable result.

Then there’s the matter of interpretation: A result applies only to the tested page and target audience—not automatically to everything else. Anyone who derives a universal rule from the test will miss the mark the next time around.

Example of an A/B Test

A mechanical engineering company is testing a CTA on its services page. Some visitors see “Request a Quote” on the button, while others see “Get a No-Obligation Consultation.” Four weeks later, it turns out that variant B generates 40% more inquiries. The non-binding wording apparently lowers the barrier to action for prospective customers. This is an insight one might have suspected, but which is now substantiated.