A/B Testing a Business Website

What an A/B test is

Google's page on minimizing A/B testing impact in Google Search defines A/B testing as testing "two (or more) variations of a change". Multivariate testing is "where you test more than one type of change at a time".

Google describes two ways to run a test:

• Separate addresses: each version has its own URL, and some visitors to the original page are redirected to a variation.
• The same address: variations are inserted "dynamically on the page", usually with JavaScript deciding which one to show.

Visitors are split at random over the same days, which makes a test fairer than comparing this month with last month.

Do you have enough traffic?

For many small business sites the honest answer is no. Nielsen Norman Group's A/B testing 101 says that to reach a statistically significant difference "you often need thousands of users interacting with your product", which "makes A/B testing a bad fit for pages with little traffic".

Its worked example: a button clicked by 3% of visitors, where you want to detect a change of 20% of that rate (from 3% to 3.6% or 2.4%), needs about 13,000 users.

If your page is in that position, try instead:

• watch a few people from your audience try to use the page, and fix what trips them up;
• make a clear improvement, note the date, and compare enquiries before and after.

Our guide to a simple monthly website report shows how to mark changes so a before-and-after comparison stays honest.

A hypothesis and one change

Nielsen Norman Group's four steps start with a hypothesis: which change you expect to have which effect, and why.

Then change one thing. The article says the changes "should address just one design element and not be an extensive design overhaul", because if several elements change at once "you will lack an understanding of the impact of each individual change".

Good candidates are the things a visitor uses to decide: the headline, the offer, the main button, the form. Our landing page guide covers what each of these should do before you test alternatives.

Choose the result before you start

Decide what counts as success before the test begins, not after you see the data. Nielsen Norman Group recommends a primary metric that tells you whether behaviour changed, plus "guardrail metrics" that show whether the change actually helps the business. A variation might get more clicks but fewer finished enquiries.

For a service business the primary metric is usually a completed enquiry or booking, counted when the form is accepted, not when the button is pressed. Our GA4 basics guide explains how to record that as a key event.

Sample size and significance

A test compares two conversion rates and asks whether the difference is larger than chance would produce. The NIST/SEMATECH e-Handbook gives the standard test for whether two groups have the same proportion, and explains the terms you will see in testing tools:

• Significance level. The handbook's page on critical values and p-values says a level of 0.05 "implies that the null hypothesis is rejected 5 % of the time when it is in fact true".
• p-value. "The probability of the test statistic being at least as extreme as the one observed given that the null hypothesis is true." It is not the chance that the variation is better.
• Power. The handbook's page on sample sizes for proportions shows that the sample size follows from the significance level, the power you want, and the size of the change "we are interested in detecting".

In practice you fix three numbers before you start, as Nielsen Norman Group lists them: the current rate, the smallest change worth detecting, and the significance threshold. A sample-size calculator then gives the number of visitors needed. The smaller the change, the more visitors you need.

The NIST handbook adds that "it is good practice to decide in advance of the test how small a p-value is required". Write the three numbers, the sample size and the end date down before the test starts.

Do not stop when it looks good

Nielsen Norman Group lists "stopping the test too early" among the most common mistakes, such as watching results daily and stopping when one version is ahead: "you must wait until the appropriate sample size of a test is reached".

The reason is statistical. In the paper Always Valid Inference: Bringing Sequential Analysis to A/B Testing, Ramesh Johari, Leo Pekelis and David Walsh write that the usual p-values and confidence intervals "are wholly unreliable if users endogenously choose samples sizes by continuously monitoring their tests". Each look is another chance for noise to cross the line.

Two ways to stay safe:

• decide the sample size in advance and read the result only when it is reached;
• or use a tool whose statistics are designed for continuous monitoring, as that paper describes.

Nielsen Norman Group also recommends running a test "for at least 1-2 weeks" even with enough traffic, "to account for potential fluctuations in user behavior".

Testing without hurting search

Google says small changes such as a button's colour or wording "often have little or no impact" on rankings. Its page sets out four practices:

• Don't cloak. Do not show Googlebot different pages from people. This is against Google's spam policies "whether you're running a test or not".
• Use rel="canonical". With separate URLs, point each variation's canonical tag at the original page. Google recommends this "rather than a noindex meta tag".
• Use 302 redirects, not 301. A 302 tells search engines the redirect is temporary, so they "keep the original URL in their index". Our guide to 404s and redirects explains the difference.
• Run the test only as long as necessary. Afterwards, "remove all elements of the test as soon as possible". Google warns that a test left running for an unnecessarily long time may be treated "as an attempt to deceive search engines".

Tools, speed and consent

Google's own free tool has gone. Its help page on the Google Optimize sunset says Optimize and Optimize 360 "are no longer available as of September 30, 2023", and that Google has made its APIs public so other testing tools can send results to Google Analytics.

Whichever tool you choose:

• Speed. A testing script is extra code on every tested page. Check page speed with the test running; our guide to Core Web Vitals explains what to measure.
• Consent. Many testing tools use cookies to keep each visitor on the same version. Our guide to cookie consent banners covers when consent is needed, including the UK statistics exception whose examples, the ICO says, include A/B tests.

After the test

When the planned sample is reached, read the result against the threshold you set. Then:

• if one version clearly won, publish it as the normal page and remove the test code and extra URLs, as Google advises;
• if there was no clear difference, keep the simpler version;
• check the guardrail metrics and your real enquiries before you call it a win;
• write down the hypothesis, dates, numbers and result, and add an annotation in your analytics.

Expect most tests not to win. Nielsen Norman Group notes that "only one in every seven A/B tests is a winning test".

A checklist

• The page has enough traffic for the change you want to detect; if not, use research instead.
• One change, with a written hypothesis.
• The primary metric is a real enquiry or sale, with guardrail metrics alongside.
• Current rate, smallest useful change, threshold, sample size and end date are written down first.
• Nobody stops the test early unless the tool's statistics allow it.
• Variation URLs use canonical tags and 302 redirects; Googlebot sees what people see.
• The winner is published, the test code removed, and the result recorded.

Frequently asked questions

How much traffic do I need for an A/B test?

It depends on your current conversion rate and the smallest change you want to detect. Nielsen Norman Group's example, a 3% click rate and a 20% change, needs about 13,000 users. Low-traffic pages are better improved through user research.

Can an A/B test hurt my Google rankings?

Not if you follow Google's guidance: show Googlebot the same thing as people, point variation URLs to the original with rel="canonical", use 302 rather than 301 redirects and end the test promptly.

Can I stop the test as soon as one version is clearly ahead?

Not with ordinary statistics. Research on A/B testing shows that p-values become unreliable when people keep checking and stop when the result looks good. Decide the sample size in advance.

What does 95% significance mean?

It means a significance level of 0.05: when there is no real difference, about one test in twenty will still show a difference that large by chance. It is not a 95% chance that the winning version is better.

Can I test several changes at once?

You can, but you will not know which change made the difference. Multivariate tests separate the effects but need even more traffic.

Planning a new website? Whoever builds it, the domain and your content should always be in your name, and our projects end with a short handover so your team can edit content. See our website development service and all our services, or email [email protected].