A/B testing in Meta Ads Manager

This guide is built from Meta's Business Help Center: About A/B testing, Best practices for A/B testing, Viewing and understanding A/B test results and the other A/B test pages linked below. It is about tests inside Meta Ads Manager, where Meta splits the audience and picks the winner. Testing changes on your own website pages is a different job, covered in our guide to A/B testing a business website. If campaigns, ad sets and ads are new to you, start with Meta ads basics.

What a Meta A/B test does

Meta describes an A/B test as a way to compare two versions of an ad strategy by changing variables such as ad images, ad text, audience or placement. Each version is shown to a segment of your audience, and Meta says it will ensure nobody sees both. Versions are compared on a cost per result basis, or on cost per conversion lift where a holdout is involved (see below).

Meta also says what not to do: We do not recommend testing informally, such as by turning ad sets or campaigns on and off manually. Its reason is that informal tests can lead to inefficient delivery, unreliable results and overlapping audiences, while an A/B test splits the audience evenly so the groups are statistically comparable. Running two ad sets side by side and looking at which one did better is not the same thing.

Start with a hypothesis

Meta's page on choosing a hypothesis says it should be specific, actionable and relevant to your business objectives. Its best-practice page shows how to narrow a vague question: from Do I get better results when I change my performance goal? to a statement such as My cost per result will be lower when I optimize for landing page views. A statement like that tells you in advance what you will do with each possible result.

The variables you can test

Meta's page on selecting a variable lists four kinds:

Variable	What changes	Meta's examples
Creative	Images, text or creative type, managed at the ad level	A slideshow video against an image carousel; the same asset with different ad text; different product images
Audience	Who is reached, managed at the ad set level	Recent website visitors against a lookalike audience; similar interests with different age groups
Placements	Where the ads appear	Automatic placements against your current setup; Feed-only against Stories-only
Custom	Several changes at campaign, ad set or ad level	Comparing two complete strategies

Meta recommends starting with a single variable if you are new to A/B testing, and its best-practice page says results are more conclusive when the ad sets are identical except for the variable being tested. Testing more than one variable is possible, for example two products in two locations, but Meta warns that you won't know exactly which variable led to the winning ad set. That answers "which of these two ad sets is better", not "what should we change across our ads".

Three ways to set one up

Meta's page on tools to create A/B tests says it generally recommends the Ads Manager toolbar, and that all methods use the same underlying technology, with results available in Experiments once the test finishes.

From the Ads Manager toolbar

Per Create an A/B test in Ads Manager: click A/B Test in the toolbar, then choose Make a copy of this ad or Select two existing ads.

• Make a copy: pick the variable to change, choose the ad set to copy, name the test, choose how the winner is decided (you can add more metrics), set the test's start and end dates, duplicate the ad set, then publish.
• Two existing ads: choose whether to compare campaigns or ad sets, pick them (you can add further ad sets), name the test, choose the winning metric, schedule it and publish.

The test schedule is separate from the campaigns' own dates. Meta says it should be set to cover the duration of your ad campaigns, or enough time to yield reliable results.

From the Experiments tool

Per Create an A/B test in the Experiments tool: go to Experiments, select Create test, find A/B Testing and click Get started. You can test existing campaigns or duplicate one. With existing campaigns you can compare up to 5, but not campaigns already in another test or campaigns using the reservation buying type. Meta suggests Experiments when you already have active campaigns to compare, want to set each one up precisely first, or want extra metrics beside the key metric.

By duplicating a campaign, ad set or ad

Duplicating in Ads Manager is the quick route for a small change to an existing ad. Note Meta's point about budgets in About A/B testing: if you duplicate a campaign, its budget is duplicated too; otherwise your current budgets are used. Meta recommends the same budget for both versions to ensure a fair comparison.

Duration, audience and budget

From Meta's best practices:

• Duration: we recommend a minimum of 7-day tests. Tests can run for at most 30 days, and an Ads Manager test must be scheduled for between 1 and 30 days. Meta says tests shorter than 7 days may be inconclusive, and if your customers usually take more than 7 days to convert after seeing an ad, run longer (its example is 10 days).
• Audience: large enough to support the test, and not used by any other campaign you run at the same time. Meta says overlapping audiences may result in delivery problems and contaminate test results.
• Budget: Meta does not give a figure. It says the budget should produce enough results to confidently determine a winning strategy.

Because Meta divides your audience between the versions, its tips page warns that test ad sets are more vulnerable to under-delivery, and you may need to broaden audiences more than usual or raise the budget.

Editing or cancelling a running test

Per Edit or cancel your A/B test: while a test is In review or Scheduled you can change anything. Once it is Running, you can't change the variable being tested; in a creative test that means no edits to headlines, descriptions or other ad content, though audience and placements can still be changed. The schedule can be changed or the test cancelled from Experiments, but Meta says either will affect the reliability of the results so far.

One detail that catches people out: Cancelling an A/B test doesn't change or stop delivery for the campaigns in it. To stop spending, pause them in Ads Manager.

How Meta picks a winner

Per How winning campaigns are determined, Meta compares the cost per result of each version, where the result is the event you chose when creating the test, such as cost per click or cost per purchase. Meta simulates possible outcomes tens of thousands of times to see how often the winning version would have won, which gives a winner with a confidence percentage.

Meta's page about confidence in tests describes confidence as the likelihood that the outcome would be consistent if the test were repeated under the same conditions, and states: For A/B tests, a 65 percent or higher confidence percentage represents a winning result. For lift tests the bar is 90 percent. Meta also notes that a winner can still have higher costs when the test was too short, there weren't enough results, or the best practices for test length weren't followed.

Reading and using the results

Per Meta's results page, you can follow a test in Experiments or in Ads Manager (View charts on a campaign, ad set or ad in the test; a beaker icon marks campaigns with a scheduled, running or recent test). When it ends you get an email with a link to the full results, unless you opted out of Meta emails. The results include:

• Test details: name, schedule, status and the campaigns or ad sets involved.
• Performance chart: cost per result for each version. Meta notes the figure will likely change while the test is still running.
• Explanation of results: the version with the lowest cost per result for the key metric, marked with a green trophy, or the top performers on other metrics if there is no clear winner.
• Age and gender charts, once enough data is collected.

Meta's own example of using a result: if product images beat lifestyle photography, you might turn off the lifestyle ads, move that budget to the winner and use more product imagery in future, then run another test, for example lifestyle photography with a different age range or region.

Conversion windows can be a test variable of their own. Meta's tips page lists them under delivery optimization tests and suggests changing either the conversion window or the optimization event in a test, not both. Our guide to Meta attribution settings explains how the windows work.

When there is no winner

Meta's tips for improving A/B tests cover tests that found no winner or under-delivered:

• Make audiences large enough to avoid under-delivery, and raise the budget if delivery was low.
• Make the versions different enough. Meta's example of a test that is too close: women aged 18-20 against women aged 20-22 who both shopped on your site. A better one is a custom audience of past shoppers against an interest-based audience.

The results page itself may also suggest a next step, for example extending a test that didn't run long enough to collect enough data.

The same page suggests follow-up tests: repeat a test under each campaign objective you plan to use (results don't automatically carry across), test creative or placements on the winning audience, and after a format test (say video beat a single image), test two videos against each other.

A/B tests, holdouts and lift tests

A normal A/B test tells you which version got results more cheaply. It doesn't tell you how many of those results would have happened without any ads. That is what a holdout measures: a group that does not see the ads, so Meta can report cost per conversion lift, the cost of each conversion caused by the ads.

Meta's pages are not consistent here. The winner page says you can add a holdout to an A/B test, while the page on A/B holdout test results says Holdout tests in the Experiments Tool are no longer permitted as of April 26, 2021, with older results still viewable. Check what Experiments offers in your account before planning around one. For lift tests, Meta treats 90 percent confidence or higher as statistically reliable.

Our checklist

This is our own working list, based on Meta's guidance above.

• Write the hypothesis and the action you will take for each outcome before building the test.
• Change one variable. Keep budget, schedule and everything else equal.
• Pick the winning metric to match the campaign goal, and check the attribution setting first.
• Keep the test audience out of other campaigns for the duration.
• Schedule at least 7 days, longer if your buyers take longer to convert, and leave the variable alone while it runs.
• Read the confidence percentage, not just which version looks cheaper.
• When a test ends, act on it: move budget, record what you learned, plan the next test.

Frequently asked questions

How long should a Meta A/B test run?

Meta recommends at least 7 days. A test can run for at most 30 days, and longer than 7 days makes sense if your customers usually take more than a week to convert.

What confidence does Meta need to call a winner?

Meta says that for A/B tests, a confidence of 65 percent or higher represents a winning result. For lift tests it uses 90 percent.

Can I edit ads while the test is running?

Not the variable you are testing. Once a test is Running, Meta locks that variable; other settings such as audience and placements in a creative test can still change. Changing the schedule or cancelling affects the reliability of the results.

Does cancelling a test stop my ads?

No. Meta says cancelling doesn't change or stop delivery for the campaigns in the test. Pause them in Ads Manager to stop spending.

How many versions can I compare?

In the Experiments tool you can compare up to 5 existing campaigns. Campaigns already in another test, or using the reservation buying type, can't be used.

For more on Meta advertising, see our guides to Meta ads basics, Meta attribution settings and Advantage+ campaigns. For tests on your own pages, see A/B testing a business website.