Testing App Store Listings: Apple Product Page Optimization and Play Store Listing Experiments

Before you test

Testing improves a listing that is already sound. Get the basics right first: the name, subtitle and keywords on Apple, the title and descriptions on Google Play, and clear screenshots. Our app store optimisation guide covers those, and this guide does not repeat them.

Then write down a hypothesis: what you will change, for whom, and what you expect to happen. "Showing the main feature in the first screenshot will raise installs" can be tested; "make the listing better" cannot. The same discipline applies as for website tests; our A/B testing guide explains why small samples and early stopping mislead.

Apple: product page optimization

Apple's product page optimization page describes it as a way to test "different elements of your App Store product page (or pre-order page) — including app icons, screenshots, and app preview videos". The alternative versions are called treatments. The main rules Apple lists:

• Up to three treatments per test, compared with your original page.
• You choose the share of traffic that sees a treatment, split equally. Apple's example: with 40% of traffic and two treatments, each treatment gets 20% and the original 60%.
• Treatments are shown to people chosen at random on the App Store in iOS and iPadOS 15 or later, and each person sees the same treatment for the whole test.
• Treatments can be localised in all or some of your app's languages; Apple notes that the choice of localisations can make a test take longer.
• You can run one test at a time, for up to 90 days, and you cannot change a test once it has started.
• New metadata in treatments must be approved by App Review before the test goes live.
• Alternate app icons must be included in the binary of your published app. Tests without alternate icons can be submitted for review without a new app version.

Apple also points out that a treatment's alternate icon shows on the App Store and, for people who download the app during the test, on their device. Treatment assets may also appear in search results and on the Today, Games and Apps tabs.

Running and reading an Apple test

When you set up a test in App Store Connect, an optional estimate uses your app's impressions and downloads to show roughly how long the test would need, and how many impressions, to reach a result with at least 90% confidence for the improvement you choose. If the estimate is longer than 90 days, Apple suggests fewer treatments or more traffic.

Results appear in App Analytics: impressions, conversion rate, percentage improvement and confidence level for each treatment against the baseline, which is your original page by default. Applying a treatment while the test runs ends the test, and Apple recommends waiting until at least one treatment has been declared better or worse than the baseline with at least 90% confidence. If the winner has an alternate icon, include that icon in your next app version.

Apple's own developer stories show why testing matters: in one, the original page without a preview video beat the treatment that added one, the opposite of what the developer expected.

Google Play: store listing experiments

Google's Play Console Help page Run A/B tests on your store listing describes experiments to find "the most effective graphics and text for your app", for your default store listing and for custom store listings. There are two types:

• Default graphics experiments test the icon, feature graphic and screenshots in your default listing language. People viewing a language for which you have added localised graphics are excluded.
• Localised experiments can test the icon, feature graphic, screenshots and descriptions in up to five languages, and are shown only to people viewing the listing in those languages.

Per app, you can run one default graphics experiment or up to five localised experiments at the same time. Unlike Apple, Play lets you test text (the descriptions) as well as graphics.

Running and reading a Play experiment

You set up an experiment from Grow users > Store presence > Store listings. You choose a target metric: unique user install clicks, open clicks or pre-registration clicks. Google defines unique user installs as people who installed the app and did not have it on any other device at the time. Play Console estimates how long the experiment will need to give a statistically significant result.

Advanced settings let you change the number of variants (up to two), the share of store listing visitors who see a variant, the minimum detectable effect (a smaller difference counts as a draw) and the confidence level. Google says the defaults give results in the shortest time with reasonable accuracy, and recommends testing one asset at a time.

Results show which version performs best, with an explanation based on the confidence interval and minimum detectable effect, and a recommended action. "More data needed" means come back later. If your current listing wins, keep it. Experiments stop automatically after 6 months, and traffic then returns to the current listing.

Tests are not custom pages

Both stores also offer extra listing versions for specific audiences, and these are easy to confuse with tests. Apple's custom product pages each have their own URL and can be used in Apple Ads; Google's custom store listings can target countries, pre-registered users, people searching for specific keywords, inactive users, a unique URL or a Google Ads campaign.

A custom page shows a fixed version to a chosen audience; a test splits the same audience at random to find out which version is better. Apple measures custom product pages in App Analytics, where you can compare them with your default page, but that comparison is between different audiences, not a controlled test. On Google Play you can run an experiment on a custom store listing itself. For custom product pages in Apple Ads, see our guide to advertising an app.

What to test first

Apple's page suggests questions such as whether a certain character or value proposition gets better results, whether highlighting a feature or culturally relevant content raises downloads in a locale, whether a different icon style or colour improves conversion, and whether seasonal content leads to more downloads. Practical starting points:

• the first one or two screenshots;
• the icon, if you have a real alternative rather than a minor colour change;
• on Google Play, the short description.

Change one thing per test, write down the result, and apply only clear winners. A draw is a result too: it tells you the asset was not what held installs back.

Frequently asked questions

What can I test with Apple product page optimization?

Apple says you can test app icons, screenshots and app preview videos on your App Store product page or pre-order page, with up to three treatments against the original. Text such as the description is not part of these tests.

How long do store listing tests run?

Apple's tests run for up to 90 days or until you stop them, one test at a time. Google Play store listing experiments stop automatically after 6 months, after which traffic returns to the current listing.

Can I test text on Google Play?

Yes. Localised experiments in Play Console can test descriptions as well as the icon, feature graphic and screenshots, in up to five languages. Default graphics experiments test graphics only.

What is the difference between a test and a custom product page?

A test splits the same audience at random to find out which version converts better. Apple's custom product pages and Google's custom store listings show a fixed version to a chosen audience, such as people arriving from a particular ad or country.

Building an app and want the store listing and analytics planned from the start? See our mobile app development service and all our services, or email [email protected].