Testing a Mobile App Before Release: TestFlight and Play Testing Tracks

Start with a short test plan

Write down the handful of tasks the app exists for: sign up, book an appointment, place an order, receive a notification. For each one, note what "working" means, such as "the booking appears in the admin dashboard and the customer gets a confirmation". These are your acceptance criteria; our guide to working with a web agency explains how to write them.

Then list where the app must work: the oldest iOS and Android versions you support, phones or tablets, and any languages. Keep the plan short, but write it down, so every tester checks the same things.

Simulators, emulators and real devices

Much day-to-day testing happens on software copies of phones. Apple's Xcode documentation on running your app on simulated or physical devices says simulators let you test on "a variety of hardware that you might not have access to", but they "don't replicate the performance or features of a physical device". Its advice: "To verify that your app runs exactly as intended, run it on one or more physical devices."

The Android Emulator can "simulate a variety of devices and Android API levels" and lets you "simulate different network speeds, simulate rotation and other hardware sensors".

A practical mix is simulators and emulators for breadth, and real phones for anything that depends on hardware or feel: the camera, Bluetooth, location, notifications, speed on an older or cheaper phone, and how the app handles a real network. Include at least one older, lower-cost Android phone, where slow screens are most likely to show.

Test the situations, not only the screens

Many bugs hide outside the happy path. Go through the main tasks again under each of these conditions:

• Permissions refused. Say no to notifications, location or the camera, and check the app still works and explains what is missing. Our guide to app onboarding and permission prompts covers when to ask.
• Poor or no connection. Turn on flight mode halfway through a task, then reconnect. Check nothing is lost or sent twice.
• Interruptions. A phone call, switching to another app, locking the screen, a low battery warning.
• Updating from the old version. Install the current live version, sign in, then update to the new build. Saved data and logins should survive.
• Display settings. Larger text sizes, dark mode, and rotating the screen if the app allows it.
• New and returning users. A fresh install, a password reset, and deleting an account.

Accessibility: VoiceOver and TalkBack

Both platforms have a built-in screen reader, and testing with it is the most direct accessibility check. Apple's VoiceOver evaluation criteria say that if you can tap, click or drag something, "you should strive to make it work with VoiceOver, too", and every important item on screen needs a label that a VoiceOver user can hear. Apple recommends starting with VoiceOver because that work also helps users of other assistive technologies. The same criteria help you decide whether to show the app as supporting VoiceOver on the App Store.

Android's guide to testing your app's accessibility walks through TalkBack, "Android's built-in screen reader". With TalkBack on, swipe through each element in turn and ask, in Google's words: does the spoken feedback "convey its content or purpose appropriately?", and "Can you complete the main workflows easily?" Google also suggests analysis tools, automated tests and feedback from real users. What to build in before testing is covered in our guide to making a mobile app accessible.

Testing purchases without real payments

If the app sells subscriptions or in-app purchases, test every path: buying, cancelling, restoring on a new phone, a failed payment and a renewal. Both stores provide test setups.

On Apple, sandbox testing lets you use "real product information from App Store Connect", and "transactions from test accounts don't incur charges". It covers development builds run from Xcode and "apps you download from TestFlight". Sandbox settings can "change the subscription renewal rate" and clear purchase history. You need Sandbox Apple Accounts, created in App Store Connect.

On Google Play, the Play Billing testing guide recommends license testers and Play Billing Lab. License testers get test payment methods, including slow test cards that approve or decline after a few minutes, which is how you check that a delayed payment is handled correctly. Play Billing Lab can speed up subscription renewals and move a test subscription into "grace period or account hold state with one click".

Beta testing on iPhone with TestFlight

Apple's TestFlight distributes test builds to people outside the development team. According to Apple:

• Up to 100 internal testers from your App Store Connect team, in roles such as Admin, App Manager, Developer or Marketing.
• Up to 10,000 external testers, invited by email or a public link. The first build for external testers must be approved by App Review for TestFlight.
• Public links can have criteria, "such as device type and OS version", so you recruit the testers you need.
• Testers can send a screenshot from inside the app, marked up with comments. When the app crashes, you get a crash report and the tester can add context.

The TestFlight overview adds that "you can test a build for up to 90 days", and that build metrics include the number of sessions and crashes. Write a clear "what to test" note for each build, because it is what testers see first. Store accounts and the review itself are covered in our guide to publishing an app to the App Store and Google Play.

Google Play testing tracks and the pre-launch report

Google Play has three testing tracks. Internal testing reaches "up to 100 testers for initial quality assurance checks". Closed testing goes to a wider, invited group; open testing is visible on Google Play and anyone can join. Google recommends "starting with an internal test, then expanding to a small group of closed testers". It also notes that personal developer accounts created after 13 November 2023 must meet extra testing requirements before an app can go public; our publishing guide explains them.

Uploads also get an automated check. The pre-launch report installs the app on "a set of Android devices in our test lab", then crawls it "for several minutes", typing, tapping and swiping. It reports stability, Android compatibility, performance and accessibility issues. If the app needs a login, give the crawler test account credentials, or it may not get past the first screen. The report runs "subject to capacity", so treat it as an extra check, not a replacement for your own testing.

Sorting what testers find

A beta produces a long list quickly. Sort it before fixing anything:

1. Crashes and data loss: fix before release.
2. A main task cannot be completed: fix before release.
3. Problems on some devices or OS versions only: decide by how many of your users have them.
4. Confusing wording or layout: fix if it is quick, or schedule for the next update.
5. Feature ideas: keep for later.

Record the device, OS version and build number with every report. After launch, crash reporting and store quality data take over from beta feedback; see our guide to mobile app analytics and crash reports.

A checklist

• The main tasks and what "working" means are written down.
• Supported OS versions and devices are agreed, including an older Android phone.
• Each task is tested with permissions refused, no connection, interruptions and an update from the live version.
• The main tasks can be completed with VoiceOver and TalkBack.
• Purchases, renewals, restores and failed payments are tested in the stores' sandboxes.
• A TestFlight and Play internal or closed test has run with people outside the team.
• The Play pre-launch report has been read, with a test login provided.
• No known crash or blocked main task remains.

Frequently asked questions

Is testing on simulators enough?

No. Apple's documentation says simulators don't replicate the performance or features of a physical device. Use simulators and emulators for breadth, and real phones for the camera, location, notifications, speed and real networks.

How many testers does TestFlight allow?

Apple says up to 100 internal testers from your App Store Connect team and up to 10,000 external testers. The first build for external testers must be approved by App Review for TestFlight, and a build can be tested for up to 90 days.

What are Google Play's testing tracks?

Internal testing for up to 100 testers, closed testing for a wider invited group, and open testing that anyone can join on Google Play. Google recommends starting with an internal test, then a small closed test.

What is the Google Play pre-launch report?

An automated check that installs your app on devices in Google's test lab and crawls it for several minutes, reporting stability, compatibility, performance and accessibility issues. Give it a test login if the app needs one.

How do we test in-app purchases without paying?

Apple's sandbox uses Sandbox Apple Accounts and does not charge for test transactions. On Google Play, license testers get test payment methods, and Play Billing Lab can speed up subscription renewals.

Planning an app? We build in short cycles with test builds installed on your devices along the way, and test on a range of real iPhone and Android devices and screen sizes before we prepare the store listings and submit for review. See our mobile app development service and all our services, or email [email protected].