The Scientific Method of A/B Testing

It’s easy to run a split test in the Google Play Store. Simply set up your test, add a few variants and start the test. Even though it’s that simple to start a split test, the key is making sure you’re running an effective test, one that is based on hypothesis and reason. Let’s step back into your sixth grade science class where we learned about the scientific method. First you ask a question, do some research, create a hypothesis, run an experiment and then, if that experiment worked, you’re done — but if it didn’t you go back and try again.

It’s crucial to apply the same concept the split testing in order to create effective and worthwhile tests. Here’s how:

Do some research

When you’re A/B testing your app’s on-page elements, the question you’re ultimately trying to answer is “how can I increase my conversion rate?” The first step is to understand the layout of the Google Play Store and understand what the user will see as a first impression. On the search results page in the Play Store, the two “split-testable” items are your apps icon and title. The icon is the first visual asset a user will see, and the title is the main way for a user to get an idea of what your app is. In the Google Play Store, users do not see your app’s screenshots in search results as they do in the App Store. The icon and app title are the only elements that you can split test that will be visible in search results.

The app page in the Google Play Store has a number of elements that are typically “above the fold” (visible without scrolling) on most devices. Almost all of these elements are things that can be split tested. Again, the icon and app title are the top elements on the page and will continue to provide a good opportunity for testing when it comes to on-page behavior. The next two elements to focus on would be the screenshots and the short description. Multiple screenshots are visible on most devices (depending on device and horizontal/vertical layout), but they’re relatively small, especially in comparison to the App Store. Your app’s video will also be listed first, ahead of any screenshots. In the App Store, Apple forces you to choose a poster frame from your video, but in the Google Play Store, you have the option of uploading a “Feature Graphic” which would become the video’s poster frame.

A consideration for split testing your app’s screenshots and feature graphic would be text size and visibility of the UI. The short description is the first piece of text (outside of the title) that the user sees when they’re on your app’s page. The short description is a fantastic opportunity to work on highlighting your app’s main features and seeing what users are generally attracted to.

Ask a question

There’s always an opportunity to split test your app’s on-page elements and now that you’ve done your research to understand how the users see your app, it’s time to figure out which elements provide the best split test opportunity.

One of the “easier” things to test would be the short description. Writing 80 characters is a bit easier than designing a full set of screenshots, so it often makes sense to start here. Consider how you’d like to explain your app to users and the top features of your app that users should know about.

The app icon is another area that can provide the most conversion lift because of its visibility in search results and on the app’s page. Consider new layouts, colors and elements in the icon that would potentially catch a user’s eye and help your stand out from the crowd, or make you look similar to your top competition. You won’t know the best route until you test!

Screenshots are the most prominent visual element on the page and it’s your opportunity to quickly introduce your app both through the visuals and the text callouts. As I mentioned previously, the screenshots are relatively small, so consider bumping up the font size and making sure the contrast of the background and the text is very readable. Implementing new design layouts and color schemes is another effective way to split test.

Construct a hypothesis

Split testing is a fantastic way to improve conversion rates and grow your app’s installs, but there needs to be a valid hypothesis involved. Come up with some reasoning behind what you’re testing. Some things to think of would be the main features that your app has, the key differentiators and the main features that your users utilize. Look at your current user behavior and reviews in order to identify how your users are interacting with your app. If there’s a main theme or feature, that’s probably a great hypothesis to work off of. “Can conversions be improved if I highlight _____ feature?”

Outside of user behavior, understanding organic search habits and keyword volumes can be a great way to capture more users. The hypothesis might be to create screenshots that work off of your highest volume keywords. For example, if you’re an app that helps users learn a new language, then you’ll want tp highlight “language learning” or “learn new languages” early on in your screenshot set, in your title and short description. Another major consideration is testing different variants in different countries. Understand that every country has different search volumes and more than likely your app is getting installs from different keywords. In the United Kingdom, users are often searching for apps where they can learn German, but in India, you’d be much better off highlighting a users ability to learn English in your app.

Creating a data-driven approach with a valid hypothesis is the most effective way to test and improve conversion rates!

Run a test

Now that you know what you’re going to A/B test, it’s time to run it! Throw your variations up on the Google Play Store and let the people tell you what they want. Once you know what’s working better, stick with it. Remember, it’s not necessarily what you like, it matters more what your users like!

After you run your A/B test, your work isn’t done there. A good scientist never stops improving on their work. So you found out that a specific variant performed better in your split test, great! But you’re not done. Now try making some tweaks and changes based on the winning variant and continue to analyze and improve.

Split testing is never truly “done” because the Google Play Store evolves and changes, and you can be sure that the competition is doing the same.

Need help with split testing? Contact us today to talk to an ASO expert about how we can help your app.