Without something like Google experiment, you’re therefore left with having to use 3rd party A/B testing tools like Splitmetrics. But you might not want to invest in these tools just yet, or be interested in evaluating the “live” impact of a change to your App Store listing.
Table of Content
In this step-by -step guide we detail how you can best assess the impact of any change during your App Store Optimization efforts, from a new icon to an updated App Store title for your app.
We’re an app video production agency so of course we often refer to a change regarding the App Store Preview videos.
The general idea of the pre-post analysis is to look at a couple of metrics in iTunes Analytics and compare before vs. after adding the video. It’s important to compare the results for at least 7 days (7 days before, 7 days after) to account for potential changes in behavior depending on the weekday or weekend.
Note: as mentioned there is a focus on video in this post, but the pre-post analysis method explains here can work with any change in your app store listing for which you’re trying to assess the impact.
BEFORE GOING FURTHER
Because this is not an A/B test, you especially want to keep marketing/acquisition efforts as consistent as possible and not change anythisaveng else (icon, description, screenshots and keywords) so it doesn’t change the results too much.
You also want to monitor any large externalities: features by Apple, PR from a top website, etc. The goal is that only one thing changes: in our case, video vs. no video.
iTunes App Analytics gives you the possibility to define the range for which you want to look at the data.
Here our goal is to assess the impact of a change in your App Store listing (adding a video). So you want to get a full week of data (7 days) before the day of the change and a full week of data (7 days) after the day of the change. This is very important, as there can be a huge difference in install conversion depending on the day.
Alternatively you can look at the data for the week before the change vs. the data for the week after the change.
If there wasn’t any other release (and external “chang” – features, PR, etc.), you can also take larger periods of time before/after as long as each period contains the same days total (example: 2 weeks before, 2 weeks after for example).
SOURCE TYPE, PLATFORM VERSION AND OTHERS
Browse or search?
As you can see in the definitions of the App Store visitor sources above, in iTunes analytics you can filter by visitors that came browsing in the App Store and visitors that find your app through a search.
No matter the metric you measure (see below), it is interesting to filter through both Source Types. The reason is that these are 2 sources of visitors that may not be searching for your app specifically. With the exception of brand searches of course. In the case of App Store Browse, there might even not be any specific intent at all.
If like in our case you are testing the impact of a video, you can’t ignore the App Store Search Source Type. Video is indeed displayed in both the search results and on the Product Page (App Store listing). As mentioned App Store Search also includes Search Ads so keep this in mind (the video can also be displayed in Search Ads)
Note: there are other source types (App referrer, Web Referrer, Unavailable). But because we want to focus on “organic” installs as much as possible (remember: Search Ads are still in the App Store Search source type) we are not looking at them here.
You can also filter by Platform version. The idea here is to get a view of the metrics only for users that have iOS 10 or only users that have iOS 11.
You can play around with this as well. We recommend to focus on users that have iOS 11 as they are the ones that see the new iOS 11 App Store design that will be used going forward (until the next update from Apple of course).
Unfortunately, because filters in iTunes Connect App Analytics only allow you to select one platform version (iOS 11.0, or iOS 11.1, etc.) you will have to use the “View By” section and export the data to be able to get numbers for all iOS 11 users.
WHICH METRICS TO MEASURE
The CTR and CVRs in this article are calculated via the data you can find in iTunes Connect App Analytics, in the Metrics tab.
What you want to do is:
- download the data (.csv file) from the view you’re seeing in iTunes Connect App Analytics;
- open it with Excel (or with Google Sheets – it automatically puts the data in columns);
- copy/paste it into the right sheets/tabs (called “Data”) of your copy of our spreadsheet;
- Check out the “Comparison” sheets/tabs.
Tip: as you go through your first pre-post analysis, save each important view in the “Saved View” panel at the bottom left. Between this and the Google Sheet above, this should make each analysis much faster!
2 ways to calculate the Install Rate (or Conversion Rate – CVR)
Whether following a search or when browsing, the Install Rate or CVR is about measuring the percentage of visitors that tap the “Get” button (or Price button) to download your app.
As this button is present outside of the Product Page, it means that with the iTune Analytics data there is unfortunately no way to differentiate these 2 things:
- Conversion Rate directly from the Search Results
- Conversion Rate only from the Product Page
What we’re looking at for the Install Rate / CVR can therefore be either:
- CVR Impressions -> App Units
- CVR Product Page Views -> App Units
Because installs (App Units) can come either after a Product Page View or an Impression and because Impressions include Product Page Views, you can NOT assume that Impressions -> App Units will give you the CVR from the search results and that Product Page Views -> App Units will give you the CVR from the App Store listing.
Do you need to measure all the possible combinations?
Depending on what the change to your App Store data is (example: video vs. no video, new video vs. previous video, etc.), the CTR and which CVR to calculate in priority might vary. For example:
- If you change something that is only on your Product Page (example: adding a 2nd App Preview), then the CTR and CVR from Impressions are less relevant than the CVR from Product Page Views.
- If you change something that is displayed in the Search Results as well (example: adding a video when there was none before, changing your icon, etc.) then the CTR, the CVR from Impressions and the CVR from Product Page views are all very relevant.
The change for which you’re trying to assess the impact also defines which filter to choose (Source Type like Search or Browse, etc.) or view (Platform version to get iOS 11 users only). For example:
- Video does not appear when browsing the App Store by Category, so calculating CVRs for App Store Browse is less relevant for a video pre-post analysis (i.e selecting App Store Browse in the filter is will give you a very partial view of things).
- Video does not autoplay for users that do not have iOS 11 or later, therefore the impact of video will not be the same for these users. There are also a few other edge cases where there is no autoplay, even for iOS 11 users (low battery, turned off in settings, etc.).
If your change is only for one localization/territory, then this is something you may want to “filter by” as well to get more relevant results.
That said, it’s easy to get lost with the different combinations possible, so start by calculating with no filter and no specific views like explained at the start of each section.
CALCULATING THE CLICK-THROUGH-RATE (CTR)
Because Impressions include Product Page Views (see Apple Glossary above: people that see your Product Page are counted twice) the Click-Through-Rate (CTR) here is not exactly the same as what you usually get (for ads for example).
But an increase in the CTR we’re about to calculate definitely gives you an indication that more people go to the product page after seeing the app, whether when browsing or searching. It is therefore a key metric. At the same time it is still a win if you get an install just from an impression in the Search Results, even though it doesn’t reflect positively on the CTR.
What we’re looking out for the CTR is Impressions -> Product Page Views.
So in iTunes Analytics you want to select “Product Page views” (with “Unique devices” checked) then “Impressions -> Unique Devices” in the COMPARE TO section.
Note: It’s very important to check “Unique Devices”, otherwise visitors are counted several times (and the difference in count is huge – about 40% more in our case).
Once you do, you’ll see a chart and a table with Product Page Views, Impressions and Ratio for each day of the period you selected.
Calculation for all Platform Versions (all OS) and Source Types
For now, we won’t filter the data by source type.
You can download or copy the data for each day and input it in Excel or a Google Spreadsheet. You’ll see that when downloading the csv file from iTunes Connect the ratio is not in there. So add a column for the CTR % which is calculated this way (obviously if the data format is set as a % like below you do not need to multiply by 100):
Next you want to add a column and define the period before the change (“Pre”) and the period after the change (“Post”).
Our goal is to compare the increase/decrease in CTR before the change and after the change, so we now want to calculate the average CTR pre and post change. You can do this easily with the Average function or using a Pivot table like below.
The % of change in CTR can then be calculated like this:
Here we have a 3% increase after the change, which is definitely not significant. Be careful about drawing conclusions with any change that would be within single digits.
Calculation by Source Type
Let’s take a look at what we get when we filter data by Source type to see only visitors coming from App Store Search.
Repeat the same steps: download the data, input it in Excel/Gsheet and calculate the CTR change between before and after the change.
The % of change here is more significant, but is still to be considered in its context (i.e how are the other metrics).
Here is what we have if we filter only by App Store Browse:
Keep in mind you can not only look at the CTR by itself to make a decision: more people on your Product Page (App Store listing) is not interesting if it doesn’t bring you more downloads.