How to measure the impact of an ASO change

Measuring ASO is a before-and-after comparison with discipline. Write down the change, choose the metric it should move, read equal windows on both sides, and check what else changed in the same weeks. When the change is visual (icon, screenshots, previews), a store experiment beats any before-and-after reading, because both versions run at the same time.

Which method fits which change?

The method depends on what you changed.

Step 1: How do you keep a change log?

Log every change on the day it goes live, not the day you submit it. Use one row per field per storefront.

Rules: one change per storefront per window if you can; if you must ship several, accept that you will measure them together. Note releases, price changes, sales, features and ad changes in the "other events" column, from your own team and from the stores.

Step 2: Which metrics should you read?

Read the layer the change targets first, then check the layers below it did not fall.

Visibility turns many keyword ranks into one number. Uptika's visibility score multiplies each keyword's popularity by a fixed tap-share weight for its position (1.00 for #1, 0.60 for #2, 0.45 for #3, 0.30 for #4–5, 0.15 for #6–10, 0.06 for #11–20, 0.02 for #21–50), adds them up and divides by total popularity. The weights are a model, not measured tap data; the full table is in the methodology.

Worked example for a habit tracker called Notara, one storefront:

Visibility before = 100 × 10.2 ÷ 140 = 7. After = 100 × 23.1 ÷ 140 = 17. Visibility more than doubled; now check that search impressions and first-time downloads from search rose too. If visibility rose and downloads did not, the keywords may be the wrong ones, or conversion fell.

Step 3: How long should the reading window be?

Use a fixed window and decide it before you look at the data. The stores publish no standard; a common practice is three to six weeks after the change, compared with the same number of weeks before it, longer for small apps with few daily downloads.

• Start the "after" window on the day the change went live, not the submission day.
• Use whole weeks, so each window has the same weekdays.
• Skip the newest data: Apple says a day's Analytics data is complete two days after the reporting date, and Google's monthly CSV reports are posted within 3 to 7 days.
• Do not extend the window because the result looks close; that is how false wins are made.

Step 4: What else could explain the change?

Before you credit your change, rule out these four.

1. Seasonality. Compare with the same weeks last year, or use a control storefront where you changed nothing. If the control rose by the same amount, the season did it.
2. Ads. Keep spend and targeting flat. Apple counts ads in App Store search results inside the App Store search source, so a new campaign changes "search" numbers. Google reports ads and referrals as a separate source.
3. Store algorithm shifts. If many apps' ranks moved on many keywords on the same day, including keywords you did not touch, the store changed, not you. A daily rank history for rivals makes this visible.
4. Rival moves. A rival that changed its title, ran a sale or got featured can push you down or up. Check rivals' listing changes by date; see app competitor analysis.

When should you use store experiments instead?

Use them whenever the change is visual, and on Google Play also for descriptions. Setup and reading are covered in product page optimization and store listing experiments.

• App Store product page optimization: up to three treatments against the original, one test at a time, up to 90 days, and a test cannot be changed once started. Results appear in Analytics, Acquisition, Product Pages; Apple labels a treatment Performing Better or Performing Worse at 90% confidence.
• Google Play store listing experiments: Grow users > Store presence > Store listings. Up to two variants; a default graphics experiment or up to five localized experiments at a time; the target metric is unique user install, open or pre-registration clicks; you set a minimum detectable effect and a confidence level; experiments stop after six months. More in store listing experiments.

Common mistakes

• Comparing a week with a release or a sale to a quiet week.
• Reading total downloads instead of the source the change targets.
• Changing title, screenshots and price in the same week, then crediting one of them.
• Reading ranks once a week at random times instead of daily at a fixed time; see keyword rank tracking.
• Calling a test early.

How Uptika helps

Before you change anything, run the free Listing X-Ray to save a baseline of your character use, repeats and rating against the category top 10. In the panel, the Visibility score (free) tracks the popularity-weighted visibility above for your storefront every day, and the Rival log (Starter) lists rivals' title, screenshot and price changes by date, so you can see whether a rival moved in your window. The free Chart Movers tool shows daily category climbers and new entrants. Create a free account.

Questions

Sources

1. Acquisition (App Store Connect Analytics Help)
2. Analytics reports API (App Store Connect Analytics Help)
3. Product page optimization (Apple Developer)
4. App Store search (Apple Developer)
5. Understand and grow your app's user base (Play Console Help)
6. Run A/B tests on your store listing (Play Console Help)