A/B Test

Important

🚧

Make sure you've added the products in Adapty Dashboard before creating an A/B test

🚧

Be sure you send paywall views to Adapty using the .logShowPaywall(). Without this method, Adapty wouldn't be able to calculate views for the paywalls within the test, which will result in the irrelevant conversion stats

👉 iOS | Android | Flutter | React Native

Creating A/B test

So, it's time to create our first A/B. There would be two ways of doing so!

Converting a paywall to A/B

This way would allow you to use your existing paywall, as an A/B test, without having to release a new version of your app.

To convert a paywall to an A/B test, click Convert to an A/B test at the right top corner of the paywall editor.

🚧

Note

It wouldn't be possible to change the paywall ID, once it's being automatically gathered from the paywall. In this use case. no additional adjustments on the SDK side are required

Creating an A/B from scratch

This way of creating A/B tests would require some adjustments on the SDK side. Before stepping to the further steps, check out our doc on displaying products

After filling out the required fields, you will want to add at least two paywalls to an A/B test, two have two different options to test

🚧

Check out our doc on paywalls

A/B test options

A/B test must have at least two options. An option is a paywall. In the example, the A/B test consists of two paywalls: Week + Year and Month + 6 Months + Lifetime. These are custom names that we've chosen for our paywalls. The first paywall has two products, the second has 3 products.

Every paywall within the A/B test has the weight. Weight represents the relative amount of views this paywall will get when running the A/B test. If the first paywall has 30% weight, and the second one has 70%, this means that the first gets 3 views out of ten, and the second gets 7 views. The sum of all weights must be 100.

You can use Balance weights button to evenly distribute the weights of the options.

To add a paywall to the A/B test, click Add Paywall button, create a paywall, and it'll be assigned to the A/B test.

Targeted A/B test options

Out-of-the-box, the A/B test is live for all the app users. But, sometimes, there could be a need in adding an A/B test on a specific segment of users. You can do this using a targeted option. First, you'll want to choose the segment you want to target. And then add the paywalls as usual.

For example, you want to provide a discount for users who cancelled a subscription.

We've added a targeted option for the users who cancelled a subscription. Keep in mind, that All users segment must be always targeted (you can't delete this option). With the setup below, a user with a cancelled subscription will be always (100%) shown the Month Discount paywall. If a user doesn't have a cancelled subscription, he'll be shown the paywalls from the All users target option. Targeted options may contain multiple paywalls, just as the normal ones.

📘

You can create multiple targeted options within an A/B test. If the customer belongs to multiple segments, Adapty will use the one with the least number of users in it. If the customer doesn't belong to any targeted option, he'll be shown the paywall from the All users target option.

📘

You can use predefined segments for targeted options. You can also create custom segments tailored to your needs. For example, you can create a segment for people from a specific ad campaign, source, or country. You can even use your app logic parameters to create a segment. For example, users who are interested in sports or users who completed 10 lessons.

Learn more about these options in the Segments section.

Versioning

Every time the paywalls, weights, or targeted options are changed, the new version of the A/B test is created. This is done to collect A/B test metrics separately for every meaningful change. The list of the A/B test shows the metrics for the latest (current) version of the A/B test. You can view the metrics for every version of the A/B test on the Paywall metrics page.

Metrics

Adapty collects plenty of metrics to help you better measure the performance of the A/B test. All metrics are updated in real-time, except for the views, which are updated once every several minutes. All metrics, except for the views, are attributed to the product within the paywall.

• Revenue is the amount of money in USD from the purchases and renewals. So every time the subscription is renewed, it adds up to the LTV metric. Calculated before the App Store / Play Store commission.
• The conversion rate (CR) to purchases is the number of purchases divided by the number of views. 10 purchases / 100 views = 10% CR purchases.
• The conversion rate (CR) to trials is the number of trials started divided by the number of views. 30 trials / 100 views = 30% CR trials.
• Purchases are the sum of the following transactions (renewals are not included):
  • New purchases made on the paywall;
  • Trial conversions of trials that were activated on the paywall;
  • Down/up/cross-grades of the subscription on the paywall;
  • Subscription restores on the paywall (e.g. if the subscription was expired without auto-renewal).
• Trials are the number of activated trials
• Trials cancelled are the number of trials with auto-renewal switched off. It happens when the user manually unsubscribes from the trial.
• Views are the number of views of the paywall. If the user visits the paywall two times, this will be counted as two visits.

🚧

Make sure to send paywall views to Adapty using .logShowPaywall() method. Otherwise, paywall views will not be accounted for in the metrics and conversions will be irrelevant.

You can view the A/B test metrics for every version, platform (OS), and targeted option.

The best performing option is highlighted in green. By default, the best one chosen based on the revenue. You can choose the other options at the right top corner of the metrics.

You can also change the sorting of the table by clicking on the label.

When you choose two options to compare by clicking checkboxes next to the titles, Adapty will show you whether one of them is significantly better than the other. If so, it usually means that you can stop testing these options and choose the best one.

Stopping an A/B test

Once you've discovered the best-performing option, you may want to stop an A/B test. Stopping an A/B test is essentially converting it to a paywall. This means that all users of your app will see the same products.

To stop an A/B test click Stop A/B test at the right top corner of the A/B test editor.

You'll be asked to choose the paywall you want to keep.

All stopped A/B tests are stored in the Inactive tab, and you can still view the metrics.