Page tree

 

Contents

Evergage’s Campaign Statistics System makes it easy for you to scientifically measure the impact of your campaigns.  

Related Articles

See below for related articles

How is impact measured?

The objective for any A/B test or personalization campaign is to generate lift over control for a business goal.

  • Lift is a statistically significant improvement in the measured business goal
  • Control is the default experience without the change being tested or the personalization being applied
  • The business goal is whatever you choose to measure (and are able to measure). For example, it could be a clickthrough, a signup, the amount of time on a page or site, a purchase, average order value, revenue per user, or something else to be measured.  

How is lift measured?

Example 1

Lift is calculated by looking at the percentage increase of the goal value after running the campaign or:

[(Goal value for campaign) – (Goal value for control)] / (Goal value for control)]

Using the numbers from Example 1 at the right, this would be calculated as:

[ 84.69 - 77.13 ] / [ 77.13 ] = 9.8%

  

  

Example 1

 

The goal of this personalization campaign is to increase revenue/user. The control for this campaign is an experience without any recommendations. When the control is run, revenue per user is $77.13.  When the personalization campaign is run, revenue per user is $84.69.  

Example 2

Example 2 shows an A/B test campaign. 

The Experience appears to be generating 34.9% lift, but with 0% confidence and inconclusive results, that may not be the case. Why is the confidence 0% and what does that mean? Learn more about statistical confidence.

Lift is calculated by looking at the percentage increase of the goal value after running the campaign or:

[(Goal value for campaign) – (Goal value for control)] / (Goal value for control)]

Using the numbers from Example 2 at the right, this would be calculated as:

[1.36- 1.01] / [1.01] = 34.9%

 

Example 2

The goal of this personalization campaign could be anything (a clickthrough, joining a segment of interest, signing up, spending a certain amount of time on the site). When the control is run, visitors achieve the goal at a rate of 1.01%. When the new Experience 1 is shown, visitors achieve the goal at a rate of 1.36%. 

 

How does Evergage calculate attribution for goal completion?

What counts as a goal completion? What counts in the revenue/user, clickthrough, signup, segment membership, average order value calculations? Evergage only calculates results as part of the analysis if a visitor meets the following criteria:

  1. Qualification–the visitor must have qualified to see the campaign, regardless of whether they are in the test group and see the campaign or are in the control group and do not
  2. Completion–the visitor achieved the campaign goal after having seen (or qualified to have seen) the campaign. Similarly, attributed revenue per user is counted for visitors who have seen the campaign and made a purchase 
  3. Time Range–Evergage attribution only considers activity inside the time frame selected. For a purchase to be attributed, both the Impressed Visit (IV) and the goal event (think of purchase, click or goal achievement) need to happen within the timeframe selected. For more information, please refer to the article on Campaign Statistics – Attribution

 

Why might your campaign statistics differ from the configured user percentage split on an A/B test?

If you adjust user percentages in your A/B test campaign after the campaign is published, you may not see campaign statistics in line with the percentages you set. For example, if you published a campaign with a 10% control user percentage and 90% experience 1 user percentage, then one week later changed the control user percentage to 50% and experience 1 to 50%, the campaign statistics would not show a 50/50 split for the campaign. It is important not to ignore the influence previous experiences may have on return visitors who qualified for the campaign. Evergage will continue to show the same version (control or experience) to visitors who qualified for the campaign prior to the change. Visitors seeing the campaign for the first time after the change will see the control or experience in the same ratio as defined in the updated A/B test.

 

Why is "confidence" needed?

Lift tells us how much better the campaign is doing than control for a goal of interest. Confidence tells us how sure we are of that lift.  Why do we need confidence?

  1. Don’t make big conclusions from small amounts of data. In Example 2 above, Experience 1 is doing 35% better than Control. But, if you look at the Goal Completions column (right), only 33 people’s actions are contributing to that Goal Completion Rate, which is not enough to be statistically significant
  2. The patterns matter. As explained in Campaign Statistics – Confidence, the patterns in the data can make you more or less sure of the lift result

 

 

 

 

 

How does Evergage calculate "confidence"?

Evergage uses Bayesian analysis to continuously calculate and update the confidence we have in every lift percentage. By default, if Bayesian confidence calculation is:

  • Greater than 85%–Evergage displays the % of confidence and whether the campaign is winning or losing vs the control
  • Less than 85%–Evergage still shows the lift, but displays that results are inconclusive

However, Evergage can configure reporting to have a different threshold for confidence. Please contact your Customer Success representative for guidance.

For more information about interpreting campaign statistics data please refer to the Campaign Statistics Overview article.

 

 

 

 

Examples of how statistical confidence is displayed:

This page has no comments.