Dana Aeschliman

Statistical Consulting - Consultant en statistique

Data Science - Science des données

AB testing scenarios and python code for resolving each of them

Dec 1, 2014. Having worked in the field of Business Intelligence (BI) for the last several years, I've noticed that marketers frequently need help in designing / analyzing A/B tests. It's for this reason that I offer the concise Python codes clickable from this page.

Scenario 1. I've finished an AB test.

I have the number of individuals queried in each of two groups and the number (or percent of the group) responding favorably in each group.

Scenario 2. I'm getting ready to do an AB test and have ballpark ideas of my two response rates. What's the minimum size of the list that I will need to query?

I have an idea of :

the habitual response rate, and
the lift or improvement in the habitual response rate that I can reasonably expect.

Scenario 3. I'm getting ready to do an AB test and have chosen my list to query. How should I split it?

I have an idea of :

the size of the list that I will query, and
the habitual response rate, and
the best possible lift that I can hope to get above the habitual response rate.

Scenario 4. I'm getting ready to do an AB test and have chosen my list to query. What's the minimum increase in the percent responding favorably that I'll be able to correctly detect (i.e., what size lift will I be able to correctly detect)?

I have an idea of :

the size of the list that I will query, and
the habitual response rate, and
the number of individuals that I'll allocate to each of the two groups.