Posts Tagged “A/B testing”

Most of what I have written this year about e-mail marketing has been complaints.  So these are my New Year’s resolutions for e-mail marketers:

1.  Target your e-mails.  Resist the temptation to blast everyone on your list regardless of whether they will be interested in what you have to say.  A good e-mail is timely and relevant.  If you send out too many e-mails, your recipients will report your e-mails as spam, hurting your reputation and possibly your ability to send e-mails in the future. 

2.  Send trigger emails.  I am a big fan of Barnes and Noble.  I love purchasing books on-line and they make it so easy for me.  For example, their website indicates how quickly each book typically ships.  When I place an order, I receive confirmation almost instantaneously and then am kept abreast of the shipping status of my order.  I love knowing exactly where my books are and when I can expect to receive them.  As a result, I appreciate trigger emails and expect them to be timely.  If I sign up for a new service on-line, I expect to receive a welcome e-mail within 24 hours, if not sooner.  I am amazed and disappointed by organizations that do not send trigger emails as they are important for reinforcing the relationship and offer an up-sell or cross-sell opportunity. 

3.  Create a preference center and follow it.  Allow subscribers to determine the frequency, content and even form of communication.   DailyLit is a great example I wrote about.  DailyLit allows subscribers to choose the amount of text they receive, the frequency and timing of communication and whether users receive emails or RSS feeds.  Thus, their communication is more likely to be read.

4.  Create your emails with image blocking in mind.  I wrote about image blocking in one post and then had to resist doing it again and again as I received more and more e-mails that clearly were not designed for e-mail providers who automatically blocked images.

5.  Reactivate or eliminate inactive e-mail subscribers.  As I noted in an earlier post, it is nice to be asked if you want to continue to receive emails from an organization.  This gentle reminder reengaged me and reestablished a relationship.  Alternatively, marketers could create a formal reactivation campaign as part of their campaign cycle. 

6.  Measure your campaigns and continuously learn.  I believe strongly in testing and measurement, comparing campaigns to benchmark rates or past campaigns, and determining the return on investment (ROI) of campaigns.  In the end, everyone one wants to know what worked, what did not work and whether the campaign was successful.  If you are interested, past posts have provided sources for e-mail metrics and a discussion of A/B testing

Happy New Year!

Comments 2 Comments »

I was catching up on Tangyslice’s blog and enjoying his 5 meaningless marketing metrics post, when I thought of another meaningless metric.  Last week I was reading a presentation which described the response rate of one group as slightly greater than the control group.  What does slightly greater mean in this context?  Well, it turns out the difference was statistically significant once I did the math.  What I find meaningless is when analysts do not look for statistical significance when comparing two groups.  This is known as A/B Testing.

Conceptually A/B testing is very simple.  You are comparing Group A to Group B.  A might be a control group and B the test group.  Alternatively, A and B might be two different offers, landing pages, e-mails, direct mail lists, or landing pages.  As the name suggests, this is a test which is why A/B testing is also known as split testing.  Ultimately, you want to know if A and B differ in a way that is statistically significant.

Here’s an example to make it concrete. Let’s say that you marketed to 50,000 customers encouraging them to purchase product A and 5,000 of them responded. That is a 10% response rate. In addition, there were 5,000 customers that you could have marketed to but that you did not.  Instead, you assigned them to the control group.  They look and act just like the 50,000 customers that you mailed. The reason for the control group is that some customers might buy product A regardless of whether you market to them or not.  In this example, 450 of them or 9% purchased the product. Is the difference between 10% and 9% statistically significant?  Was the campaign successful?

In this case, we perform the two-proportion z-test for equal variances using the following formula:

z=\frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1 - \hat{p})(\frac{1}{n_1} + \frac{1}{n_2})}} and
\hat{p}=\frac{x_1 + x_2}{n_1 + n_2}

where…

p1=10% (response rate for Group A)

p2=9% (response rate for Group B)

x1=5,000 (number of responders in Group A)

X2=450 (number of responders in Group B)

n1=50,000 (quantity mailed in Group A)

n2=5,000 (quantity mailed in Group B)

If the value of z is greater than 1.96 then the difference is significant at 95% confidence.  In this case, the z value is 2.26 so the difference is statistically significant.

In order for the test to be valid a few assumptions must be met:
1. Your control group needs to contain customers or prospects that look and behave like the treatment group
2. You need to have sufficient numbers of direct mail recipients and responders such that n1 p1 > 5 AND n1(1 − p1) > 5 and n2 p2 > 5 and n2(1 − p2) > 5 and n2>29 and the groups contain independent observations

The math might look scary but really the hard part is making sure that the test is done properly.  It is vital that the control contains a random selection of customers who are similar to the treatment group.  If not, you could end up with very strange results

Comments No Comments »

This is a blog about marketing analytics.  I am a direct marketing professional who loves marketing, strategy and analysis and I welcome your thoughts and feedback on these topics as well.

 

First, let me tell you a little more about me.  At a dinner several years ago, a client announced to my colleagues that he knew how to make me smile.  He simply had to say that he had data.  It is very true; data makes me happy.  I love analyzing data because it can answer business questions and provide insight into marketing challenges.  At the end of the day, your recommendations are grounded by data. 

 

That is one of the reasons that I work in direct marketing.  There are lots of data elements that can be analyzed and almost endless possibilities.  For example, you can create models for targeting, conduct A/B testing, analyze response rates and calculate ROI (return on investment) by customer segments. 

 

This blog will discuss current trends in marketing analytics, various techniques and the field more generally. 

Comments 1 Comment »

© Lynne Harrold. All rights reserved.