I was catching up on Tangyslice’s blog and enjoying his 5 meaningless marketing metrics post, when I thought of another meaningless metric.  Last week I was reading a presentation which described the response rate of one group as slightly greater than the control group.  What does slightly greater mean in this context?  Well, it turns out the difference was statistically significant once I did the math.  What I find meaningless is when analysts do not look for statistical significance when comparing two groups.  This is known as A/B Testing.

Conceptually A/B testing is very simple.  You are comparing Group A to Group B.  A might be a control group and B the test group.  Alternatively, A and B might be two different offers, landing pages, e-mails, direct mail lists, or landing pages.  As the name suggests, this is a test which is why A/B testing is also known as split testing.  Ultimately, you want to know if A and B differ in a way that is statistically significant.

Here’s an example to make it concrete. Let’s say that you marketed to 50,000 customers encouraging them to purchase product A and 5,000 of them responded. That is a 10% response rate. In addition, there were 5,000 customers that you could have marketed to but that you did not.  Instead, you assigned them to the control group.  They look and act just like the 50,000 customers that you mailed. The reason for the control group is that some customers might buy product A regardless of whether you market to them or not.  In this example, 450 of them or 9% purchased the product. Is the difference between 10% and 9% statistically significant?  Was the campaign successful?

In this case, we perform the two-proportion z-test for equal variances using the following formula:

and

where…

p1=10% (response rate for Group A)

p2=9% (response rate for Group B)

x1=5,000 (number of responders in Group A)

X2=450 (number of responders in Group B)

n1=50,000 (quantity mailed in Group A)

n2=5,000 (quantity mailed in Group B)

If the value of z is greater than 1.96 then the difference is significant at 95% confidence.  In this case, the z value is 2.26 so the difference is statistically significant.

In order for the test to be valid a few assumptions must be met:
1. Your control group needs to contain customers or prospects that look and behave like the treatment group
2. You need to have sufficient numbers of direct mail recipients and responders such that n₁ p₁ > 5 AND n₁(1 − p₁) > 5 and n₂ p₂ > 5 and n₂(1 − p₂) > 5 and n2>29 and the groups contain independent observations

The math might look scary but really the hard part is making sure that the test is done properly.  It is vital that the control contains a random selection of customers who are similar to the treatment group.  If not, you could end up with very strange results

If you have been reading Limeduck like I have, you might have read about a Rockport print ad showing shoes that are not available for purchase.  As I noted on that website, this has happened before.  In a Marketing class several years ago, my professor showed a television ad featuring a car that could not be purchased.  As you can imagine, consumers saw the car and went to their local dealerships looking for that car only to learn that it wasn’t available.  Given that high profile mistake, I am surprised that Rockport made the same gaffe.

And yet, just today I received an e-mail that I wanted to share.  In an earlier post, I talked about how some e-mail programs do not load images.  This was in the context of measuring the open rate of an e-mail.  However, the fact that some e-mail software turns images off by default also affects the look and feel of an e-mail.  Here’s what the e-mail looked like:

In the image above, pictures have been replaced by boxes featuring red squares, blue triangles and green circles.  All of the time spent crafting a beautifully designed e-mail is lost if recipients cannot quickly read about the offer(s) and easily engage with the e-mail. I certainly did not bother to display the images in this e-mail.

Clients ask me about e-mail open rates and, honestly, they are not what they used to be.  In fact, they no longer matter for many reasons but here are my top three:

1. False negatives.  An e-mail is considered opened when a tracking image is downloaded.   However, major e-mail clients like gmail disable images by default.   If you read the e-mail with the images disabled, it will never be counted as an open.   And what about text e-mails?  They do not include images and thus do not count as opens unless you click on a link (and even that might be e-mail software dependent).

2.  False positives.  Let’s assume that images are enabled.  E-mails displayed in a preview pane are considered opened because the images were downloaded.  But who always reads the e-mails in their preview pane?  I don’t and I bet you don’t either.   So you have undercounting due to the disablement of images and overcounting due to the use of preview panes.

3.  What really matters is the action taken.  To me, the true success of an e-mail marketing campaign is whether you drove the desired action.  Did you sell more widgets as a result of the e-mail campaign?  If not, the open rate is moot.