I ran a 5K race recently and so I have been thinking about my pre-race routine.  I may not have a lucky rabbit’s foot but the morning of a race I have a ritual of sorts.  I eat my usual breakfast and of course drink coffee.  There are some things I just can’t do without!  I wear the same clothes and sneakers for the race as I wore training.  I will not do anything new or different.   

This ritual keeps me from being distracted so that I can concentrate on the race.  In this case, my ritual helps me.  But in the office, rituals can be limiting.  Always doing something the same way can get old and stale.  A colleague asked me about identifying best customers.  My first thought was an RFM or RAD segmentation because I was in the midst of a RAD segmentation.  It would have been easy to stop there.  However, I couldn’t stop until I also suggested clusters and CHAID.   If she had let me, I would have added modeling and NPV.   The trick is knowing when to stick with rituals and when to avoid them.

I was catching up on Tangyslice’s blog and enjoying his 5 meaningless marketing metrics post, when I thought of another meaningless metric.  Last week I was reading a presentation which described the response rate of one group as slightly greater than the control group.  What does slightly greater mean in this context?  Well, it turns out the difference was statistically significant once I did the math.  What I find meaningless is when analysts do not look for statistical significance when comparing two groups.  This is known as A/B Testing.

Conceptually A/B testing is very simple.  You are comparing Group A to Group B.  A might be a control group and B the test group.  Alternatively, A and B might be two different offers, landing pages, e-mails, direct mail lists, or landing pages.  As the name suggests, this is a test which is why A/B testing is also known as split testing.  Ultimately, you want to know if A and B differ in a way that is statistically significant.

Here’s an example to make it concrete. Let’s say that you marketed to 50,000 customers encouraging them to purchase product A and 5,000 of them responded. That is a 10% response rate. In addition, there were 5,000 customers that you could have marketed to but that you did not.  Instead, you assigned them to the control group.  They look and act just like the 50,000 customers that you mailed. The reason for the control group is that some customers might buy product A regardless of whether you market to them or not.  In this example, 450 of them or 9% purchased the product. Is the difference between 10% and 9% statistically significant?  Was the campaign successful?

In this case, we perform the two-proportion z-test for equal variances using the following formula:

and

where…

p1=10% (response rate for Group A)

p2=9% (response rate for Group B)

x1=5,000 (number of responders in Group A)

X2=450 (number of responders in Group B)

n1=50,000 (quantity mailed in Group A)

n2=5,000 (quantity mailed in Group B)

If the value of z is greater than 1.96 then the difference is significant at 95% confidence.  In this case, the z value is 2.26 so the difference is statistically significant.

In order for the test to be valid a few assumptions must be met:
1. Your control group needs to contain customers or prospects that look and behave like the treatment group
2. You need to have sufficient numbers of direct mail recipients and responders such that n₁ p₁ > 5 AND n₁(1 − p₁) > 5 and n₂ p₂ > 5 and n₂(1 − p₂) > 5 and n2>29 and the groups contain independent observations

The math might look scary but really the hard part is making sure that the test is done properly.  It is vital that the control contains a random selection of customers who are similar to the treatment group.  If not, you could end up with very strange results

Last week I attended a Marketing Analytics conference  in Boston sponsored by iKnowtion.  At a time when companies are cutting expenses, including staff and marketing budgets, iKnowtion is investing in their future.  They are also engaging in a dialogue with the larger Marketing Analytics community through the conference and their blog.  (In the interest of full disclosure, I know several people at iKnowtion but have never worked there.)

The conference began with a talk by Tom Davenport, the Author of Competing on Analytics with Jeanne G. Harris.  He set the stage by providing examples of how companies recognize the importance of analytics but reminded us that marketing is still a combination of art and science.  As the emphasis shifts more towards the science of marketing, we need to recognize that the “art” is still relevant.  He further challenged us to move beyond reporting to provide more value and insight.

Next was a panel on driving business value featuring speakers from GM, CVS Pharmacy, and ConstantContact.  Each speaker provided a brief case study of how analytics has helped their business.  In one case, analytics changed the focus of the business.  In another, it led to the rebalancing of product marketing.  Finally, the rigors of “test, measure, and learn” enabled one company to optimize media effectiveness across channels.

After lunch there was a lively digital panel discussion around social media, the future of web-enabled communities and the challenges of measuring the impact of companies’ efforts in this space.  Given the evolving nature of social media, it is no surprise that there were divergent opinions.  I, for one, appreciated the candor and the healthy discussion that ensued.  To quote Jane Austen, “My idea of good company…is the company of clever, well-informed people, who have a great deal of conversation.”

The conference wrapped up with a return to the theme of competing on analytics.  This free flowing discussion touched upon a range of topics, including how to become a company that uses analytics for competitive advantage.  Interestingly, one of the panelists thought that finding good talent was the biggest challenge we face.  As a Marketing Analytics professional who hires and develops staff, I am in complete agreement.  There is stiff competition for the best analytic staff and I have found it difficult to find technical competence coupled with business acumen.  In fact, the discussion about finding, training and retaining analytic staff continued at the bar, after the conference formally ended.

iKnowtion has plans to hold the conference again next year and I encourage you to attend.

Do you spend a lot of time on YouTube? If so, you may have already seen this but I was surprised to see the MC Hammer video on YouTube.  Am I the only one surprised to hear the words “behavioral targeting” being spoken by MC Hammer?  Who knew that MC Hammer and I would have something in common.  We both believe that analytics enables you to allocate your marketing dollars effectively.

If you haven’t seen it, watch MC Hammer on Analytics from YouTube.