TMAI #120: Don’t Measure Everything.

Click here to view original web page at

Something surprising coming from an author of two best-selling books on analytics: A plea to stop measuring everything.

Given the complexity of the initiatives we undertake, the desire to measure everything (even the unmeasurable) extracts a cost that leaders may not fully appreciate.

I call these unmeasurables faith-based initiatives. I do so with respect.

On top of the raw costs, faith-based initiatives have an additional cost below the water-line: Because they end up using questionable methodologies and metrics that stretch the limits of sanity, they produce false success or failure signals. Ill-informed decisions follow.

Today's example came from a long walk I took with a dear friend (why talk about the flowers or politics or family when there's a juicy analytics topic to be discuss! :)).

Your company decides to sponsor a leading golfer, Mr. X. In exchange, no matter where he is, he has to wear a hat with the initials of your company.

As the analyst, your CEO is going to ask you: So, what did we get for our money?

You, an overachieving smart person, will immediately jump to assessing the GRPs for the golf tournaments on TV during which Mr. X was playing golf. You share that number with your leader framed as the number of eyeballs reached. Everyone’s smiling at the “exposure” from the $22 mil deal.

After a couple months, someone smart will look at the idea of “exposure” and realize that it may not be enough to justify a $22 mil deal. They’ll ask for smarter measurement.

Undeterred, you partner with a specialized agency to help measure brand awareness. Surveys are sent to 80 or 120 people, results are analyzed, and lo and behold — Awareness is up 21%.

This will work for a while, but it will gnaw at the soul of the smart leader that Awareness is at best activity and not an outcome.

Then, you’ll be asked but what are the chances that these people who see Mr. X’s hat will give us a multi-million dollar contract the next time they are looking for strategic consulting?

You wonder how much can be expected from a poor hat. But, you go off and do some more surveys “targeted” at CEOs (even though you know that no CEO worth their salt is going to fill out a survey). You are able to prove Consideration and even Purchase Intent.

Analytically speaking, this is on really, really thin ice. You still feel good because by now this has taken a few months out of your life.

Then... You are asked to use a Media Mix Model to prove attributable impact of the hat compared to all the other marketing activities your company is doing.

Realizing the insanity of what is being asked of you for this faith-based initiative, you finally ask the question: Why are we sponsoring Mr. X’s hat?

Answer: The CEO loves golf and the hat gets her invited to the hospitality tent. There, she likes to spend quality time with Mr. X and other top golfers. Often, possible clients of your company are there as well.

You finally realize what proving the unmeasureable really means.

Two important things.

Faith-based initiatives are important. Some of the best marketing campaigns were based on faith.

Mr. X's hat should not be thought of as a useless exercise. Some people will certainly notice the hat when he plays golf or when he is doing post-game interviews. Maybe one day, one of those people will become the CxO of a fortune 500 company. Perhaps he will remember Mr.X and his hat when his company wants to hire your consulting company. I mean this sincerely.

The problem is that nothing you are measuring can actually assess that outcome.

Yet, measurement is being demanded - real-time if possible. This imposes a massive opportunity cost on people, process, priorities. This negative impact is rarely accounted for.

Here’s a little sketch I made to illustrate the impact:

faith initiatives measurement impact

While faith-based initiatives are few in number at any company, they usually tend to have high leadership sponsorship. Hence, they become immediate priorities.

Measuring these initiatives is impossible or really, really difficult. Hence, they take up way more measurement resources. This, in turn, cuts into the measuring the absolutely measurable non-faith-based initiatives such as search, display, prime-time TV ads, etc. This has an often overlooked negative impact on non-faith based initiatives.

From an overall company perspective, you get questionable data back for the faith-based initiatives and no data for the non-faith-based initiatives. A disappointing lose-lose.

Hence, my recommendation: Be brave, be bold, and don’t measure faith-based initiatives.

Wanting to measure is so intrinsic to our belief system that it may feel like I’m asking you not to breathe. If it helps you make the leap, here’s the impact you can expect:

non faith initiatives measurement impact

An objectively incredible increase from 35 to 140.

Awesome. But, why?

In this case, your measurement resources (people, process, tools) are focused on what’s actually measurable. They have more time to measure what’s real, so they can focus their time and energy on modeling, proving incrementality, testing hypotheses, and doing a million other things that add predictable value to your enterprise.

Hence, much as you think this sounds crazy: Faith-based initiatives should be accepted as a matter of faith.

Your CMO wants the hand towels at Wimbledon to sport your company logo because she loves tennis and all her friends and peers watch Wimbledon. It is her right to decide to allocate money to this. What is the measured impact? Nothing, just CMO happiness.

Your CMO wants to have a luxury tent at the Super Bowl where he can host your top clients. The hypothesis is that this will yield results over five years. It is his right to decide to allocate money to this. Measured impact? Nothing. Anything worth measuring (incremental revenue, happiness) can’t be measured for a few years anyways. Why use up valuable resources now to show some fake analytical impact? Just use faith.

There are plenty more scenarios just like these.

If it helps... In my experience, faith-based initiatives are a small part of any marketing team’s overall strategy. Perhaps 10 or 15% of the overall budget. In a world where even the most confident measurement is full of assumptions, why expend extraneous people, processes, and tools trying to measure that small number?

Let faith assess the impact of faith-based initiatives.

Focus your precious and limited people, processes, and tools on thoughtfully measuring your 85 to 90% non-faith-based initiatives.

Bottom line: No company has such abundant resources that they don’t have to think critically about resource allocation. It is against deeply ingrained human nature to say: We won’t measure that. I hope I’ve convinced you of the immense value of being able to say exactly that when you must.

How much of your time as an Analyst/Marketer is spent chasing answers when you know you can’t find any worth finding? Is half of your time spent measuring faith?


PS: You might believe that this is only a problem for large companies. It is not. Being sent on wild chases with little critical thinking applied is a universal challenge.