Something surprising coming from an author of two best-selling books on analytics: A plea to stop measuring everything. Given the complexity of the initiatives we undertake, the desire to measure everything (even the unmeasurable) extracts a cost that leaders may not fully appreciate. I call these unmeasurables faith-based initiatives. I do so with respect. On top of the raw costs, faith-based initiatives have an additional cost below the water-line: Because they end up using questionable methodologies and metrics that stretch the limits of sanity, they produce false success or failure signals. Ill-informed decisions follow. Today's example came from a long walk I took with a dear friend (why talk about the flowers or politics or family when there's a juicy analytics topic to be discuss! :)). Your company decides to sponsor a leading golfer, Mr. X. In exchange, no matter where he is, he has to wear a hat with the initials of your company. As the analyst, your CEO is going to ask you: So, what did we get for our money? You, an overachieving smart person, will immediately jump to assessing the GRPs for the golf tournaments on TV during which Mr. X was playing golf. You share that number with your leader framed as the number of eyeballs reached. Everyone’s smiling at the “exposure” from the $22 mil deal. After a couple months, someone smart will look at the idea of “exposure” and realize that it may not be enough to justify a $22 mil deal. They’ll ask for smarter measurement. Undeterred, you partner with a specialized agency to help measure brand awareness. Surveys are sent to 80 or 120 people, results are analyzed, and lo and behold — Awareness is up 21%. This will work for a while, but it will gnaw at the soul of the smart leader that Awareness is at best activity and not an outcome. Then, you’ll be asked but what are the chances that these people who see Mr. X’s hat will give us a multi-million dollar contract the next time they are looking for strategic consulting? You wonder how much can be expected from a poor hat. But, you go off and do some more surveys “targeted” at CEOs (even though you know that no CEO worth their salt is going to fill out a survey). You are able to prove Consideration and even Purchase Intent. Analytically speaking, this is on really, really thin ice. You still feel good because by now this has taken a few months out of your life. Then... You are asked to use a Media Mix Model to prove attributable impact of the hat compared to all the other marketing activities your company is doing. Realizing the insanity of what is being asked of you for this faith-based initiative, you finally ask the question: Why are we sponsoring Mr. X’s hat? Answer: The CEO loves golf and the hat gets her invited to the hospitality tent. There, she likes to spend quality time with Mr. X and other top golfers. Often, possible clients of your company are there as well. You finally realize what proving the unmeasureable really means. Two important things. Faith-based initiatives are important. Some of the best marketing campaigns were based on faith. Mr. X's hat should not be thought of as a useless exercise. Some people will certainly notice the hat when he plays golf or when he is doing post-game interviews. Maybe one day, one of those people will become the CxO of a fortune 500 company. Perhaps he will remember Mr.X and his hat when his company wants to hire your consulting company. I mean this sincerely. The problem is that nothing you are measuring can actually assess that outcome. Yet, measurement is being demanded - real-time if possible. This imposes a massive opportunity cost on people, process, priorities. This negative impact is rarely accounted for. Here’s a little sketch I made to illustrate the impact: |