Numbers, data, performance indicators, graphs … in our modern-day world they rule. They rule our decision-making, they rule our courses of action, they rule our work and often also our personal lives. And yes, they also play an important part in high performance organizations: HPOs explicitly measure everything that matters to the organization’s performance and report this financial and non-financial information to organizational members so they know how they are doing and thus can improve their performance. However, there is one problem with numbers, or to be more accurate, with the people that are using these numbers. They are often seen by these users as objective, ‘hard’ data while in reality they are frequently far from objective. They can be measure and collected in the wrong way, be misrepresented, be manipulated, be just plain wrong. And even when the numbers are correct, they can be misinterpreted by the users because they did not look at the context of these numbers. A nice example (taken from Nierop & Wielaard) is the fact that sailors who went overboard without a life jacket are saved more often than sailors who did wear a life jacket. So, despite going against our intuition, the conclusion is clear: don’t wear lifejackets when on a boot because that will significantly reduce your chances on survival in case of disaster. However, looking at the context will lead you to a completely different conclusion. It turns out that the sailors chose to wear a lifejacket especially in bad weather conditions, circumstances in which rescue was difficult or sometimes impossible and more sailors perished.
And thus we encounter the so-called ‘perverse effects’ of numbers: we think we make the right decisions based on good, hard data while in actuality we are led on the wrong path. Also, numbers are regularly used to make a point, after all, a graph with numbers says more than a 1000 words. But if this graph is based on erroneous numbers it doesn’t say anything at all; so while we think we understand an issue the opposite is the case. And the scary thing is: this happens more often than you think. Newspapers, research reports, news items, they quite frequently turn out to be based on shaky ‘number grounds.’
Recently two interesting books were published discussing the problem of incorrect, erroneous and false numbers and how you can arm yourself against them. These Dutch books are called Spookcijfers (Ghost Numbers) and Het Best-verkochte Boek Ooit … met deze titel (The Best-selling Book Ever… with this title). The authors give some good advice regarding how to recognize and deal with numbers and I did not want to withhold this advice from you, the non-Dutch reader of my blogs. So here goes.
When the numbers originate from a survey, ask yourself the following questions:
- Who is the client and/or financer of the survey: who has a vested interest in the outcome of the survey so objective are the results?
- How was the survey structured, who was surveyed and how: what is the quality of the process and how representative is it?
- Which questions were asked and which answering possibilities were given: how leading (to certain answers) was the survey?
- How big was the sample and how was it obtained: how random was the responding population?
- How big was the response in percentages and absolute numbers: 1% of 100 is quite different from 1% of 10.000 respondents.
- Has non-response been taken into account and how: because if non-respondents would have answered the survey, the outcome could have been very different.
- Are uncertainty percentages given: how sure are we that the results represent the actual situation?
When dealing with numbers, use these guidelines:
- Use mind and feeling when interpreting numbers (do they ‘feel right’, if not, investigate)
- Use the five w’s: do the numbers tell you who, what, where, when and why?
- Look at how the numbers were obtained.
- Distrust too much accuracy, this might be done to give you a feeling of false certainty.
- Ignore numbers that are used to impress, they are often exaggerated.
- Always look at the context of the numbers.
- Do not be afraid to look dumb: it you don’t understand the numbers of distrust them, start asking questions.
- Be curious, investigate whether the numbers and their sources are trustworthy.
Also, always keep the ‘Law of Goodhart’ in mind: when an indicator becomes a target, then it no longer is a good indicator. Because people will always find ways to manipulate the numbers to fit the target. Finally, look at how people in HPOs treat numbers and indicators: as interesting information, as warning signals, as triggers to act in order to remedy problems or take advantage of opportunities, while always taking the context into account and not elevating the numbers to the absolute unassailable truth.
More information in Dutch:
Sanne Blauw (2018), Het-best verkochte boek ooit … met deze titel, hoe cijfers ons leiden, verleiden en misleiden, De Correspondent
Tom Nierop & Nart Wielaard (2019), Spookcijfers, over misleidede cijfers en hoe je ze herkent, Business Contact