The Tyranny of Numbers

What gets measured gets done is a popular principle in business management, first articulated by Austrian-American author/management consultant Peter F. Drucker. In the era of big data, the idea is applied to many different things outside business management, from law enforcement to fitness. New public management is the government application of this quantification methodology, schools are measured by the results they produce in terms of grades, national standard test results or graduated students. Hospitals bill public insurance authorities for treatments and procedures, one ward buying services from another. But the focus on numbers hide the bigger picture, creating misled priorities and taking away responsibility from the individual doctor or teacher instead creating an ever-growing control bureaucracy attempting to measure and control every minute detail.

The “quantified self”-movement uses similar methods to improve quality of life. By measuring calorie intake, sleep cycles, exercise, heart rate and many other factors, they believers hope to create better bodies and minds. By measuring everything, we can gain control and change behaviour, the thinking goes. Belarus-American writer Evgeny Morozov slams this idea in his 2013 book To Save Everything, Click Here (Public Affairs), suggesting it loses sight of context, understanding and the bigger picture. While quantified self might influence a particular metric, it will not inspire a more profound lifestyle change.

In a Wired-column earlier this year, financial journalist Felix Salmon discusses the rise of the “quants” – the bean counters. He details their journey from sports to finance to government and finds a pattern of four stages. In “pre-disruption”, decisions are made by people based on experience, established consensus and gut feeling. Enter the quants and the phase two is “disruption”, where algorithms, numbers and analysis trump the old systems. Think of targeted advertising in social media, compared to the carpet bombing communication strategies of old. Phase three, what Salmon calls “overshoot”, is when things get really interesting: the quants create new systems that are vulnerable to gaming. Such as the eCPM-business of online marketing, with millions of Twitter followers sold for pocket change, as Rhoda Crockett has described in a series of Netopia-columns.

The phenomenon can be observed in many places, but to take one European example: the fuel consumption test cycle, known as the NEDC (“New European Driving Cycle”), is designed to encourage auto makers to build more fuel efficient vehicles. In practice, that means smaller engines with turbo-chargers for peak power capability and on-demand systems such as electric servo-assisted steering (as opposed to the old, always-on, hydraulic power steering). The problem is that auto makers design for good scores on the NEDC tests, not necessarily real-life driving. Many drivers experience much higher consumption than the advertised rate because they end up using the turbo-charger a lot more than the test cycle suggests. Worst case, NEDC could make some cars less fuel-efficient than older models in real life.

Salmon takes an example from police forces measured on number of arrests (rather than severity of crimes or resulting court convictions) and I hope readers will forgive me for taking another example from motoring: while express highways with several lanes and broad center dividers and shoulders are by far the safest roads to drive, in spite of high speed limits, they are also an easy target for traffic police who need to write a few more fines at the end of the month, as it is easy to find speeding drivers on such roads. Not that increases traffic safety much, but it makes the numbers look better.

Can infinite granularity make the quant method work? Would it be possible, even in theory, to make a system based on quantified results without the drawbacks of gaming or misled priorities? Salmon says no, his fourth stage is the synthesis of quant algorithms and established understanding. Informed decisions should draw from both sources. Quantity and quality in harmony. When the post-ideological legislators fly the flag of evidence-based decision-making, it’s worth keeping this point in mind. Evidence is subject to interpretation and will vary with context. Quantitative models rely on assumptions that are often based on ideology or bias. And while facts may look like convincing evidence, it is just as true that they can be ideology masquerading as evidence. Regardless of circumstance, be it finance, sports, fitness or government, numbers can be deceiving.

It’s true that what gets measured gets done, the problem is what doesn’t get measured does not get done.

Per Strömbäck,
Editor Netopia