Abstract
I began reading Jerry Muller’s (2018) The Tyranny of Metrics around the time I received yet another ‘strategic plan’ for our Faculty. As any plan necessitates, it came complete with goals, initiatives, actions, key performance indicators (KPIs), and timelines. There were goals such as ‘pursuing excellence’ which translated to activities such as ‘publish in well-respected journals’ and measured by KPIs such as ‘number of publications’. As expected in an era which Muller appropriately calls ‘the era of metric fixation’, the measurements for ‘excellence’ were specific: one column showed the average ‘per capita publications’ while another showed ‘leading publications,’ as these were defined by yet another measure, the ABS list and rankings. A final column labeled ‘targets’ translated what ‘pursuing’ meant: ‘increase citations by +0.5% and leading publications to 0.3 per capita per annum’. Unsurprisingly, these measures were tied to another document outlining more numerical measures for promotion and tenure. As ironic as it may seem to ‘pursue these measurable efficiencies (which are clearly) at odds with the strategic thinking required’ for such complex situations and such complex outputs as ‘creating new knowledge’, this is all quite common behavior, according to Muller, from universities to schools to hospitals to prisons. In addition, it is also indicative of what he calls the ‘spreadsheet worldview’ (p. 47).
I begin my review of this fascinating and informative book with the above vignette because I felt that every line I read was an accurate description of our daily organizational (and academic) lives. We live in a metrics obsessed world in which the ‘uncritical adoption of metric ideology’ (p. 9) is the norm. In this context, without demonizing metrics and – on the contrary – carefully pointing out the situations in which measurements are useful, Muller, a professor of history at the Catholic University in Washington D.C., notes that ‘while we are bound to live in an age of measurement, misleading mismeasurement and counter-productive measurement’ (p. 4) seem to have taken center stage. As he aptly comments in the Introduction, the book is ‘not about the evils of measurement’. It is about ‘the unintended negative consequences of trying to substitute standardized measures of performance for personal judgment based on experience’ (p. 4).
Muller’s book is timely and important. It comes in the middle of heated debates in the United Kingdom and the United States about the future of business schools and academia in general, amid the protests of Polish academics about proposed changes in the governing structure of their universities, as well as wider scholarship in organization studies criticizing ‘a metric ideology’ (p. 9). This well-written, accessible book, which I consider a must-read for all policy and decision-makers in academia and all critical scholars, follows what is becoming a vast literature concerned with the future of university education and the problematic repercussions of university rankings, the ABS list, the REF and TEF exercises in the United Kingdom, which is too lengthy to mention here, 1 as well as an array of articles in the daily press, 2 blogs, 3 even collectives. 4 Muller’s book comes to join these conversations but also ties their different threads in a comprehensive monograph that is clearly argued and thoroughly researched. The wealth of examples is admirable, and their poignancy is noteworthy.
Another reason to recommend this book is its fascinating historical perspective. Chapter 3 specifically discusses the roots of the discourse and how ‘accountability’, ‘metrics,’ and ‘performance indicators’ became cultural memes. Tracing ‘the tyranny of metrics’ to policymakers in Victorian England, Muller explains how the conversation crossed the Atlantic and became tied to Taylorist principles around 1911 and then mainstreamed into public policy by Robert McNamara, an accountant who rose from Harvard Business School professor to Kennedy’s Secretary of Defense. The historical perspective is evident throughout the book and is absolutely enlightening for the fields of medicine (Ch. 9), policing (Ch. 10), and the military (Ch. 11)—mentioning tragic examples such as the metric of ‘body counts’ as a reliable index of American progress in winning the Vietnam War.
One particularly illuminating example analyzed is Bush’s 2001 No Child Left Behind Act, discussed at length in the chapter on Schools (Ch. 8). The provenance of the Act, which was a precursor to Obama’s Every Student Succeeds, was the wish to deal with the disparity in educational attainment among ethnically or racially defined groupings. Although noble in its outlook, Muller argues that the legislation’s testing-and-accountability regime exemplifies first the components – and then the pitfalls – of ‘metric fixation’: the belief that it is possible and desirable to replace professional judgment with numerical indicators based on standardized data; the belief that making these metrics transparent ensures personal and institutional accountability; and the belief that the best way to motivate people in organizations is to reward or penalize measured performance. Sadly, several dysfunctions arose: not only did teachers start ‘teaching to the test’ by focusing narrowly on the subjects and skills tested but also ‘creaming’ (e.g. classifying weaker students as disabled so as to remove them from the assessment pool) or downright cheating (e.g. tossing out tests or altering student answers). (What is worse is that treating schools as ‘gap-closing factories’ did not even work: the gap in the scores of students from different race and class backgrounds was no different in 2013 than it had been in 1992 and in some cases was even larger). ‘Creaming’ and cheating are typical unintended consequences of metric fixation, as much as ‘skewing’ and ‘gaming’. The case of hospitals is particularly horrifying, since ‘treating to the test’ meant, for example, that cardiac surgeons became less willing to operate on severely ill patients lest they “mess up their stats” or kept patients alive for 30 days, if that is the cut-off point for measuring post-op success, but not for 31 days (Chapter 9: Medicine). In the chapter on Policing (Ch. 10), Muller gives yet another fascinating analysis of ‘gaming’ behaviors, using the London Mayor’s Office for policing and crime as an example. When the Office set a performance target of ‘20 percent reduction in crime’ without backing it up with the appropriate resources, ‘massaging statistics became an ingrained part of policing culture’ (p. 128): serious crimes such as robbery were downgraded to ‘theft snatch’ and rapes were underreported to meet targets.
I have purposely left Muller’s discussion of academia last. His Chapter 7 was too accurate and too painful to read. From university rankings to citations, impact factors and journal lists, Muller writes ‘here is a case where standardizing information can degrade its quality’ (p. 78). As he aptly notes, citations cannot distinguish between ‘Muller’s illuminating and wide-ranging book on the tyranny of metrics …’ and ‘Muller’s poorly conceived screed deserves to be ignored by all managers and social scientists …’ (p. 79). What’s worse, these auditing exercises do not just lead to inaccurate assessments, ‘they lure scientists into pursuing high rankings first and good science second’. Having sat through as many meetings where the work of a scholar was described as ‘she only has two 4s’ or ‘he is a high flyer, he has four 4-stars’ (the gendered nuance is deliberate), as the number of times I was asked about my own metric worth ( ‘Did you get a 4 yet?’), I admit to feeling a profound affinity with all the problematic scenarios Muller described. In addition, he voiced what many of us probably want to shout in such meetings: ‘There is a better way to evaluate the importance of a paper or the research output of an individual scholar. Read it!’ (p. 80).
The discourse reflected in my Faculty’s strategic plan, which could be any strategic plan in any department in any country, as well as the push to publish specific research in specific journals at the cost of good science, effective teaching, collegiality, or caring for students, is indicative of what Muller says: ‘Metrics influence behavior … (in that they) divert us from our original goals’ (p. 116). In other words, we become attuned to focusing on what can be measured rather than what matters most. I cannot stress enough how important this book is for all organization studies scholars. If anything, I see it as an act of resistance to the plethora of publications that ‘count’ but are completely uninteresting, unimportant, and unread.