Abstract
We live in an age where increasingly sophisticated algorithms are being designed to aid and sometimes completely replace human intervention in decision-making tasks: political targeting, autonomous driving, facial recognition, medical diagnosis; it is hard to think of a domain that has not been touched by these omniscient, omnipotent algorithms. The laudatory discourse which underpins these trends, lists the wonders of Big Data: increasing advertisement click-through rate by better targeting, detecting financial fraud in real time or sifting through lakhs of resumes to find the right pool of candidates for hiring. Algorithms seem to have done it all at a lower cost and improved efficiency than human effort.
Cathy O’Neil’s book is a much-needed attempt to highlight the perils of Big Data and how it works to intensify the rich-poor divide. Cathy, a mathematics PhD and a former Wall Street quant, substantiates her arguments by first-hand evidence from her experience as an academic and an ex-D. E. Shaw employee
The core theme of the book is to dispel a widely held misconception of mathematical models and their results being fair, objective and unbiased. The book is divided into 10 chapters: the first two deal with demystifying the jargon associated with the field of Big Data and the author’s journey from being a math geek since her childhood to an insider who saw the ‘dark’ side of equations and models. The remaining chapters focus on how these models are covertly working to exploit the hidden insecurities in human psychology in the fields of education, advertising, job markets, credit and insurance assessments.
The author has christened these mathematical models and algorithms “weapons of math destruction” (WMD) because of their attributes: opacity, scale and damage. WMDs work like black-boxes which take poor proxies to abstract human behaviour and churn out results, without having to explain how they arrived at them. The author, with her gift of analogy, has likened their working to religion: their diktats are unquestionable and their workings only known to the enlightened few, mathematicians and statisticians in this case. These models are being adapted from one field to another and being applied to the masses, who often cannot afford personal human interaction and are left to the mercy of an algorithm while applying for loans or jobs.
The author argues that the use of poor proxies to measure and abstract reality can often be discriminative in nature. Case in point: zip code is often taken as an attribute to access the creditworthiness of an individual. Including such a factor in a credit risk model ends up denying credit to individuals of a particular ethnicity or race, who live in clusters in our segregated cities. So building a model that dictates loan approvals codifies the prejudice of the society and keeps working to sustain the status quo, which is laden with inequality.
In the case of a human decision maker, there is a feedback loop which allows for correction of errors in judgement. According to the author, this critical feedback loop is missing in the case of WMDs as they seek to create their own version of reality which in turn lends further evidence in favour of their judgements. The author has given the example of the “credit score,” a crude proxy for a responsible, dependable citizen, being used indiscriminately by firms while making hiring decisions. Without considering the bias and poor approximations of a credit-scoring system, firms use WMDs to screen out job applicants with a low credit score, creating a vicious circle of unemployment and poverty.
In the context of the legal sphere, the author brings out a distinct contrast between the objectives of a legal system and a WMD. The legal system works towards fairness: it seeks to protect the innocent which may come at the cost of a criminal walking free. But WMDs subscribe to the cult of efficiency: the model is regarded as good as long as it is detecting sufficient number of actual criminals. To make matters worse, WMDs in education, policing, insurance and marketing have synergies which are exploited by rampant data misuse to target the poor from all directions.
Even though the author has given numerous examples of the socially undesirable outcomes of WMDs, ranging from university ranking systems to recidivism tests, her objective is not to argue for a complete ban on the use of mathematical models and algorithms. She believes that these models are here to stay but they need to used with caution; appropriate regulation is required to ensure that humans are not treated simply as collateral damage for the sake of efficiency. Acknowledging that quantifiable arguments are regarded as more convincing, she has suggested algorithmic auditing where humans from diverse backgrounds are modelled as users and observations made on how these models treat them differently.
This book with its conversational writing is an enlightening read for students, policy makers and practitioners who want to understand the repercussions of treating Big Data as a panacea for all organizational and social problems.