From abstract to ideal–The limits of models. A reply to Pawson’s ‘boxed in by models’

Abstract

Pawson’s article raises the important question of what constitutes good and bad modelling during a pandemic. His article makes the case for more involvement of social scientists to capture the complex adaptive nature of governmental policy. While articulating a welcome critique of epidemiological models, his article fails to recognise that all model use simplifications which make some models better than others. I will suggest a useful way of differentiating between good and bad, useful and less useful, models based on the difference between idealisation and abstraction, concepts I borrow from Onora O’Neill and political theory. They allow us to apply a more nuanced criticality to the current models used by the government. Refining our critique of the government’s COVID response is important since we need to account for the fact that current government responses to the pandemic, while open to criticism, have had some effect in reducing infection rates.

Pawson’s critique

Ray Pawson (2020) has recently put forward a fascinating critique of the modelling practice employed by the UK Government during the pandemic. His ire has been focused on two important issues. First, Pawson argues that advisory bodies such as SAGE have insufficient representation from social scientists and evaluation experts, leaving decision makers poorly informed about the social dimensions of viral transmission. Second, Pawson maintains that the government’s response to the pandemic largely resembles a policy response without the evaluative follow up. The absence of evaluating its impact leaves decision makers exposed to the dangers of repeating poor policy choices.

That the pandemic response by the government has left much to be desired is widely recognised. There has also been sustained criticism of SAGE’s modelling or insufficient transparency around it. Pawson’s critique thus appears to resonate with the unease articulated by some in the scientific community and wider public.

A key criticism appears to centre on the insufficient depth and scope of the virological and epidemiological models and their inability to predict disease progression in different communities. The argument is that these models are insufficiently flexible and fail to incorporate differential human behaviours that should inform the public policy responses to emergencies such as COVID-19. Fixed estimate models, such as those used by epidemiologists, may be based on simplifications that will prove at best inflexible or at worst inaccurate, since human responses to implemented policies, such as restrictions of mobility, are characterised by feedback loops and emergent behavioural patterns characteristic of complex adaptive systems (Cairney, 2012; Kaehne, 2016; Paley and Eva, 2011). Intersectionality of social, economic and behavioural factors influencing virus transmissions may be at the heart of this issue (Chin et al., 2020). Indeed, SAGE recognised as much. As early as 28 January 2020 its papers speak of

‘the importance of behavioural science informing policy’ and indicate that ‘SAGE will keep under review whether further sub-groups, such as a behavioural science sub-group, are needed.’ It states its intention ‘to open lines of communication with SAGE behavioural scientists’ (Van Tam et al., 2020)

And in its papers from the 22 October 2020 meeting, citing available evidence, SAGE notes,

‘the COVID19 pandemic is strongly shaped by structural inequalities that drive household and occupational risks, emphasising the need to tailor effective control and recovery measures for these disadvantaged communities proportionate to their greater needs and vulnerabilities.’ (Her Majesty Government (HMG) et al., 2020)

This may not have been translated into actual modelling practice, however, since the minutes from SAGE meeting on 15 January 2021 still show no sign of going beyond the simple epidemiological models (HMG et al., 2021).

The debates about who should or should not sit at the table is still playing out. So where shall we draw the dividing line between these general epidemiological models of impact and detailed descriptions of reality? Is there no perspective that can act as a bridge between the epidemiologist and the social scientist in this matter?

To carve out some shared space for everyone, I will argue that modelling always relies on simplifications based on abstraction; the problem arises where abstraction veers into idealisation, a distinction I borrow from Onora O’Neill who advances this to good effect in the context of political theory and justice.

Modelling as simplification

Speaking of something means to simplify it. Removing complexity from something that exists in reality is part and parcel of using language. The problems in epidemiological modelling Pawson rails against are of course an extension of the desire to reconcile universals with individuals. Words are reducing nuances that exist in the real world. Whenever we speak we make a trade off between infinite subtleties and crude generalisations for the purpose of making ourselves understood. The tension between the intricacies of reality and words is well known and widely recognised. It underpins the perennial debates about individuals and universals in philosophy. It rumbles through the debates between logical positivists and their empiricist colleagues.

As much as words may be a straight jacket for existing variations and nuances, they also enable us to communicate purposefully with others, breaking down linguistic and interpretative barriers. Negotiating the different meanings, interpretations and simplifications of language stands at the heart of understanding, the cognitive work we do.

Science, as a linguistic practice characterised by standardisation and an ambition for exactitude, is no exception. Previous attempts to reduce science to statements that are patently true or contain only logically falsifiable pronouncements may tempt us to engineer a new, less problematic, language but there is little chance this would become a norm for scientific discourse or indeed any public discourse. That leaves us with the flawed instrument we have and the desire to use concepts that we define with a high degree of precision.

The intersubjective surplus of meaning of the words when used in speech however continued to trouble scientists. For Weber, it was the subject matter for sociologists. In Wirtschaft und Gesellschaft Weber (1980) relates the linguistic labour to the purpose of the sociological discipline:

Sociology creates its concepts and searches for their rules in respect of whether or not [these concepts] can help us with the historical and causal attribution of the occurrence. (p. 9, translations my own)

Weber’s willingness to recognise that concepts have human and scientific intent written into them separates him from others such as Popper who inquired whether scientific concepts could be defined by the interplay between individuals and universals. For him, science is a linguistic activity based on making universals fit particular cases. In a central passage in Logik der Forschung [Logic of Science] he notes the immediate difficulties when we are trying to define universals through individuals:

The attempt to characterise an individual by attributing to it universal qualities and relationships [to universal concepts] cannot succeed. This would not mark out a specific individual but only a specific universal class of individuals with which these characterisations match. (translations my own). (Popper, 1994: 37)

Popper concludes that defining universals through individual characterisations is doomed to fail. To put it into the language currently en vogue in population health management: someone’s population segment is someone else’s sub-segment.

The trials and tribulations of the theory of science and the birth pangs of sociology are of course well known and I am only rehearsing both accounts here because they are instructive for our issue of modelling. All models operate with concepts that are purposeful as well as retain an element of universality. Weber coined the term ‘ideal types’ for the practice of formulating concepts that fit a class of occurrences or individuals. The fact that he was comfortable with their purposive residue reflected his conviction that all scientific work is a product of self-directed behaviour, rational or otherwise. On the other hand, all models approximate universals in Popper’s sense. This duplicitous nature of models makes them so tricky to use in real life situations. We want models to describe reality, but it is a reality we perceive as purposeful agents. The contours of our scientific work are coloured by our intentions. Models have these intentions written into their DNA. Since we differ in what we consider relevant we also differ on the utility of the models we create. Weber and Popper would have well recognised the arguments about the correct SAGE models. It is a dispute that is at the heart of science as a human endeavour.

But is this all we can say about models? Pawson argues that models would be better if taking into account the complexity of reality. Involving social scientists, he argues, makes for better modelling; leaving it to epidemiologist and virologists makes for less useful models in the face of the pandemic. But Pawson tells us little about why some models are better than others. It may be the limited applicability of some models to reality, their ‘fit’ to local communities and to the behaviour they exhibit. But views on what constitutes the right level of ‘local’ come thick and fast. The individual community we use as benchmark for judging the fit of a model may be just another sub-class of cases.

So how can some models be better than others? Do we have a criterion we can apply to discriminate between good and bad models? I think we do and to illustrate how to do this I will employ an argument from political theory. This may be unusual, yet I think political philosophers produced some unique insights into the limits of models. The problem they faced closely resembled our troubles with scientific modelling in a pandemic.

Abstraction versus idealisation

Liberal political theory wishes to construct social or political institutions through the lens of fairness and justice. The nub of the problem has been the just allocation of rights. What gets into the way of fair dissemination of rights is our bounded rationality, our inability to transcend our own place in time and space (Canovan, 1998). The dangers of personal preferences determining the allocation of rights is not an academic one. We know that belonging to a particular group may incline you to privilege that very group when it comes to allocate rights (Sandel, 1984, 1998). Blanket allocation of the same rights to all does not solve the problem either since starting points of individuals are different in life. Granting everyone the same rights results in significant injustices playing out in real life. Note the parallel here to the epidemiological population approach.

Theorists formulated different answers to the question of how to eliminate our propensity to prefer our own group over others when allocating rights. Hobbes’s notion of binding everyone to the same sovereign is nothing else but an attempt to create a level playing field. Rawls’ (1971) veil of ignorance is similarly based on the idea that differences in real life structure our ability to define terms of fairness when building social institutions. So, what is their answer to this challenge? Are all attempts to design fair institutions out of bounded self-interested rationality doomed to fail?

Onora O’Neill points to a discriminating factor between good and bad models that may help us in this matter. She frames the difference between good and bad models within the debate between universalists and particularists. Both are interested in determining the acceptable boundaries of instrumental decision making. Contrasting the desire of universalists to justify ethical claims with those who recognise particular claims, she writes,

‘When universalists seek to justify ethical claims by relying on abstract and minimal starting points they see their strategy as innocuous and uncontroversial. Their hope is that by pruning away assumptions whose truth cannot be ascertained, and relying on a meagre and parsimonious set of plausible assumptions, they will lend credibility to their conclusions . . .. Their particularist critics think the hope forlorn, and that an abstract conception of the human subject or of action can lead only to an abstract view of ethical principles and to an impoverished ethical vision . . .’ (O’Neill, 1996: 39–40)

The language of ethical principles may sound foreign in the context of public health modelling but O’Neill goes on to suggest a compromise that should be instructive. She continues,

‘The objections many particularists raise in condemning abstraction is not strictly to premises which abstract from known truths, but to the ungrounded introduction of idealised premises. (O’Neill, 1996: 40, italised in original)

Abstraction, . . . is a matter of bracketing but not denying predicates that are true of the matter under discussion. Abstraction in this strict sense is theoretically and practically unavoidable, and often ethically important. All uses of language must be more or less abstract; so must all reasoning. . . . Reasoning that abstracts from a predicate makes claims that do not depend on that predicate holding, or on its not holding. . . The important merit of abstraction is that it . . . will not lead one validly from a truth to a falsehood. Idealisation is another matter: it can easily lead to falsehood. . . . a theory idealises when it ascribes predicates . . . that are false of the case in hand, and so denies predicates that are true of that case.’ (O’Neill, 1996: 40–41)

She acknowledges in an aside about rational choice models of human behaviour that idealisations may be useful, for example when we learn something about consumers in idealised scenarios. However, she maintains,

‘Practical reasoning that assumes that ‘ideal’ predicates are satisfied will not reach conclusions safely and soundly for cases where they are not satisfied.’ (O’Neill, 1996: 41)

These prescient words should help us discriminate between poor and good modelling. All modelling is of course instrumental rationality at work. We are trying to understand actual reality by constructing categories of human conduct, based on predicates that we believe to be true. Seen in this light, Pawson’s criticism appears to point to the inability of abstract models to capture the relevant attributes of communities and their differential response to the policies. What goes into the models defines the model’s utility to draw conclusions on whether or not governmental policy works. O’Neill tells us to watch out for unsound idealisations, whilst asking us to recognise that all models are abstract to a certain extent.

But what could constitute illegitimate idealisations in the case of the current governmental policy? This takes us back to Weber and his key insight that all concepts are purposive creations. For Weber, the point of idealising a quality or attribute of something is an expression and distillation of what matters to someone. On the flip side, being aware of this tendentious nature of conceptualisations in modelling enables us to create critical distance to our own intentions, to understand them and, ultimately, to build better models that are in tune with equitable notions of justice and fairness.

If we apply this to the modelling used by SAGE as the basis for the government’s policy response we understand that the epidemiological foundation of modelling may have a double nature. First, the models may be self-interested articulations of virological scenarios of COVID spread in a population. The aspect of idealisation pertains to the conclusions drawn, the rapid and equal spread amongst all members of that population. This was the false conclusion Pawson rightly criticised. We now know that this simple epidemiological model neglected local conditions by virtue of generalised assumptions about specific living conditions of some communities.

A sociologist of science may say that this idealisation served virologists and epidemiologists well in the struggle for discursive hegemony of the airwaves. It defined the government’s policy response of complete lockdown. Since the response was predicated on any type of human contact, the model also, second, reflected the assumption that the virus was highly contagious, so restricting human contact would assist in reducing its spread. It was a simplistic model that may have had its justification in the initial period. As it transpired that different communities were differentially reacting to the policy, the epidemiological model proved fatally flawed. Its illegitimate idealisation that all communities are the same became the deadly defect that subsequently led to unsound policies in the Summer.

O’Neill’s distinction between abstraction and idealisation widens the horizon of the debate that Pawson framed as a struggle between statistical estimates generated by epidemiologists and social scientists understanding the viral reality vis-à-vis the government’s response. Employing the concepts of abstraction and idealisation helps us acknowledge the utility of the initial response in the face of a population wide threat, whilst also recognising its limits as assumptions made by SAGE about the reality of community responses in England reflected a denial of difference. Historians will need to find out whether or not this denial may be embedded in our scientific institutions, or simply a reflection of weak community infrastructure as a legacy of hollowing out local autonomy over decades. O’Neill spoke of the innocuous nature of abstractions contrasting them with idealised reasoning that leads to falsehoods. For some communities in the United Kingdom, the dividing line between abstraction and falsehood meant the difference between life or death. We owe it to them to reflect on the ends of our policies and what they mean to us all.