Possibility space: The role of social sciences in understanding,mapping and shaping the future

Abstract

Social science no longer does enough to map out the possibilities for the future, at a time when there is a serious need for more options. There are many reasons for this including the structure of incentives within universities, and the impact of an otherwise healthy focus on evidence and data. This piece describes both how social sciences can better understand the future and their role in helping to shape options, including methods for creativity, and the relationship between broad goals and experimental methods to find pathways. It addresses the problems of ‘materiality bias’, a bias towards exaggerating the influence of material over non-material factors and concludes with a discussion of how to think about future consciousness.

We are in a time of accumulating crises. The word crisis is sometimes overused but it’s hard to avoid now. The still unresolved financial crisis of the late 2000s left incomes flat for huge numbers, then fed in turn into the long political crisis that gave us Trump, Brexit and much more, including a crisis of democracy that leaves each generation less confident it’s the best way to run a society, and has overlapped with the climate crisis, apparent last summer in temperatures above 60° in India. Add to these the crises of inequality; COVID; war; energy; and hunger, and it’s not surprising that many feel that history may be going backwards with the return of strong men leaders, tanks and brutal civilian deaths, worries about hunger displacing worries about obesity.

A time of mounting crises should be when social sciences are in their element – diagnosing, prescribing, proposing, making sense of both what is and what could be. This is a time when the idea of possibility studies should be more relevant than ever, exploring how we understand, shape and act on the possibilities ahead of us. ¹

Yet our systems and structures often discourage this work and that as a result we have too little design, too few good options, not enough in the way of roadmaps and routes to the future.

I have interviewed hundreds of activists, and leaders of all kinds and found that most find it easy to picture ecological apocalypse or climate catastrophe, and easy to picture a technological future – a world dominated by robots or drones. But they struggle to get beyond the fuzziest blur in describing what society might be like, our social future as opposed to our technological future.

This matters. Any society needs options, a menu of possibilities from which to draw, particularly when facing mounting crises, and every society needs some sense of the road ahead, a map of the future that doesn’t just take us to ecological ruin or to being enslaved by robots.

There’s an important psychological reason why without some picture of a better future it is quite hard to thrive in the present. The psychiatrist Viktor Frankl showed in his classic study of concentration camp survivors that whether prisoners survived depended on whether they had hope for the future. If they lost it their mental and physical health tended to decline.

Yet polls now show large majorities expecting their children to be worse off than them: their sense of possibility has shrunk. Fascinating recent research surveyed the patterns of sentiment in all books published in English, German and Spanish over the last 150 years (as gathered on Google) which showed symptoms of a collective depression, on a scale greater than during the world wars, in recent decades. The authors wrote of an upsurge of ‘cognitive distortions’ since around 2000, leading them to comment that ‘large populations are increasingly stressed by pervasive cultural, economic, and social changes’ linked to ‘the rising prevalence of depression and anxiety in recent decades’ and they show that what they call ‘catastrophising’ ways of thinking have risen sharply as utopias have been displaced by distopias in our collective mind.

This shrinking of the future matters politically too – because it fuels what the German sociologist Andreas Reckwitz has called the switch from positive politics, which emphasises the openness and possibility of the future, to a negative politics which is defensive, sceptical and nostalgic, convinced that the best years lie in the past.

Now it’s possible that we are all, indeed, objectively, doomed. But it seems to me implausible that we could know this with any certainty and one lesson of history is that it’s rarely possible to judge prospects accurately.

So it’s surely better to try to think our way out of our many crises – and if we don’t even try then we certainly will be doomed and will deserve to be.

So how might we collectively do better, how we might use the incredible knowledge of social science now to help us see and shape better than we are doing now, and to fill up the fuzzy pictures in our minds with sharper pictures of what’s possible and better? How might possibility studies help us?

Possibility space

I’ve borrowed the phrase ‘possibility space’ from maths and statistics where it refers to all the possible results or outcomes from an experiment (in maths, e.g. all the possible results from rolling a dice). I use the term in a similar way to describe the options open to an organisation or a society at any point in history.

The size and shape of this space can never be defined precisely. Some things are more likely (most big nations will be roughly the same shape in 20 years), some less probable ones (like the various dreams of creating Greater Russia, Greater Turkey, Greater Hungary being realised) and some that are to all intents and purposes impossible (that a Mongolian empire will reappear or Donald Trump will become a Buddhist monk).

The size this possibility space depends on many things – on systems, cultures, resources and institutions. But it also depends on imagination and action which transforms the unlikely into the possible, and it’s a matter of choice for any society whether it seeks to amplify this space or allow it to shrink. So, for example the preparatory work over many decades on a possible universal health service expanded the UK’s possibility space and made it much easier to create the NHS once the political conditions were right in the 1940s. Decades of work on how a circular economy might work has expanded our possibility space and made it at least possible that we might retreat from fast fashion or the mountains of e-waste that are such an ugly side of contemporary consumer capitalism.

And of course there are long traditions in social science of trying to amplify it. Indeed, for many of the greatest social scientists, diagnosis and prescription were tightly interwoven, from Adam Smith, Condorcet and Comte to Jeremy Bentham and Marx whose grave is inscribed with his famous comment that ‘the philosophers have only interpreted the world, in various ways. The point, however, is to change it’.

From their origins, in other words, the social sciences were concerned with how to shape the future and not just how to analyse the past and present.

But this tradition has partly disappeared. A century ago HG Wells wrote that ‘sociology is the description of the Ideal Society and its relation to existing societies’ but few take that view now.

There are many exceptions – including global figures like Amartya Sen or Eleanor Ostrom – who worked on theory as well as design, echoing the story Jon Agar tells in his history of science in the 20th century that even the most apparently theoretical breakthroughs in the natural sciences often came from engaging with what he called ‘working world’ problems.

There are pockets of work on mechanism design, or societal transformation.

But overall this kind of work has become harder. Search out well informed proposals for how welfare, democracy, tax could be a generation or two from now and you’ll find surprisingly little.

Why has this happened? My suggestion is that this has partly happened as a unfortunate by-product of perfectly sound, well-intentioned shifts. Healthy pressures to attend to hard data and evidence have had the unintended consequence of squeezing out attention to the future since by definition evidence and data refer to the past and present. A well intentioned focus on impact has encouraged incremental work on policy – how to tweak a little, ideally aligned with the interests of the government of the day but discouraged the serious design of how our society or economy might be a generation out since of course a brilliant idea that will flourish in 30 years time won’t show up in the REF.

An equally healthy commitment to rigour has made it hard or even career threatening to be creative, since any genuinely new idea risks sounding flaky, vague or half-baked (as any radical idea will be in its infancy).

Similarly, as evidence now shows very clearly, the very valid reliance on peer review as a near-universal assessment method, by its nature discourages the boldest most speculative thinking, favouring safe proposals over more radical ones that tend to get a mix of very high and very low scores. So many of the brightest opt either for analytical work or for the safer space of commentary and critique – often brilliantly – but steer clear of the riskier space of saying what they think should be done.

And although within every university there are pockets of bold thinking, some very creative and dynamic, and although many want to play a part in the great transitions that may be needed in the next few years, they are almost without exception on the margins of their fields, happening despite, not because of, the incentives of the system.

Understanding futures

So how might we explore and expand our society’s possibility space making the most of the social sciences? There are long traditions of forecasting, scenarios, foresight, sci-fi, counterfactual history. Utopias helped past societies think ahead. So did model towns and pioneer nations. Some forecasts are fascinating, and some comically wrong. Philip Tetlock in his work on expert political judgement showed that in forecasting geopolitical events involving Russia, not only non experts but even Norwegian rats could perform better than experts.

Social scientists learn early that social change is far more complex than our capacity to understand it. Everyone sees the world ‘through a glass darkly’ and real societies are messy and complex with few linear connections between actions and results. Indeed, this is the paradox that faces anyone who studies history. There are steady long-term patterns of change: developments in the economy, demographics, technology and even consciousness, but it is impossible to know in any detail what the future will actually bring. Few in 1900 expected a brutal world war and revolutions in the next generation. Few in 1925 anticipated a boom, a depression and then another war. Few in the 1950s expected the scale of cultural change of the 1960s. Few in the 1980s expected the imminent collapse of the USSR, resurgent Islamic fundamentalism, or the rise of personal computing and the internet. Few in the 2000s anticipated the scale of the financial crisis, or the boom in populist authoritarianism, or that the world would grind to a halt thanks to a pandemic. Few in 2021 predicted a brutal war in 2022 or a glorious time for oil companies.

Nor does the vast explosion of data help us very much, again in part because that data will always be from the past and in part because the world is not probabilistic or very predictable. Models can give insights but in an uncertain world they risk deceiving us into thinking we understand more than we do.

One option is to just stop bothering. But one main justification for exploration is that it can help sharpen our models of the world. Regular exercises to predict – and then reflect on what actually happened – is wonderful for cultivating a little humility and learning about how the world works (I speak as someone who twice forecast revolution in Saudi Arabia, many recessions which didn’t materialise, and who bet that Boris Johnson would be out by the end of 2021). Clearly something was a bit awry with my models of the world and hopefully I learned something from my errors.

It also protects us against confirmation bias – the tendency, even more marked amongst the intelligent and highly educated, to select the information that confirms our views.

I know more than a few brilliant thinkers, with complex understandings of how the world works, who can fit almost any new information or event into their frameworks, however much it appears to challenge their assumptions.

Foresight, scenarios, fictions and games also help us to avoid under-estimating plasticity of the world. Karl Marx was right to talk about reification – how social relationships take on the appearance of solid things and serious engagement with the future – with possibility space – can protect against this.

The great novelist Ursula Le Guin – in retrospect one of the most important social thinkers of the last century – justified science fiction in these terms, describing it as training for the imagination, and its particularly useful for apparently hardnosed realists who every now and then become incredibly unrealistic (as when the best informed experts almost without exception told us in February that Putin wouldn’t invade Ukraine, and then, like his media, quickly shifted from saying war was impossible to saying it was inevitable).

Thinking about which trends will continue, which will bend, invert, break sharpens our thinking and focuses us on the pace of change in different fields – on the one hand the slow but remorseless pace of demography or infrastructures that may take 50 years to change; on the other the feverish pace of social media that gets a billion people onto Tiktok in a couple of years. All of us usually overestimate how much changes fast and underestimate how much can change over longer periods, yet we still lack a plausible social science of time.

Shaping the world

But what I want to focus on is slightly different – not so much the analysis and interpretation of the future but rather its design. In sciences – whether life science or computer science – it’s taken for granted that if you are ambitious you speculate and design options for the future. You are encouraged by research councils, university departments and venture capital to generate ideas, the more radical the better. A future orientation is seen as necessary and admirable. And it’s recognised that although most ideas will fail, the rare successes will be useful and some incredibly valuable.

In science and technology there is no shortage of support for imagination – thinktanks, conferences, accelerators, funds – on smart cities, smart homes, AI, genomics: much of it speculative and much of it hype and hot air but some of it very real and feeding into well-developed systems. Drug discovery and new surgical procedures on the one hand and much of technology have long-established systems for generating options, selecting the best and then scaling them up.

By contrast social action and policy lacks any comparable systems of support. There is little support for radical thought and variation – little support for experiment and testing and only patchy systems for then selecting and scaling the successes.

There are some interesting attempts in Canada and Australia to grow a more systematic approach to social R&D but they remain tiny compared with R&D for hardware, and while we have thousands of labs for new materials, drugs, we have very few for new ways of living, learning or cooperating.

The Internet is a striking example of this imbalance. It’s now 30 years since it became part of daily life, a period which has seen vast spending on exploration of technologies of search, targeting, manipulation. But during that same time there was remarkably little serious attention to the downsides even when in the 2000s it became clear that more social connectedness was correlating with apparent weakening of mutual support. Only in the last few years, probably since the election of Donald Trump, have serious research programmes addressed misinformation, democratic corrosion, manipulation of children, while serious policy work to fix these problems has been even thinner on the ground. Shoshanna Zuboff’s work on surveillance capitalism is a classic example. Hundreds of often fascinating pages of critique but almost literally nothing to suggest what might be done.

This lack of attention to generating options in the social sciences has had consequences. It’s contributed to the unbalanced development of so many technologies, and its impeded our ability to make the big changes we need, and will make it harder to handle the crises around us.

Milton Friedman – not everyones favourite economist – made the case very well. He once wrote that ‘only a crisis—actual or perceived—produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That I believe, is our basic function: to develop alternatives to existing policies, to keep them alive and available until the politically impossible becomes the politically inevitable’.

Whether or not you agree with a single other word that Friedman wrote, he was surely right about this, and it highlights just how important it is to have those alternatives worked out, thrashed through, informed by the best available knowledge.

Yet often there just aren’t enough alternatives around. You can see the problem if you work with governments. I have worked in the core of quite a few – from Downing Street to Canberra and Washington.

You soon learn how much they struggle to make sense of the world around them, or how often apparently powerful leaders feel powerless.

But you also learn that they lack options – and again and again I’ve had the same experience when working with a government diving into a new field, an expectation of a wide menu of options to act on a particular policy problem, and the slow dawning realisation that there are very few, and those there are, are mostly vague and sketchy.

I recently heard Adam Price, leader of Plaid Cymru, call this the 10th chapter problem: he had read lots of books which offered brilliant diagnoses of what was wrong with our economy or society in their first nine chapters, but then in a 10th chapter offered answers and prescriptions that were bland and unconvincing.

Now there are good reasons for wanting social science to be empirical, attuned to data and evidence rather than to speculating about the future or designing in the hope that someone might come along in the future and want it. I spend much of my time helping to run IPPO – the International Public Policy Observatory – which synthesises evidence from around the world on issues like homelessness or care to help decision makers make the most of what is already known.

I would most of the time rather that the people with power over me – whether surgeons in a hospital or ministers in a government – were using tried and tested methods rather than using me as a guinea pig.

But there is no reason why this should squeeze out the space for imagination – we should be synthesising the best available knowledge and creating new knowledge not seeing these as alternatives.

A prescription

So what could be done? I believe we need to grow the spaces and places within the social sciences that expand our shared possibility space, the options for our societies. This is what I call exploratory social science – generating not finalised blueprints but rather generative ideas and prototypes that can be adapted, expanded and experimented with. We need this work both within disciplines and across them. We need it to draw on deep knowledge but also to avoid being trapped by past orthodoxies. We can take inspiration from other fields.

One is art. The film ‘The Mystery of Picasso’ is a good prompt, a film showing the painter at work through a glass screen, painting a picture, then obliterating it, shifting it, mutating it – and you slowly realise that his creative process was endlessly fertile, trying things out and then keeping the best ones. Biology suggests that evolution does much the same – endless differentiation and variation and then selection.

Yet these processes are much rarer in social science and rarer still in policy which both prefer a much less creative, linear process from analysis to proposition, and from my own research very little use is made in social science of established methods from design, arts and business to amplify creativity.

But there are many methods that can be drawn on to expand a possibility space. Applying a few simple rules can help anyone or any group to generate options, for example to transform an existing activity like childcare, pensions, libraries, tax so as to multiply options.

First you think about extension taking an aspect of existing practice and going further, like Bach’s extension of fugues to six voices or extending the idea of rights to new fields, extending school hours, extending suffrage by giving the vote to 16-year-olds, or 6-year-olds.

Then you try grafting (or combining) taking an idea from another field and applying it to another. Again, this is very common in the arts – for example, grafting ideas from photography back into painting – and other examples include the way that the idea of auctions was grafted onto the management of the electromagnetic spectrum (the radio waves used for mobile phones, satellites or television), or how the idea of the jury was grafted onto democracy in the form of citizens’ juries.

A more radical approach is to use inversion, as practised in the Middle Ages during Carnival, when for a day the poor pretended to be rich and vice versa. What if farmers became bankers (as happened with the microcredit provided by Grameen Bank); patients became doctors; or social care were provided by people who had themselves been recipients of care? What if consumers became makers of things?

Addition and subtraction are also useful. Baroque and traditional Hindu architecture are good examples of extreme addition, and any social service can easily add on new elements – like a family doctor who also offers advice on welfare. Much modernist art and music favoured subtraction, leading to Malevich’s painting ‘White on White’ in 1918 or the silence of John Cage’s 1952 composition 4′33″. This way of thinking can also be generative in social contexts: what if you took away half of the roles in a hierarchy or introduced a maximum income? Or what if you had to cut a budget by half? Noone likes cuts but I’ve worked with public parks that faced a 50% budget cut and were prompted to come up with dozens of creative ways of raising money, through events, music, festivals and food, leaving the parks more vibrant than they had been before. The cheap prices of today’s supermarkets are only possible because Clarence Saunders in Memphis in the early 20th century had the inspired idea of subtracting service staff and letting customers pack their own bags.

Sometimes, not doing things is better than doing them. A surprising example of this was found in the military’s experience that taking immediate action to treat soldiers suffering from PTSD tended to make it worse. It proved better to let people mobilise their own resources and then to focus on the 4% or 5% for whom that approach hadn’t worked. Less can be more. And of course veganism is an approach which subtracts – excluding meats and dairy products from diets – while much law and regulation is now focused on reducing energy use, carbon emissions and travel, rather than increasing them.

If these are some basic methods creative thought can also be helped by mobilising metaphor and analogy – seeing one thing and thinking of another (a variant of the grafting process described above). Much of social change comes from shifts in metaphors. Do we see society as a war, a body or an organism; a building, a machine or a family? Is the economy analogous to a household, which means being very careful not to spend more than you earn, or is it more like an entrepot or trading post, in which case debt may be essential?

There are many other methods that can be used, from the sophisticated worldbuilding methods Hollywood uses to formal models, and we can learn much from science fiction, including five centuries of feminist utopias from Christine De Pizan in the 15th century and Margaret Cavendish in the 18th to LeGuin and Marge Piercy in the late 20th.

My argument is that we need to use more of these methods to expand possibility space – the options for our societies and we also need institutions that can encourage this.

At present the incentives of peer reviewed journals and funding, and the logics of disciplines, all strongly push against everything I’ve described.

Faced by challenges – from rising levels of isolation to food insecurity – we lack systems to multiply options, select the best and then spread them and no one quite sees it as their job to do this.

But there are counter examples. Last year Cardiff launched Spark, its Social Science Park bringing together academics, thinktanks, civil society in what could become an ideal focus for the kind of work I’m describing. At UCL we run a 2 week programme with 1,500 students working on practical problem solving, this year around air quality in Delhi, Nairobi and London, interdisciplinary, practical, linking diagnosis to prescription.

There are attempts to create museums of the future – from Rio and Dubai to Barcelona – to give a city or region a sense of what might lie ahead, how to shape care or food or media, and given that we have several thousand museums in this country its surely not unreasonable to hope that some might focus on this work, commissioning from the sociologists, historians, designers in their nearest universities.

These examples from universities confirm not only that this exploratory work can be done but also that it can be judged. Ultimately only history can tell us which imaginative ideas are brilliant and which are deranged.

But there are ways to assess a social design: is it logical, does it align with what we really do know, is it at all plausible – economically, psychologically, culturally? Indeed, the process of questioning and challenging is itself useful, the more so if a proposition is sufficiently sharply described to be pulled apart and one of my complaints with much futurology is that its too vague. As Wolfgang Pauli famously commented of an unimpressive theory, ‘It’s not only not right; it isn’t even wrong’, meaning that it wasn’t sufficiently sharply defined to be proven either true or false.

Imagination, experiments

The other common objection to exploratory social science is the justified fear of half-baked utopias being imposed on helpless populations by figures like Pol Pot or Chavez: social life is indeed far more complex than our brains or our theories can understand.

But there is a simple answer. Experimentation has long been normal in health (particularly for new drugs) and is now mainstream in many parts of business with companies like Amazon and Google doing AB testing on new services of all kinds. Many governments – like Finland and Canada – have become persuaded of the advantage of testing ideas out in practice, through labs, trials and sometimes Randomised Control Trials applied to welfare or education, and this direction was given greater impetus when the 2019 Nobel Prize for economics was awarded to Esther Duflo and Abhijit Banerjee for their work promoting experimentalism in development.

Experiments and RCTs are not a panacea and are particularly ill-suited to more systemic change. Moreover on its own the experimental method doesn’t tell you what to test. But a commitment to trying things out and learning is a very good vaccine against the crazy and destructive, and ideally we should want imagination and experiment.

There are also other protections against overreach. Democracy is quite a good one – especially if the public and prospective beneficiaries are closely involved in the process of design. The Lottery Fund’s emerging futures programme is a particularly interesting example engaging 50 communities bottom up in imagination and around the world there are many examples of large numbers of people taking part in serious exercises to think about what lies ahead and what choices might need to be made now to prepare for ageing, climate, water shortages or job destruction, a constructive alternative to amplified anger of social media.

At their best too these can be educational. So, for example, as a community thinks about its long term care needs, start with a fact like that amongst the elderly cold symptoms less likely amongst the highly sociable than the less sociable and then work outwards.

Or in thinking about equality start with the fact that 48% of the lowest paid workers in the US were deemed essential during the pandemic, a far higher proportion than amongst the highest paid, or that during the same period the numbers of billionaires doubled – at a time when wealth is largely untaxed, while real pay for the majority – which is taxed – went down.

Consciousness

I want to touch here on a common problem of futures work and an issue that is particularly important for social science. This is what I call materiality bias. Work on the future usually emphasises stuff: drones, weapons, flying cars. The TV programme of my youth – Tomorrows World – was a case in point, lots on new planes and cars but I don’t remember anything on transgender identities or climate consciousness.

We exaggerate the importance of the things we can see and touch and underestimate the things we can’t, even though its obvious in history that ideas matter as much as things, feelings as much as infrastructures, and obvious too in economics that productivity advances came as much from organisational practice – the design of the factory or office – as from electricity and computers, or that advances in health came as much from behaviours as sewers.

Yet this materiality bias is deep in our culture and it takes us to perhaps the most fascinating and difficult aspect of exploratory social science, and the exploration of possibility space – thinking about how people might think and feel in the future. Its obvious in retrospect that changes in consciousness – how we think about gender, race, ecology – have more profound effects than technological change. In the month of the Jubilee, for example, it’s worth remembering that two centuries ago most people assumed that monarchy was inevitable, a fact of nature rooted in human nature, and democracy was an interesting idea that had been tried and failed. Now the palaces are still there but the ideas that supported them are largely gone and we take it for granted that inheritance is not a good way to fill any jobs that really matter.

If there is a direction of travel in consciousness it appears to be towards a bigger awareness – of our connections to nature and biology, of the perceptions of others, or our inner selves. Many have tried to theorise and grasp these patterns, from Piaget and Gebser to Robert Keegan and Clare Danes, and it was nicely described recently by the Zapatista movement in Mexico as idea of aspiring towards ‘a world where many worlds fit’. But it’s very hard to be definitive; hard to grasp both the trends and the backlashes, the ways in which times of stress we revert to older values.

Yet if we are to have any optimism about the world has to be an optimism that humans can evolve in their ways of thinking and feeling as well as in their material conditions, and I hope we might see more serious social science in this space, exploring the dynamics of collective consciousness – and wisdom in ways that are both empirical and exploratory.

Conclusions

We are at a time of extraordinarily fertile analysis of the past – the ‘long durées’ of inequality, governance, values, families – and just as fertile analysis of the present. But we’ve made it harder for social scientists to engage with understanding or shaping the future.

If we really are in a time of multiplying crises then we badly need options, and social scientists need to be part of this work. We need the best brains to be working out how to design and run a zero carbon economy; a society with more disability; how to make ubiquitous smart technologies serve us rather than the other way round; how to counter polarisation; misinformation.

We need to populate our fuzzy pictures of the future with complex, rich, plausible deas, pictures of the possible – a possibility space that is capacious and helpful for action in the present.

There may not be an immediate demand for these, not least as governments attend to the immediate. But it is precisely at these times that we need to look ahead, just as in dark days of 1930s and WW2 some worked hard to think about what could come after, from designing welfare states to macroeconomics, decolonisation to human rights and the creation of the UN, which a decade before it was founded seemed utterly utopian.

About 2000 years ago Seneca warned that there are no fair winds for those who don’t know where they want to go. Many governments seem stuck in a bizarre eternal present of culture wars and tactical manoevres, and bereft of a compass or a map. But so too are many of us as individuals.

We need possibility studies, and the social sciences and humanities in all their forms, to be part of our collective effort to redress that and help us navigate through the difficulties of the years ahead. And if this community isn’t going to help, who will?