What is the Impact of Political Theory?

Abstract

I am very grateful for the contributions by Andrew Vincent, James Alexander and John Dunn to this symposium on ‘impact’ and political theory. Their papers provide insightful perspectives and different critical engagements with my recent piece for this journal. Their reflections force me to revisit my central argument that the impact agenda unveiled in the United Kingdom's new Research Excellence Framework (REF) need not be the negative development for political theorists that many fear – and perhaps even that it should be embraced.

Before I respond to the criticisms of Andrew Vincent, James Alexander and John Dunn, it is necessary to clarify misconceptions about the REF's understanding of impact. The Research Excellence Framework (REF) is an assessment of research quality of academic departments (as ‘units of assessment’) across three indicators. The first is the quality of research outputs, such as monographs and journal articles. The second is the quality of the research environment taking into account research activities such as external speaker seminars, postgraduate students and research funding. These indicators count for 65 and 15 per cent, respectively, of an academic subject's final score.

The newest indicator is the impact of research beyond academia, which counts for 20 per cent of the final score. So the first myth we must dispel is that the REF is mostly about impact – to be clear, it is not. Impact has attracted much attention, perhaps because of its novelty coupled with the relatively brief period between its first announcement in 2008 for use in a REF assessment that had already started. There was understandable concern about whether this measure could be assessed reliably. Nevertheless, impact is only one of several factors assessing research quality; to argue that impact is the main driver for research quality is to overlook the role it plays in the overall assessment.

A second myth is that the inclusion of impact as an indicator of research quality in the REF requires that all research-active academics demonstrate impact in their work. This is manifestly untrue. Impact is assessed through narrative case studies describing impact within a period from 1 January 2008 to 31 July 2013. Academic departments were to submit roughly one impact case study for every ten full-time academic staff included in the assessment (REF 2011; 2012). While departments had an interest in securing a number of high-quality impact case studies, there is no requirement or need to submit an impact case study for every member of staff.

A third myth is that impact is assessed only by non-academics. This is also untrue. Impact might be about the impact of research beyond academia, but its assessment is not placed entirely beyond the hands of academics. The impact case studies are scored by members of the relevant assessment panel. The group judging impact in Politics includes some of the leading political scientists – and political theorists – in Britain, including Richard Bellamy, Chris Brown, Iain McLean and Judith Squires. The Politics and International Studies REF Panel does include a few non-academics from organisations such as the House of Commons and the Institute for Public Policy Research. However, the seven panel members from non-academic institutions are well outnumbered by the 22 that are academics.

Exposing these myths is important because it clarifies misconceptions about the use of impact in the REF. Alexander discusses impact ‘as measured by various governmental and professional agencies’ (Alexander, 2015, pp. 485–486). This leads him to examine the relation of theory and practice where we should ‘recognise that the possibility of adopting a political or a philosophical ground is always open’ (Alexander, 2015, p. 492). Alexander is simply wrong to believe impact is determined by government and related organisations. Yes, government has supported its introduction, but academics have developed how it was measured and led its assessment. Perhaps Alexander thinks this was a poor decision and much can be debated about that. Nonetheless, I suspect his concern about the inability of impact to capture certain political or philosophical possibilities rests on his mistaken belief that ministers and not academics – and especially not political theorists or philosophers – will lead on assessing impact. So if his fellow theorists and philosophers are unconvinced by his arguments, Alexander cannot blame non-theorists for how impact is understood. Much of his critique rests on a mistake about how impact is assessed.

Dunn is critical of political theorists who aim ‘to affect the world for the better’ for they run a risk of ‘professional impropriety or self-deception’ (Dunn, 2015, p. 497). But this rests on his acceptance of the myth that political theorists ‘measure the value of their own work by a metric devised by public officials’ or ‘Treasury officials’ (Dunn, 2015, pp. 496 and 498). It was not politicians who designed how impact could be scored, but academics. Moreover, Dunn understands ‘impact’ purely in terms of affecting political decisions. This is a mistake: impact is about influencing non-academic users, including (but not exclusively) public policy makers. Even if his critique is correct, it is a partial victory at best.

Vincent argues that the use of impact creates ‘the “Celebrity Come Dancing” generation of academics, rushing into TV studios or other assorted external media outlets, for their REF impact narrative’ (Vincent, 2015, p. 481). This mistakes media engagement with impact: having an active, visible media presence is good public engagement, but does not constitute ‘impact’ for REF purposes unless there is some identifiable change in relation to non-academic users of 2^* or better graded research. If impact should be rejected because it is thought to be nothing more than speaking to the media, then Vincent need not worry because this is not included as ‘impact’ in the REF. He cites a passage by Martha Nussbaum in a 2010 book to claim REF impact ‘means above all economic impact’ (Vincent, 2015, p. 482). This was an understandable concern five years ago, but the REF panels’ assessment of research impact should now help put such concerns to rest.

The discussion thus far is not a defence of the use of impact in the REF assessment. Instead, I seek only to highlight some common myths about what this impact is. It is important to expose certain views as mythical fictions to help us separate criticisms aimed at the target from those addressing concerns or issues that are based on mistakes. Each of the authors in this symposium focuses at least some part of their discussion on concerns that do not exist – or at least not in the form identified. My argument is not that this renders their critiques irrelevant, but rather I want now to turn our attention to some of their criticisms aimed at the use of impact and not myths about its use.

Replies to Critics

Vincent (2015) raises two issues worth considering further. The first is that not all impact is positive. I raised the concern that many may view the impact of political theory with suspicion given that so many in the canon opposed democracy and political values many of us cherish today (Brooks, 2013, pp. 209–10). The pursuit of impact is not a good in itself. However, impact for REF purposes has a specific meaning: is the potential impact linked to 2^* or better quality research? I suspect many possible counter-examples of essays that have contributed to negative consequences would not meet this test. Nonetheless, I have no doubt we can find illustrations of research that contributed to negative impacts, but recall that impact is one of several indicators of a department's overall research quality where the greatest weighting is afforded to publications. We should be concerned about the kinds of impact that research can have, but it is best judged by panels led by academics with less importance than the quality of publications.

A second issue Vincent raises is that research should not be market-led. However, it is unclear how the higher education system is market-led because of REF impact. It is too early to tell if students choose universities because of a department's impact score in the recent REF. This is also highly unlikely. Perhaps doctoral students will give extra weight to a department's REF score, but it is difficult to imagine ‘impact’ scores outweighing the importance of the department's overall score (or that many students will consider and compare the different scores departments receive on ‘impact’). This is not to disagree with him that marketisation of higher education is taking root and problematic in important respects. However, it is also worth noting context. Almost every Member of Parliament I have spoken to tells me they do not know exactly why impact was missing in earlier assessments as a means of assessing ‘value for money’. Future prophesies are hostages to fortune, but it is a fairly safe bet that ‘impact’ will remain in further REF exercises for the foreseeable future.

I have already noted how Alexander's central concern about impact rests on a mistake about how it is assessed. ¹ I now address two connected issues that relate more directly to my original article for this journal. The first issue is Alexander's worry that my ‘argument’ that ‘theorists ought to welcome the requirement their “research” should have “impact“’ is a problem because ‘if taken seriously, [it] would, I think, lead to the death of political theory in all but one possible sense’ (Alexander, 2015, p. 485). Alexander again rests his concern on a mistake. I do not argue that all political theory research ‘should’ have impact or that theorists ought to ‘imitate’ those that do. My actual argument is more modest: impact is something that ‘political theory may – and often does – possess’ (Brooks, 2013, p. 209). Not all research in this area may have impact nor should it be required to have it to be valuable. Remember that impact is only one of several factors in an overall assessment of research quality where outputs will continue to reign. It is an error to think that all of a subfield might die off because of some requirement it simply does not have. Nor do I argue anywhere that this should not be so. ²

The second, related issue is Alexander's further mischaracterisation of me and ‘the standard position of political philosophers and theorists today’ as endorsing the view that political theory can and should be ‘untainted by practice’, as theorists develop their lofty ideals for implementation in an imperfect world (Alexander, 2015, PP. 486; 489). First, Alexander appears to overlook the significant rise of non-ideal theory that is increasingly a part of any ‘standard’ discussion in the mainstream literature. And second, Alexander's characterisation of political theory may be at odds with how most might think about the subject as theories about the political, and thus related to our practices and institutions. There are then reasons to reject Alexander's characterisation.

Curiously, Alexander is critical of the view of the political theorist constructing theories separately from worldly concerns while later embracing what he calls the ‘phlegmatic’ relation of theory and practice where he ‘lets the world go to the dogs with equanimity and simply writes philosophy where he can’ (Alexander, 2015, p. 491). This suggests not that political theory must lack practical relevance, but rather that Alexander does not attribute too much importance or value to it. This is an issue worth considering, but it seems to rest on a further mistake. In another accusation towards me, he says that the ‘sanguine’ relation of theory to practice ‘supposes that theories can be framed like policies for implementation’ (Alexander, 2015, p. 491) – and among such theorists he includes myself. But this is nonsense. A theory is not a policy, as anyone who works in public policy-making can attest. I may have a theory of citizenship and political justice; these may guide or inform policies relating to them like citizenship tests and language requirements, but they do not serve as the policies themselves. My conclusion here is that Alexander's interesting distinctions and statements reveal more about his individual concerns about theorists and the possibility of their practical relevance than the actual use of impact in British higher education or how theory does, in fact, relate to practice.

Dunn rejects the pursuit of creating impact by political theorists focusing exclusively on their potential influence on politicians. He is dismissive of the ‘bewildering indiscretion of the leadership’ in the UK and elsewhere, doubting that they have coped ‘reasonably successfully over the last few decades with their responsibilities to most of their own citizens’ (Dunn, 2015, p. 495). Such sweeping statements are offered as obviously true, which I would dispute.

Dunn says that ‘a lecture room is not a church’ and that ‘[m]ost useful educational advice must either be broadly exemplary or highly contextual and very specific’ (Dunn, 2015, p. 497). A political theorist might not be a preacher, but this does lead to the conclusion that there is nothing for political theorists to contribute for revising existing public policy or developing new policy. Perhaps Dunn is correct that theorists move at a very different speed and perspective than the policy making world he finds so poorly run. But this is no argument against trying ‘to do less badly’ and it is unclear he would disagree about this point. Dunn is either mistaken or overlooks how impact is assessed, and what it measures. However, there is much else of good sense about the distinctive kind of projects that many political theorists – and especially historians of political thought – engage in that I readily accept in his well-presented and thought-provoking piece.

Conclusion

My article ‘In Defence of Political Theory: Impact and Opportunities’ was meant to contribute to a debate about whether the UK's impact agenda sounds the death knell for the subfield. I argued that we, as political theorists, should welcome the use of impact and many of us have been creating it for some time. Perhaps I should reconsider my claim that ‘the primary obstacle for political theorists is overcoming scepticism about the kind of impact theorists may offer’, which I directed at non-theorists (Brooks, 2013, p. 209). The essays in this symposium make clear that impact continues to be greeted with suspicion and some degree of alarm. I hope this reply to critics highlights that many of these suspicions focus on myths about what impact is and how it is assessed that are unsupported by the reality. My point is not that there is nothing to criticise in the REF and its use of impact (there is), but to argue a case rarely considered: that impact is not bad news for political theorists. While the essays here are useful in raising important issues, I am unpersuaded to change my conviction. At least for now.

Many of the problems raised about impact relate to myths and other misunderstandings about what it is. Political theorists have a talent for conceptual analysis and grasp the finer details about how ideas of freedom or justice take different meanings from the perspectives of different philosophers. Cicero, Hobbes and Fuller may each appeal to natural law, but each does so in different ways and generates different meanings. Likewise, the concept of REF impact refers to a specific practice that is not the same as more everyday understandings of impact. Once we become clearer about what REF impact is, I believe we find that many of the objections and concerns miss their targets and it is less objectionable than often thought. This is not to claim that REF impact is unproblematic – this is certainly not my view – but rather it does not warrant the worry many have and is not bad news for political theorists more generally.