Sage Journals: Discover world-class research

Abstract

Introduction

This opinion piece was inspired by a conversation with a colleague about education policy ‘gone wrong’. Our brief exchange was as frustrating as it was illuminating, and we resolved that policymakers should be held more accountable for the decisions they take, and that regardless of its relative success or failure, policy formation should be routinely appraised and its evolution documented. Policy archaeology offers a means by which these decisions, and their resulting aftereffects, can be further examined.

The article takes as a starting point the depiction of policy not as an object, product or outcome, but as a process, and something that is ongoing, interactional and unstable (Ball, 2021). This policy volatility is neatly encapsulated in the Welsh approach to accountability, with schools and their teachers subjected to myriad mechanisms designed to measure performance. Using this as an example, the article makes the case for a deeper and more forensic interrogation of policy design and development; it urges policy archaeologists to scratch beneath the surface of policy and ask, with the benefit of hindsight, what happened and what might have been done differently to generate more favourable outcomes. A series of key questions is offered as a potential scaffold for policy excavation.

The swinging policy pendulum

All policies have histories (Ball, 2021) and there are always, therefore, stories to be told. In Wales, the tendency of guiding coalitions to change policy indiscriminately and with little or no warning has given rise to what the Organisation for Economic Co-operation and Development (OECD) call ‘reform fatigue’, a diagnosis that brings with it the risk of only partial implementation or diminishing support from within the education workforce (OECD, 2014: 34). That education policy is so susceptible to change is no major revelation given its intrinsic link to the political establishment, and the want of ministers to leave their own stamp on the system they are employed to serve. More striking in the Welsh context is that policy divergence has taken place during a time of unstinting political stability. Since the devolution of legislative powers from the UK Parliament to Senedd Cymru (as it is now known) in 1999, Welsh Labour has been the dominant party, leading on its own or in coalition in each of the six parliamentary terms (Larner et al., 2022).

On the subject of school accountability, the landscape has been far more unsettled, with associated policy at best confused and, at times, even contradictory. Wales has lurched from laissez-faire to regulation, and back again, with the collective voice of teachers all too often ignored. Indeed, one wonders what effect the swinging policy pendulum has had on the profession's confidence to plan for the longer term; such uncertainty does not lend itself to more systemic growth.

With no obvious sign of electoral rejection (Welsh Labour increased its share of the vote at the last Senedd Cymru election in 2021), Welsh ministers appear safe in the knowledge that they can pull apart the policy of their predecessors without any real threat of backlash at the ballot boxes. Yet that does not mean those who make the decisions should not be held accountable for them and at the very least, key enablers of policy have a right to ask whether what they are required to do was determined by evidence or ideology; in short, teachers are entitled to better understand the grounds on which profound ministerial choices were made. It is against this backdrop of political permanence and reform fatigue that I propose an exercise in policy archaeology.

Policy archaeology – understanding the ‘rules of formation’

According to Scheurich (1994), policy archaeology is concerned with the social construction of education problems, rather than focusing on what happens after these problems have become visible. This is an important distinction, and one that shifts the attention of policy analysts from what follows policy introduction, to what preceded launch of the policy itself. In his paper on social policy analyses, Gale (2001: 388) describes this process as ‘the analysis of constitutive rules’ and, reflecting on Foucauldian theory, highlights the interest of policy archaeologists in the ‘licensing of policymakers and their relations’ in the act of policy formation. Specifically, he asks why some items are on the policy agenda, and others not; why some policy actors take a more prominent role than others in the production of policy; and what conditions regulate the patterns of interaction of those involved.

In the field of archaeology, excavation refers to the process by which archaeological remains are exposed, explored and recorded. Though what I am advocating is about more than just policy examination and appraisal; it is about digging up and learning from old policy to inform new; about contemplating the past to better prepare for the future. I adopt the term policy archaeology to frame this process of policy review, and help conceptualise how historic policy can be unearthed, dusted off and pieced back together so as to track its relative efficacy.

My understanding of policy archaeology draws heavily from post-structuralist theory and the work of Foucault, who resolves that individuals are bound by the ‘rules of formation’ that determine the objects, modes of statement, concepts and thematic choices within a particular discourse (Foucault, 2002: 42). The implication here is that we can never fully separate ourselves from discourse and discursive formations and, as Fadyl and Nicholls (2013: 25) explain, ‘a person's account of themselves and their experiences cannot be seen as a point of origin for the construction of meaning’. Instead, they argue, the subject is constituted through discourse, and discourse provides the basis for articulation and action. If we take this as square one for policy archaeologists, we can begin to make sense of the policy environments in which politicians, as the flagbearers of policy, took their decisions and how those conditions influenced their decision-making.

Framing the excavation

Based on what is in effect a historical account of education policy, policy archaeology requires a retracing of one's steps and the digging up – or exhuming, to use more applicative parlance – of old values, interactions and negotiations leading to policy formation. According to Kincheloe (1991: 234), these accounts are built on a desire ‘to trace the processes of educational change and to expose the possible relationships between the socio-educational present and the socio-educational past’. But to do so objectively requires a policy archaeologist that is far enough removed from the decision-making process that they can chart educational change without fear or favour (so as to avoid the subjective narration of one's own history).

I am drawn to the notion of policy framing (Rein, 1983) as a means of scaffolding the archaeological excavation, and exposing how policy agents see their reality and act upon it (Laeni et al., 2021). A sort of ‘storytelling’ (van Hulst, 2012) or ‘narration’ (Rein and Schön, 1996), framing helps explain ‘what has been going on, what is going on, and, often, what needs to be done – past, present and future corresponding to the plot line of a policy story’ (italics in original, van Hulst and Yanow, 2016: 110). Drawing on this representation, and to ensure detachment from the act of policymaking, I offer the following questions as a basis for policy archaeology:

In what environment was the policy developed?

What were the factors influencing the policy approach taken?

Did the policy have its desired effect, and how do you know? (i.e. What is the evidence base?)

How has the policy evolved over time?

And crucially, what are the implications of this experience for future iterations of policy?

These questions are put to the test in the coming section, as I don my metaphorical gloves, brush off my imaginary trowel and begin to chip away at more than 20 years of Welsh accountability policy.

The changing face of accountability – a Welsh example

In this brief example of policy archaeology in action, I respond to each question in turn so as to explore the constitutive rules from which the Welsh Government's approach to accountability has emerged. This is intended as a demonstration of policy archaeology in practice, and whilst deliberately concise, recognises the need to dig deeper (considering chronology of events over time) and more widely (recognising the impact of more contextual factors) so as to unearth the historical artefacts of most significance.

In what environment was the policy developed?

Understanding the environment in which accountability policy was developed is crucial to our making sense of its relative effectiveness. In this case, the abandonment in the mid-2000s of two prominent forms of accountability marked a sea-change in the way schools in Wales would be held accountable for their performance. Secondary school league tables, which ranked schools according to their relative performance in external, end-of-year examinations, and Standardised Assessment Tests (SATs), for pupils aged 7, 11 and 14 years, were phased out in Wales between 2001 and 2005. With regards to the former, the then Welsh Assembly Government (WAG, known today as Welsh Government) was of the view that parents were interested more in the performance of their child's school than the performance of schools more generally, which was ‘of little relevance to them’ (WAG, 2001: 35). Similarly, ministers believed that the frequent testing of children was no longer required and removing SATs would help ‘reduce the administrative burden’ on teachers and ‘release more time for other classroom activities’ (Western Mail, 2001, par.5).

A subsequent review of pupil assessment, commissioned by WAG after the removal of SATs for 7-year olds, argued that the tests were damaging to the breadth of the curriculum and put unnecessary strain on teachers and pupils (Curtis, 2004). Approving the review's recommendations, Wales’ then education minister Jane Davidson confirmed amid pressure from teaching unions that SATs for 11 and 14-year olds would also be removed and replaced by strengthened teacher assessments in core subjects (McNulty, 2004). Interestingly, the review's chairman Daugherty (2007: 71) would later reflect on a policy decision ‘seemingly without a long gestation period and certainly not preceded by a lengthy public debate’, although perversely, the decision was ‘almost universally welcomed in Wales’.

What were the factors influencing the policy approach taken?

For some, the decision to row back from SATs and league tables represented a ‘radical departure from the English vision’ (Jones and Roderick, 2003: 124) and, given Wales’ long association with policy reform in England, was reflective of WAG's ‘attempt to get education by consensus, [and] to generate education change by being nice to people rather than being nasty’ (Archer, 2008, par.20). Indeed, that Wales opted to do away with two powerful accountability mechanisms that England had chosen to retain served to exacerbate the growing wedge between the two countries in their approach to education policy reform (Evans, 2015; Power, 2016). The key difference, according to Reynolds (2016: 164), was that teachers in Wales ‘were to be trusted, to be listened to and to be respected, rather than criticised and “shamed”, as in some English educational policy discourse’.

Did the policy have its desired effect, and how do you know? (i.e. What is the evidence base?)

WAG's policy of abandoning SATs and league tables was severely tested in 2010, following the OECD's publication of Programme for International Student Assessment (PISA) results. Considered a major turning point for Welsh education post-devolution (Evans, 2021), the results reflected poorly on Wales’ schools and showed pupil performance to be significantly below UK and OECD averages in the key domains of reading, mathematics and science (NFER, 2010). Wales’ PISA scores attracted widespread denunciation in the media and politically (Lemke & Zhu, 2018; Evans, 2021), with opposition parties claiming the results were ‘a blow for our hopes of boosting growth in the economy’ and that a generation of pupils, parents and teachers had been let down (see, e.g. BBC Wales, 2010).

Some of the more bracing criticism came from within the government itself, with the then education minister Leighton Andrews describing PISA as a ‘wake-up call to a complacent system’ (Dauncey, 2016, par.3). Andrews said the results ‘made it clear that schools in Wales are simply not delivering well enough for students at all levels of ability’ and ‘there can be no alibis and no excuses’. He called on schools, local authorities ‘and ourselves as government’ to look honestly at the results and accept responsibility for them (Andrews, 2010, pars.5–6). Wales’ education inspectorate, Estyn, drew similar conclusions, noting in its annual report that progress had been slow and too many aspects of provision were not up to the standards expected (Estyn, 2011).

A subsequent study into the impact of school-level accountability on pupil progress, published in 2010 by the University of Bristol (Burgess et al., 2010), provided further evidence of the Welsh plight. Researchers compared GCSE (external examinations, typically taken at age 16) outcomes in England and Wales, before and after the abolition of league tables in Wales, to test the policy's impact on school effectiveness. The research found there was ‘systematic, significant and robust evidence that abolishing school league tables markedly reduced school effectiveness in Wales’ (Burgess et al., 2010: 2).

How has the policy evolved over time?

Andrews’ response to the ensuing ‘national hysteria’ (Reynolds, 2016: 166) surrounding PISA was dramatic, and the unveiling of a ‘20-point plan’ designed to resurrect Welsh education included a very noticeable volte-face. The revival of a form of national testing and the reintroduction of a public-facing school ranking system signalled a turnabout in approach and what could be seen as transition to a more neoliberal style of policymaking.

Launched in 2012, a new national ‘banding’ system (known now as the National School Categorisation System) would rank every primary and secondary school in Wales annually using a ‘traffic-light system’ to identify ‘the schools that need the most help, support and guidance to improve’ (Welsh Government, 2019: 2). National Reading and Numeracy Tests for all pupils aged between 4 and 14 years were introduced a year later to provide a national picture of how pupils were performing and make it easier for teachers to identify strengths and weaknesses. The decision by Andrews to renege on earlier policy and reinstate these two manifestations of accountability appears to have stemmed from a perception that schools in Wales had been allowed to coast and pupils were not being sufficiently stretched. In a major address to the education system, Andrews spoke of the need for ‘rigorous benchmarking’, ‘learning from the best’ and using data to empower people (Andrews, 2011, par.55).

Since their inception, both categorisation and national testing (which, until recently, have remained important features of the Welsh Government's standards agenda) have courted controversy, not least because of their similarity to SATs and league tables (Evans, 2015, 2021). And as Wales’ education system transitions into a new era characterised by collaboration and a renewed sense of teacher professionalism (Welsh Government, 2017; Evans, 2021), the Welsh approach to accountability looks set to shift once more (Table 1).

Table 1.

Evolution of school accountability policy in Wales, 1999-present.

Timespan	School-level accountability
Approach 1: 1999–2012	SATs and league tables removed; new focus on teacher assessment
Approach 2: 2012–2022	National tests and school categorisation introduced
Approach 3: 2022-present	National tests and school categorisation scaled back; new focus on self-evaluation

What are the implications of this experience for future iterations of policy?

A new school improvement framework was unveiled by the Welsh Government in June 2022 to better reflect the changing educational environment. The introduction, later that year, of a new purpose-based national curriculum, designed to empower local decision-making for the benefit of individual learners, has brought with it a reconceptualising of accountability (Donaldson, 2018; Welsh Government, 2022); narrow, data-driven performance measures and a focus on headline descriptors has been replaced by a more qualitative and dialogic approach, centred around learner progression.

National Reading and Numeracy Tests have been phased out in favour of more formative, online personalised assessments and National School Categorisation withdrawn in favour of a ‘robust self-evaluation system where good practice can be shared and failure is urgently addressed’ (Welsh Government, 2022). Donaldson (2015: 10) presented the case for change in Successful Futures, an independent report on the future of curriculum and assessment arrangements in Wales, in which he noted that the extent of legislative control and associated accountability mechanisms had ‘inhibited professionalism, agility and responsiveness in dealing with emerging issues’ and ‘forced too-frequent political intervention in non-strategic matters’. The document can be considered an artefact of great significance in this context.

In some respects, the return to a ‘lighter touch’ mechanism for accountability and school improvement, and the quiet erosion of some of the more draconian measures put in place under Leighton Andrews, means that the education system in Wales is starting to resemble much more obviously that presided over by Jane Davidson in the early part of the twenty-first century. However, with ongoing concerns about the variation that exists between schools’ curriculum realisation and teacher quality (see Estyn, 2023), one suspects the story of Welsh accountability policy is far from complete.

Implicit in all of this, is the need for some sort of policy congruence and a longer-term approach to reform built on political permanence and stability. So too is the notion that policy should be able to withstand the force of major events (e.g. the publication of PISA results and Successful Futures) and in so doing avoid the type of causal sequence that results in reform fatigue. Moving forward, policymakers would do well to consider the impact of such policy oscillation on teachers, in particular.

Final thoughts

I began this opinion piece by advocating a more considered review of educational policy design and development through means of policy archaeology. My condensed exploration of Welsh accountability policy since devolution provides a working example of how this might be done in practice, using a series of key questions to map out areas of most interest. Framing the dig in this way allows policy analysts to explore policy breadth as well as depth, and take account of both the chronological and contextual factors impacting on policy maturation.

In the Welsh case, accountability policy appears to have been driven, at least in part, by external influences (e.g. PISA, Successful Futures) rather than any significant concern or protestation emanating from within the education system itself. However, surrender to such influences is not inevitable and policymakers have in their gift the power to decide who and what is ascribed value. Returning to the rules of formation, it is what Scheurich (1997: 98) calls the ‘grid of social regularities’ that allows ministers to construct both the problem and its solution.

In Wales, ministers have identified PISA as a problem (despite its known limitations – see, e.g. Berliner, 2020), and thus socially legitimised school categorisation and national testing as a policy solution. Subsequently, and upon publication of Successful Futures, categorisation and testing were considered barriers to Donaldson's vision for education and thus removed in favour of more qualitative measures of success. As Walton (2010) makes clear, like the labelling of the problem and the problem group (in this case schools and, by implication, their teachers), the grid of social regularities also constitutes policy solutions.

Striking, however, is how little these Welsh policy solutions have been scrutinised and debated; particularly given their material impact on teacher behaviours. The lack of formal consultation on the future of SATs and league tables means that the decision to remove and then resurrect them was almost entirely political, based on sitting ministers’ interpretation of events and how best to respond to them. This raises a number of questions; for example, what prevents meaningful discussion of such issues? And how might a minister of the same political affiliation as their predecessor manage transition between policy that is so diametrically opposed?

The answer, I suspect, relates to the wider political context in which these debates are located; namely, one that apportions near total control to a single political party. Ministers in Wales are reluctant to engage in ‘blame games’ (Hinterleitner, 2018; Hood, 2002, 2011) and have not been openly critical of each other's work, content that their party's continuity in government is seemingly all but assured; there is far more to lose than there is to gain from personal rebukes made public. This political security has also likely protected ministers from more intense examination; there appears a reluctance in Wales to bite the hand that feeds.

There is a case for the ruling Welsh Government to answer when it comes to its handling of school-level accountability, evidenced by a system suffering the effects of reform fatigue and a profession pulled from pillar to post, underpinned by an incoherence that belies two decades of administrative constancy. However, I am conscious that this brief exercise in policy archaeology tells only part of the story, and a more expansive dig would doubtless reveal other sites worthy of surveyance. But policy archaeologists beware – in exposing policy flaws, and the questionable decisions from which they originated, one exposes themself to possible retaliation. This is particularly true in a system governed for so long by a dominant political force. Then again, Indiana Jones never had it easy.

Footnotes

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Gareth Evans

Author biography

Gareth Evans is director of Education Policy at Yr Athrofa, the University of Wales Trinity Saint David's Centre for Education. He is a specialist in the design, development and implementation of education policy, particularly in the Welsh context, and has written extensively on Wales’ education system since devolution. He is a member of the International Educational Assessment Network (IEAN) and leads the university's involvement in the National Professional Enquiry Project (NPEP).

References

Andrews

(2010) In G. Evans (Ed.), Alarm as education in Wales comes bottom of the class. WalesOnline . Available at: https://www.walesonline.co.uk/news/wales-news/alarm-education-wales-comes-bottom-1879106 (accessed 20 August 2022).

Andrews

(2011) Teaching makes a difference. Institute of Welsh Affairs . Available at: https://www.iwa.wales/click/2011/02/teaching-makes-a-difference/ (accessed 20 August 2022).

Archer

(2008) Wales eliminates national exams for many students. Education Week 26(16). Available at: https://www.edweek.org/policy-politics/wales-eliminates-national-exams-for-many-students/2006/12 (accessed 20 August 2022).

Ball

(2021) The Education Debate, 4th ed. Bristol: The Policy Press.

BBC Wales (2010) Pisa tests show pupils in Wales falling behind. BBC Wales . Available at: https://www.bbc.co.uk/news/uk-wales-11930257 (accessed 20 August 2022).

Berliner

(2020) The implications of understanding that PISA is simply another standardized achievement test. In: Fan

Popkewitz

(eds) Handbook of Education Policy Studies. Singapore: Springer. DOI: 10.1007/978-981-13-8343-4_13.

Burgess

Wilson

Worth

(2010) A natural experiment in school accountability: the impact of school performance information on pupil progress and sorting. CMPO Working Paper Series; No. 10/246 . University of Bristol, CMPO, pp. 0–45.

Curtis

(2004) Wales to scrap school tests. The Guardian . Available at: https://www.theguardian.com/education/2004/jul/13/schools.uk5 (accessed 20 August 2022).

Daugherty

(2007) National curriculum assessment in Wales: evidence-informed policy? Wales Journal of Education 14(1): 69–84.

10.

Dauncey

(2016) In brief: improving school standards. National Assembly for Wales . Available at: https://seneddresearch.blog/2016/06/10/improving-school-standards/ (accessed 20 August 2022).

11.

Donaldson

(2015) Successful Futures: Independent Review of Curriculum and Assessment Arrangements in Wales. Cardiff: Welsh Government.

12.

Donaldson

(2018) A Learning Inspectorate: Independent Review of Estyn. Cardiff: Estyn.

13.

Estyn (2011) The Annual Report of Her Majesty’s Chief Inspector of Education and Training in Wales: 2009–10. Cardiff: Estyn.

14.

Estyn (2023) HMCI Annual Report: 2021-22. Cardiff: Estyn.

15.

Evans

(2015) A class apart: Learning the lessons of education in post-devolution Wales. Cardiff: Welsh Academic Press.

16.

Evans

(2021) Back to the future? Reflections on three phases of education policy reform in Wales and their implications for teachers. Journal of Educational Change. https://doi.org/10.1007/s10833-021- 09422-6.

17.

Fadyl

Nicholls

(2013) Foucault, the subject and the research interview: a critique of methods. Nursing Inquiry 20(1): 23–29.

18.

Foucault

(2002) The Archaeology of Knowledge (A.M. Sheridan Smith, Trans.). Abingdon: Routledge.

19.

Gale

(2001) Critical policy sociology: historiography, archaeology and genealogy as methods of policy analysis. Journal of Education Policy 16(5): 379–393.

20.

Hinterleitner

(2018) Policy failures, blame games and changes to policy practice. Journal of Public Policy 38(2): 221–242.

21.

Hood

(2002) The risk game and the blame game. Government and Opposition 37(1): 15–37.

22.

Hood

(2011) The Blame Game: Spin, Bureaucracy, and Self-Preservation in Government. Princeton: Princeton University Press.

23.

Jones

Roderick

(2003) A History of Education in Wales. Cardiff: University of Wales Press.

24.

Kincheloe

(1991) Educational historiographical meta-analysis: rethinking methodology in the 1990s. International Journal of Qualitative Studies in Education 4(3): 231–245.

25.

Laeni

van den Brink

Trell

, et al. (2021) Going Dutch in the Mekong Delta: a framing perspective on water policy translation. Journal of Environmental Policy & Planning 23(1): 16–33.

26.

Larner

Jones

Poole

, et al. (2022) Incumbency and identity: the 2021 Senedd Election. Parliamentary Affairs 2022. DOI: 10.1093/pa/gsac012.

27.

Lemke

Zhu

(2018) Successful futures? New economy business logics, child rights, and Welsh educational reform. Policy Futures in Education 16(3): 251–276.

28.

McNulty

(2004) SATs should be ‘phased out’ says Welsh report. The Guardian . Available at: https://www.theguardian.com/education/2004/jan/22/schools.sats (accessed 20 August 2022).

29.

NFER (2010) PISA 2009: achievement of 15-year-olds in Wales. NFER.

30.

OECD (2014) Improving schools in Wales: An OECD perspective. OECD.

31.

Power

(2016) The politics of education and the misrecognition of Wales. Oxford Review of Education 42(3): 285–298.

32.

Rein

(1983) Value-critical policy analysis. In: Callahan

Jennings

(eds) Ethics, The Social Sciences, and Policy Analysis. The Hastings Center Series in Ethics. Boston, MA: Springer, 83–111.

33.

Rein

Schön

(1996) Frame-critical policy analysis and frame-reflective policy practice. Knowledge and Policy 9: 85–104.

34.

Reynolds

(2016) Education in Wales: how do we move from troubled and troubling to transformational? Welsh Journal of Education 18(1): 161–178.

35.

Scheurich

(1994) Policy archaeology: a new policy studies methodology. Journal of Education Policy 9(4): 297–316.

36.

Scheurich

(1997) Research Method in the Postmodern. London: Falmer Press.

37.

van Hulst

Yanow

(2016) From policy “frames” to “framing”: theorizing a more dynamic, political approach. The American Review of Public Administration 46(1): 92–112.

38.

van Hulst

(2012) Storytelling, a model of and a model for planning. Planning Theory 11: 299–318.

39.

WAG (2001) The learning country: a paving document. Welsh Assembly Government.

40.

Walton

(2010) The problem trap: implications of policy archaeology methodology for anti-bullying policies. Journal of Education Policy 25(2): 135–150.

41.

Welsh Government (2017) Education in Wales: Our National Mission; Action Plan 2017–2021. Cardiff: Welsh Government.

42.

Welsh Government (2019) National School Categorisation System: A guide for Parents and Carers. Cardiff: Welsh Government.

43.

Welsh Government (2022) School improvement guidance: framework for evaluation, improvement and accountability. Hwb . Available at: https://hwb.gov.wales/evaluation-improvement-and-accountability/school-improvement-guidance-framework-for-evaluation-improvement-and-accountability (accessed 15 January 2023).

44.

Western Mail (2001) Davidson scraps tests and tasks for seven-year-olds. The Free Library . Available at: https://www.thefreelibrary.com/Davidson+scraps+tests+and+tasks+for+seven-year-olds-a079643099 (accessed 20 August 2022).

Scratching beneath the surface: Using policy archaeology to better understand education reform in Wales