Policy misrecognitions and paradoxes: Developing more contextually attuned access and equity policies in Australian higher education

Abstract

In this paper we discuss recent policy attempts (in 2017) to introduce new frameworks for Australian higher education access and equity programs. These include introducing fees and a tendering process for access or ‘enabling’ programs, as they are called in Australia, and an evaluation framework based on an evidence hierarchy for widening participation or ‘equity’ programs. We illuminate how those policymaking attempts contradict the conditions required for equity-oriented programs because they misrecognise the experiences of the participants. We argue that different conceptual approaches to provision and evaluation are required for practitioners, providers and policymakers to shape future policy together (Heimans and Singh, 2018) so that enabling and equity programs can be understood in ways that value the knowledges and experiences of the participants involved (Sayer, 2011). Our aim is to contribute to work that disrupts the positioning of ‘objective’ policy evaluation frameworks vs ‘subjective’ practices because this decontextualises (Burke and Lumb, 2018) and oversimplifies (Tesar, 2016a), and may serve paradoxically to reduce the programs’ impacts.

Keywords

Higher education evaluation policy widening participation equity access

Introduction

Policy is both a discursive field (Ball, 1993) and an emotional arena (Fineman, 2000) in which complex power dynamics operate (Simons, Olssen and Peters, 2009; Burke, 2012; Southgate and Bennett, 2014). Paying attention to policy proposals is important because ‘policies govern our being in this world … they guide our decision-making to achieve particular outcomes that are supposed to better our existence and conditions’ (Tesar, 2016b: 141). Given that policies are powerful in shaping both personal and programmatic aims, activities and outcomes, we need to demand that they are contextually attuned (Bennett, 2016) and responsive to contextual shifts and changes (Tesar and Arndt, 2017: 235).

This paper aims to provide the conceptual tools for understanding the potentially destructive ways that two recent Australian access and equity policy proposals (Australian government, 2017; Department of Education and Training (DET), 2017) posed policy ‘problems’ and ‘solutions’ (Bacchi, 2009). Drawing on a range of illuminating theoretical concepts, we seek to identify how those policy desires paradoxically produced disconnected and limiting discourses about, and potentially for, the targeted populations. We discuss how people from disadvantaged groups to whom higher education equity and widening participation policies matter the most are often (mis)represented in neo-liberalised policy discourses as problematic and wasteful of public money because the neo-liberal measures of successful educational engagement that are applied miss much of what is valuable about those programs for the participants. Therefore, narrow, misrepresentative and dismissive policy discourses require scrutiny and change (Tesar and Arndt, 2017; Arndt and Tesar, 2018; Springer, 2016: 4). We argue that equity and widening participation policy understandings and approaches need to be reframed to both value and respond to diverse experiences, values and knowledges (Biesta, 2010; Peters and Tesar, 2016c).

Drawing on recent theory about the problems with governmental desires for specific program evaluation approaches (Burke and Lumb, 2018; Gale, 2017; Morrison and van der Werf, 2018), we argue that conceptual distancing from the programs and participants will increase if objectifying methodologies continue to be prioritised at the expense of those that enable understanding of participant experiences and values. We also explain that if a decontextualising approach to tendering is introduced for providers to apply for capped places for enabling (access) programs, the purpose and impact of the programs could be compromised. The historical contribution of this important national investment strategy, which for almost half a century has enabled access to higher education and other forms of social and economic development for hundreds of thousands of students across Australia, is at serious risk of being reduced if places are not allocated with contextual diversity at the forefront of considerations.

In the following pages, we briefly outline the Australian enabling and equity context. We then discuss the concepts of policy ‘affects’ (Webb and Gulson, 2012; 2013; Tesar and Arndt, 2017; Sellar and Storan, 2013) and ‘intensions’ (Webb and Gulson, 2012, 2013), which are the powerful microperceptive and intensely felt, subjective and inter-subjective dimensions that drive policy positionings, desires and impacts. We also introduce the concept of ‘policy misrecognition’, which involves the imposition of judgements on others that serve to limit their opportunities. These concepts enable critical understandings of the ways that seemingly objective policies are implicated within hidden, but deeply invested and subjective, politics of difference. In the last section of the paper, we argue that the concepts and approaches driving neo-liberal equity policy framings need to become more open, rigorous and iterative to gain the necessary insights and understandings of what matters to the participants involved (Sayer, 2011). The theoretical concepts discussed emphasise the importance of care-full consideration and contextualisation, so that the policies can continue to support programs that ultimately matter to, and make a difference for, the program participants.

Policy context

Equity policy is a relatively recent political phenomenon for higher education in Australia, although there has been a related history of social justice initiatives (Gale and Tranter, 2011) since the Green and White papers of the late 1980s (Dawkins, 1987, 1988) and, in particular, the 1990 policy A Fair Chance for All (Department of Education, Employment and Training (DEET), 1990). The current form of the national access and equity policy agenda was developed in the Review of Australian Higher Education (Bradley et al., 2008), which was commissioned by the Australian Labor government (2008–2013). This review was largely influenced by the UK access and equity agenda, specifically the Dearing Report (1997).

In Australia, the equity groups identified as underrepresented in higher education policy are: people who: identify as Aboriginal and/or Torres Strait Islanders; are from low socioeconomic (SES) backgrounds; have a disability; are from non-English speaking backgrounds; are from regional and remote areas; and women in non-traditional areas (DEET, 1990; Martin, 1994). In 2009, the Australian Labor government set the target that by 2020, 20% of all undergraduate students would come from low SES backgrounds and 40% of all 25- to 35-year olds would hold a bachelor’s degree. The target was intended to be met through uncapping undergraduate places and by providing funding through the Higher Education Participation and Partnerships Program (HEPPP) introduced in 2010. The aim of HEPPP was to provide funding to universities to undertake activities to increase access to undergraduate courses for people from low SES backgrounds and to improve their retention and completion rates. The transition from elite to mass education has been important for disadvantaged groups in Australia in terms of absolute numbers, but it takes ongoing commitment to achieve substantial changes in relative representation and inequality (Naylor and James, 2015). The major inequities in higher educationremainin access, with research showing that this is not due to a lack of aspiration (James, 2002; Spohrer, Stahl, and Bowers-Brown, 2018; St Clair and Benjamin, 2011);instead,‘students clearly make pragmatic choices about acting on those aspirations’ (Naylor and James, 2015: 11). Many students from underrepresented backgrounds already have ‘high’ aspirations for university study and careers requiring degrees (Gorard et al., 2007; Whitty and Clement, 2015; Bennett et al., 2015; Bowden and Doughney, 2010; Prosser et al., 2008). Therefore, as Gore et al. (2017) argue, policies to ‘raise’ aspirations are ‘missing the mark’; ‘the underrepresentation of low SES students in higher education is more about factors such as the financial implications of attending university, perceptions of longer-term debt and potential constraints on capacity to navigate pathways to university …’ (p. 16) than lack of aspirations for it. Indeed, as Appadurai (2004) argues, the capacity to aspire is based on the development of navigational capabilities, which are developed within socio-cultural contexts in relation to others and according to opportunity.

In this way, a dominant policy focus on aspiration (in the United Kingdom (UK) and Australia in particular) sits in tension with the reasons why particular groups are underrepresented in higher education. In the UK context, Whitty, Hayton and Tang (2015) considered the reasons that ‘barriers’ to widening participation were said to exist, for example, aspiration, awareness, finance and prior attainment, and assessed claims that inequalities in access largely disappear once prior attainment is taken into account. They found that ‘while participation and socio-economic background appear to be correlated, this relationship seems to be mediated through educational attainment’ (p. 46) and they strongly advocated ‘…that we need to consider students’ social and cultural capital and not just their socio-economic status if we are to get a grip on patterns of participation’ (p. 47). Certainly, ‘getting a grip’ on participation patterns is of great interest to many stakeholders in the various contexts of equity and widening participation policy, research, evaluation and practice. Harrison (2012) argues that the statistics used to evaluate the success of Aimhigher, a well-funded UK widening participation initiative, focused on data over which Aimhigher had no control, and this evaluation contributed to that program’s demise. The consequences of poorly conceived approaches to understanding programs must be considered carefully as they can seriously affect many. In the next section we introduce some important concepts to contextualise the initiatives in focus and related policies.

Policy affects and intensions

We are indebted to traditions of policy discourse analysis (in particular, Ball, 2013) concerned with exposing the taken for granted or ‘obvious’ ways of viewing phenomena and discourse. This work shows how policy becomes normalised and ‘taken for granted’ (Ball, 2013; Foucault, 1978) through governmental ‘problem-based solutions’ (Bacchi, 2009). Neoliberalised forms of policy, normalised over time, have enabled the development of contradictory discourses in policy-related texts so they appear in ways that make them look consistent, convincing and straightforward. Although appearing solution based, this policy approach is deeply political and frames problems selectively. Despite claims of objectivity, as with all positions, policy approaches and their narratives are implicated within broader political investments and subjective positionings. Policymakers embed politico-affective investments into policy framings. The weight of powerful ‘policy sensings’ and ‘policy seductions’ (Webb and Gulson, 2012) drive what is considered desirable and of value in an educational domain. Policymakers, like all of us, form attachments to ideas (even though we may not realise or may deny this) (Sayer, 2011). Our beliefs, commitments and politics are always situated and embodied; they are developed from our affections, our responses to relationships and experiences (Hughes, 2012).

Webb and Gulson (2013) developed the important concept of policy intensions to introduce an understanding of how ‘educational policy is frequently engaged with folding subjects along numerous affective lines, if not twisting them into knots’ (p. 57). Helping us to understand that policy is not only productive of depersonalised, institutional-level impacts, the notion of policy affect ‒ how we both affectively produce and read policy through subjective sensings (Webb and Gulson, 2013) ‒ problematises the assumption that policy is unbiased, objective and experienced without experiential or personal depth. Instead, the point is that people do not create and experience policy in a socio-cultural, political or temporal vacuum, without deeply felt stakes or affect (Bennett and Burke, 2017). Policy is ‘invested’ in a political point of view more than is generally recognised or confessed, especially by policymakers. In addition, policies have effects, and in the case of higher education policy, they often conflict with the views and experiences of educators, practitioners and program providers who sometimes express that they experience the affective ‘pressure’ of policy investments.

Enabled by these concepts, we want to focus attention on how policy is ‘affected and affective’, inescapably imbued with specific onto-epistemological commitments that are different across sites of policy creation, practice and experience. These commitments, we argue, offer possibility. As Heimans and Singh (2018) and Boyask, Vigurs and Lubienski (2018) explain, rather than considering policy politics to be about simple imposition, which should be met with ‘resistance’, we are interested in ways that policy might be collaboratively re-formed. This is something that critical-dissensual, collaborative education policy research seeks to do, in extending the focus on ‘how schools do policy’ to ‘how researchers and schools can (re)do policy together’ (Heimans and Singh, 2018: 185; our emphasis). We consider this approach important for finding better ways of developing and practicing policy through collaborative approaches, while recognising that this work is difficult and uneven (Burke, 2002).

In the following section we discuss the important concept of (mis)recognition, which is intended to enable understanding that how we think about others and ourselves is subjectively (re)developed. The process of misrecognising others, including processes of cognition or thinking about phenomena and others, also involves misunderstandings and misrepresentations developed from on our own socio-cultural positionings, attachments and values. We recognise and misrecognise people based on our own experiences, exposure and developed understandings of others’ experiences and values (or not), which is why education is considered such an important part of enabling peaceful societies. So too, policy (mis)recognitions come from socio-politically subjective positionings and conceptions of ‘problems’; however, those subjective positionings are denied within dominant Western neo-liberal discourses that claim simple objectivity.

Policy (mis)recognitions

Webb and Gulson (2015) explain there is an important ‘feel’ of policy, a ‘politics of recognition’ in which policy is shaped by:

… recognition or its absence, often misrecognition of others, and so a person or group of people can suffer real damage, real distortion, if the people or society mirror back to them a confining or demeaning or contemptible picture of themselves (Taylor, 1994: 27; cited in Webb and Gulson, 2015: 517).

Recognition, or its absence, is a key dimension of a framework of social justice presented by Fraser (1997) that can help us to identify the ways in which parity of participation is made either possible or impossible within socio-political arrangements. Fraser’s three realms of social justice – redistribution (economic/material dimensions), recognition (cultural dimensions) and representation (political dimensions) – offer a way to identify and challenge policymaking approaches that (dis)regard the participants in the social fields they ‘intend’ to (re)arrange. For Fraser, the three Rs need to co-exist for socially just participation to be possible. However, our focus here is recognition, or the lack of it, given that many widening participation strategies ‘have in some ways exacerbated cultural injustices through the misrecognition of those constructed as excluded and/or lacking aspiration, experience and knowledge’ (Burke, 2012: 39).

Burke (2017) explains that many people from equity groups in higher education have ‘suffered histories of misrecognition’ (p. 2) because perceptions of ‘Others’, through value judgements and points of comparison, are conventionally (and often subconsciously) considered in terms of white, middle-class frames of reference. McNay (2008) (engaging a debate between Fraser and Honneth) presents misrecognition as ‘institutionalised cultural value patterns that have discriminatory effects of juridical discrimination, government policy, professional practice, or sedimented moral or ideological codes’ (p. 284). Burke (2017) acknowledges the value of Fraser’s conceptualisation of misrecognition as objectively verifiable status subordination in that, for widening participation, this can help to focus the gaze on how institutions can change systems and practices rather than the focus always being the student. Importantly, however, Burke (2012; 2017) also draws on McNay to remind us that experiences of misrecognition are deeply felt in and through the body impacting ongoing identity formation. In this sense, policy misrecognitions are powerful and impact self-perceptions of capability and belonging in higher education (Bennett and Burke, 2017; Burke et al., 2015). It is therefore important for access and equity policy and initiatives to be working to enable opportunities, not limit them, which we argue is important for the programs identified in the following pages. The programs discussed provide opportunities for students to develop capabilities for navigating their way into and through higher education (Bennett et al., 2015).

Access and equity policy and practice

Case 1: University-based access programs – a policy reform attempt

Australian open access programs (known, and federally funded, as ‘enabling programs’) typically run for 6–12 months and are designed for people who do not have the necessary entry qualification (eg. recent school leavers) or academic preparation (eg. mature age students) to enrol in a university program. Through an enabling program, a person can experience university-style study and, if they satisfy the requirements, they are able to qualify to progress to further university study (Bennett et al., 2016; Motta and Bennett, 2018). Enabling is an important post-school entry pathway that provides an experience of university for people who did not move straight into university from school.

Approximately half of the students enrolled in enabling programs are from identified equity groups (Martin, 1994), including Aboriginal and Torres Strait Islander students (Pitman et al., 2016). Since 2004, open access university-based programs have been provided through a combination of government-funded places and enabling ‘loading’, which has meant students have not been required to pay substantive fees. Research has demonstrated that the attraction and thus success of these programs is that they offer stimulating university-based courses that are risk free financially (see Pitman et al., 2016). Individuals are able to determine whether they are able to continue on to undergraduate programs, or to pursue other outcomes as a result of their learning in enabling programs. However, as part of the Higher Education Reform Package (Australian government, 2017), the government proposed that capped enabling places be put out to tender on a 3-year basis, with university-based programs bidding for this funding along with private providers (Bennett, Fagan and Harvey, 2017), and that students pay a contribution fee of A$3271 (or instead start their higher education debt through the HECS-HELP loan scheme). In its outline for ‘new arrangements for enabling courses’ (Australian government, 2017), the reform proposal stated that:

… The arrangements for enabling courses will be overhauled with a fixed number of enabling places to be allocated on a cyclical basis through a three-year competitive tender process. The process will be designed to identify those higher education providers, universities or non-university providers, which achieve high standards of academic preparation and deliver high quality student outcomes, for example, measured by student completion rates or student success in further study … This reform will help improve higher completion rates and ensure a better return to both the student and taxpayers. (p. 25)

This language – with a strong commitment to notions of ‘competition’, ‘measurement’ and ‘return on investment’ – exemplifies the neoliberal approach to policymaking, reinforcing and reifying the individual as a responsible agent making rational choices on a level playing field in the interests of their own social mobility to feed economic productivity. The focus on ‘academic preparation’ and ‘high-quality student outcomes’, measured by completion rates and success in further study, is the first time that policy has attempted to define measures for the outcomes of enabling programs, which also marks the first surreptitious insertion of a definition of aims.

The term ‘enabling’ for university-based access programs has been used in Australian higher education since A Fair Chance for All was released by the DEET in 1990. In 2003, the government defined an ‘enabling course’ as:

A course of instruction provided to a person for the purpose of enabling the person to undertake a course leading to a higher education award, but does not include:

a course leading to a higher education award; or

any course that the Minister determines is not an enabling course for the purposes of this Act. (Department of the Attorney General, 2003: 384)

The aims and outcomes of enabling were grounded in social justice principles, with the Minister for Employment, Education and Training, JS Dawkins writing that:

Social justice is a keystone of Labor policy … This government is committed to the achievement of a fairer and more just society, and is working towards the removal of the barriers which prevent people from many groups in our society from participating fully in the life of our community. (DEET, 1990)

At that point in time, the focus of university awareness and entry programs was on enabling people ‘to consider options that may open up future vocational, social or cultural opportunities’ (DEET, 1990: 45), not squarely on those pre-university programs being measured by how many people continued in them and then gained a degree. This narrowed focus is more recent and does not hold the same wider concerns of people having the chance to consider different opportunities in the context of other vocational and training options. The 2017 proposals introduced a new set of desires for enabling programs focused on relatively short-term completion and progression through higher education. This increasingly myopic focus misrecognises the contribution of the programs, which is to provide an understanding of higher education and the pathways available through that route. These learning journeys are often not undertaken in a linear sequence; they are commonly returned to later on and/or taken over longer periods of time as people juggle competing responsibilities and challenging personal and socio-structural circumstances (Bennett and Burke, 2017). Some people do not to pursue higher education and report that the programs have provided them with the knowledge and skills that open up other work and training opportunities.

This diversity of needs and functions is lost in the proposed changes, where subjective policy positionings, investments, values and commitments are hidden from view. Ironically, the proposed ‘objective’ measures work to conceal subjective political biases. As Harvey (2017) writes, research about the specific contexts and outcomes or existing evidence (see e.g. Hodges et al., 2013; Bennett et al., 2013) was not engaged with; nor was adequate consultation undertaken with the universities with a long history of provision that regularly evaluate their programs through rigorous approaches, including research-informed, external forms of evaluation.

However, following the 2017 policy reform package, political debate developed, with government department representatives and Federal Opposition government leaders visiting the programs to meet with staff and students. In the House of Representatives (12 September 2017) Deputy (Labor) Federal Opposition Leader, Tanya Plibersek, argued:

Enabling courses are actually designed to get people who missed out the first time round – who had troubles in high school or who had a job in a sector which is rapidly changing, lost their job and want to retrain and go to university … I've visited a lot of universities that have brought some of their best and brightest students in through the front door because of these enabling courses. Many of those students are from disadvantaged backgrounds; if they had been told they had to pay thousands of dollars to have a taste of universities, they would not be there … (Commonwealth of Australia, 2017: 10,015)

Sociologists of education have long demonstrated the ways in which education systems (schools, in particular) are commonly structured to exclude certain groups, and how this can replicate privilege and inequality over time (Naidoo and Whitty, 2016). Above, Plibersek highlights the ways in which enabling courses have developed to overcome dimensions of this exclusionary structuring, and how this important equity orientation built over decades could be ended as a result of this policy proposal. Plibersek then describes the ways in which a competitive tender process threatens to undermine quality of provision:

They'll all be put out to tender. They will be put out to the lowest cost. So there's no guarantee that students will get the same sort of high standards that they're getting now. It will be devastating for a significant number of universities, particularly those in the regions. (Commonwealth of Australia, 2017: 10,015)

Given multiple federal governments’ messy, expensive and problematic efforts to deregulate and privatise the Vocational Education & Training (VET) sector (Noonan, 2016), it is arguably prudent to discard this direction for enabling education into the future. With VET, an open-market approach ‘backfired disastrously’, with governments spending billions of extra dollars to train fewer students (Ross, 2018).

The Australian Senate finally blocked the higher education reform package, with universities informed that the government would instead develop a new allocation method and that consultations would be undertaken with the sector to find ways forward. What was problematic initially was that rather than seeking to recognise and understand the programs and participants, the programs were compared to other fee-based university entry programs. In contrast to open-access fee-free enabling, fee-paying programs have lower attrition rates because the students enrolling in them are more sure that they are able to study, having decided they can ‘afford’ the time alongside other competing responsibilities, including paid work and parenting, which many of the students enrolled in fee-free enabling programs consistently say they cannot do. Howeverm despite these initial challenges, the vast majority of the ∼50% of students who do complete enabling across the sector eventually continue on to higher education and progress through to professional careers with a degree or higher qualification (Hodges, 2013).

So, instead of comparing across the different program contexts, comparisons within program types through contextualised forms of evaluation (Bennett, 2016) would provide better understandings of whether the programs serve the needs of the contexts and participants involved.We need to be careful not to reduce impact through applying simplistic decontextualised approaches and policies. As Tesar (2016a) argues:

Policy, at times, gets caught up in positivist discourses, of easy, nice solutions, trying to return to the era of objective truth and simplistic answers … it is often the smallest cracks in the system that make it possible for particular challenges/challengers to open frontiers to new changes, ontologies and epistemologies. (p. 312)

We hope that subjecting policy to analysis as we have done here creates better, more contextually appropriate, approaches to policy frames. Indeed, policy frameworks themselves must be evaluated to assess their appropriateness to the field and to the lives of the stakeholders they claim to serve, as will be discussed below in relation to the proposed evaluation methodologies for equity programs.

Case 2: University-based equity initiatives – another policy attempt

University-led initiatives across the spectrum of pre-access, access, participation, attainment and transition out (Bennett et al., 2015) existed in higher education before the current period of expansion, which has seen growth in the funds allocated to universities for equity initiatives and subsequent proliferation in the range and scale of activities. The HEPPP has consisted of ‘Partnerships’ and ‘Participation’ components. The ‘Partnerships’ component provided funding to build connections with institutions and groups to ‘build capacity’ for ‘people from low SES backgrounds’ for participation in higher education. The ‘Participation’ component focused on access, support, and retention for ‘people from low SES backgrounds’ once entering undergraduate study.

Following a federal DET-commissioned evaluation of the HEPPP in 2016, which recommended the development of a ‘standardised national HEPPP evaluation framework’ (Acil Allen: xxii), a call for competitive applications to develop the framework was released (DET, 2017). Information for applicants was provided in a briefing presented by the DET on 20 June 2017 where it was argued that: ‘the guidance for evaluations of effectiveness will need to be extensive, rigorous and prescriptive’. The key component of the evaluation framework was to provide ‘direction on how to establish an acceptable counterfactual or control group, which will require establishment of a trial-registry, where universities must pre-register their study protocols before beginning data collection’ (DET, 2017). The brief specified that the framework should be constructed according to ‘evidence – categorised in decreasing levels of rigour’, including:

Systematic reviews (meta analyses) of multiple randomised trials;

High-quality randomised trials;

Systematic reviews (meta analyses) of natural experiments and before after studies [sic];

Natural experiments (quasi experiments) using techniques such as differences in differences, regression discontinuity, matching, or multiple regression;

Before-after (pre-post) studies; and

Expert opinion and theoretical conjecture (Leigh, 2009, in DET, 2017).

Following this call for competitive applications to carry out the work, the participants in the briefing session shared concerns based on their expertise as researchers and program providers in the field. Of greatest concern was the hierarchy privileging randomised trials, a method hierarchy that has been identified as problematic for evaluating social programs in other ‘open systems’ (Sayer, 1992; Morrison, 2001; Clegg, 2005; Pawson, 2006). Due to a lack of applications addressing the call, the DET reported it would not award funding and in an email from the DET (8 September 2017), it was explained that:

Unfortunately the department was unable to recommend a provider from the responses received. As a result, we have decided to manage the project in house. This will involve departmental staff managing development of the evaluation framework and undertaking some of the work involved. We will consult closely with the sector and will outsource development of some aspects of the framework.

The major problem with recent policy work about equity and widening participation programs in Australia and the UK is that it limits both provision and evaluation to decontextualised objectives. In contrast, practitioner- and research-informed frameworks have instead focussed on wider, context-based approaches (see Gale et al., 2010; Bennett et al., 2015) to engage and enable students to access and complete higher education. As contextualised approaches take into account, the complex dynamics operating within programmatic spaces are shaped by and situated within wider complexes (of life trajectories and socio-economic dynamics). This brings into question the suitability of methodologies requiring both control and the systematic removal of context.

As Gale (2017) notes, ‘because randomised controlled trials (RCTs) see no difference between physical and social worlds, or [they] regard the social world as operating in the same way as the physical world, they tend to search for the same kinds of knowledge they imagine to exist in the physical world’ (p. 220). Policy and practice-oriented ‘clearinghouses’ have become increasingly popular, promoting research studies on various social interventions and commonly positioning RCT methodologies as the ‘gold standard’ approach to producing evidence. The United States (US) Department of Education standard for ‘strong evidence of effectiveness’ requires a ‘well-designed and implemented’ RCT; no observational study can earn such a label (Deaton and Cartwright, 2017; 2). For example, a What Works Clearinghouse (WWC) within the US Department of Education as part of the Institute of Education Sciences describes its mission as: ‘The goal of the WWC is to be a resource for informed education decision-making. To reach this goal, the WWC reports on studies that provide credible and reliable evidence of the effectiveness of a given practice, program, or policy’ (Ginsberg and Smith, 2016; 1). Yet Ginsberg and Smith (2016) reviewed 27 RCTs in this WWC to determine whether they could help teachers and school administrators make decisions about their choice of mathematics curricula. They found that the existence of 12 serious methodological threats meant ‘none of the RCTs provides sufficiently useful information for consumers wishing to make informed judgments about which mathematics curriculum to purchase’ (2016: iii).

An extensive and growing literature in Western contexts directly challenges the epistemological, ontological and methodological claims upon which this evidence hierarchy operates, including the problems associated with adopting RCTs in social fields of investigation (Cowen, Virk, Mascarenhas-Keyes and Cartwright, 2017). Although largely beyond the scope of this paper, leading scholars and practitioners in research and evaluation methodology continue to raise issues with the ways in which the RCT, and particular problematic notions of causality, have come to be desired by policymakers (Cook, Scriven, Coryn and Evergreen, 2010). Even advocates of scientific approaches, such as Michael Scriven, who accept the RCT can provide a basis for causal inference in certain conditions argue that the RCT does not deserve to be the standard at the exclusion of all other approaches, as is sometimes the case (Lingenfelter, 2015).

Just as policy is constructed through affective experiences and values, it impacts people affectively as well (Webb and Gulson, 2012; 2013; Tesar and Arndt, 2017; Sellar and Storan, 2013). Program participants and providers feel policy implications and threats, and they respond based on their values and their knowledges, which are subjectively developed via their experiences in the field. These policy issues are therefore axiological (Tesar, 2016c; Peters and Tesar, 2016); they are also ethical, and concern whose lives, whose experiences, knowledges and values matter. In dynamic relational spaces with participants of many different backgrounds, interacting in a myriad of ways, it is not feasible to assert the same kinds of claims about certainty and causality made by researchers conducting evaluation of mechanisms within non-inter/intra/personal relational spaces (such as the clinical conditions created as part of drug trials). It is important for policymakers to rethink decontextualising approaches that are ‘in tension’ with the equity intent of the programs. Other contextually attuned methodologies and approaches are better suited to evaluation in the field and for identifying recurring themes and trends across intervention types (see e.g. Bennett et al., 2015; Gale, 2010).

Conclusions: Provision and evaluation as an ongoing relational process

Sayer’s ‘What Matters to People’ (2011) brings attention to the fact that we are all evaluators, including and especially the students at the heart of outreach and access programs (Burke and Lumb, 2018). Such a recognition moves the focus from participants as decontextualised, idealised projections or ‘policy abstractions’ towards more engaged considerations of evaluation processes (Burke and Lumb, 2018) through collaborations between participants and researchers with the necessary insights and knowledge in the field (Bennett et al., 2015). In addition, Lumb and Roberts (2017) argue that it is important for providers and practitioners to work to develop and maintain ‘a critical awareness’ to ‘challenge our own imaginations, and those of practitioners operating across the field of access’ (p. 22).

To prevent reductive and decontextualising approaches to equity and widening participation policy, practice, evaluation and research, we therefore argue for consideration of value/s (Biesta, 2010). This involves questions of whose values are valued, which ontological and epistemological perspectives are ‘value-able’, and whose understandings about the value of an initiative are used to evaluate social programs. Methodologies and approaches that seek to identify trends and understand the views and values of participants are necessary for co-producing cycles of both provision and evaluation that do not operate in paradoxical ways to reproduce the very inequities they intend to address.

The desires present in the policy reform attempts discussed in this paper have resulted from highly neoliberalised values (based on assumptions about students as a particular kind of independent, rational learner who has specific dispositions and values: mostly middle-class ones). These policies achieve appeal through engaging traditional notions of quality with neoliberal concerns about (taxation) ‘burden’, ‘waste’, savings, value-for-money and accountability (Apple, 2013; Southgate and Bennett, 2014). This ‘neo-liberal imaginary’ (Rizvi and Lingard, 2010) and preferred methodology for assessing value, is an intensifying aspect of global higher education (Ball, 2003; Savage et al., 2013) and a dominant feature of policy desires for evaluating equity programs in the UK and Australia in particular (Crawford et al., 2017; Harrison and Waller, 2017; Burke et al., 2018). Our discussion of how certain access and equity policy texts have become preoccupied with valuing specific approaches, processes, outcomes and methodologies, and with certain methods and not others, is intended to highlight how policy is politically invested and subjectively positioned. Such an understanding assists in disrupting problematic conceptual assumptions about ‘objective’ policy vs ‘subjective’ practice.

As Tesar and Arndt (2017) argue, ‘the challenge for policy research, studies and scholarship is … for “keeping it complex” and making space for multiple experiences of diverse cultural knowledges and lived educational experiences’ (p. 665). Drawing on the concepts discussed in this paper, we hope that our theorisations contribute to developing the policy understandings about equity that are important for multiple marginalised groups in our various educational communities. The government’s recent establishment of a national Equity Research and Innovation Panel (2018), and requests to institutions to provide information about enabling programs and ways to evaluate them, may move towards developing better recognition of what matters at both program and policy levels, if the approach is rigorous and contextualised in the ways we have outlined.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Anna Bennett is a Sociologist with publications and projects focussed on issues of access, participation and equity in higher education. She is Interim Director and Head of Research Engagement and Development at the English Language and Foundation Studies Centre (ELFSC), which has offered open-access university entry programs since 1974 at the University of Newcastle, Australia. Over 60,000 people have engaged in the programs over that time.

Matt Lumb is Praxis Fellow at The Centre of Excellence for Equity in Higher Education (CEEHE) at the University of Newcastle, Australia. He has an interest in the ways sophisticated participatory methodologies can make evaluative research processes more productive, and deliver contextualised understandings of the underlying dynamics that produce program impact.

References

Acil Allen Consulting (2017) Evaluation of the higher education participation and partnerships program, Melbourne. Available at: https://docs.education.gov.au/system/files/doc/other/final_heppp_evaluation_report_2017.03.16_0.pdf

Appadurai

(2004) The capacity to aspire. In Rao V and Walton M (eds) Culture and Public Action: A Cross-Disciplinary Dialogue on Development Policy, pp. 59–84. Stanford: Stanford University Press.

Apple

(2013) Knowledge, Power, and Education: The Selected Works of Michael W. Apple. New York: Routledge

Arndt

Tesar

(2018) True Fake News: Reshaping educational policies with the #MarchofOurLives. Policy Futures in Education 16(3) 233–236.

Australian government (2017) The Higher Education Reform Package. Available at: https://docs.education.gov.au/system/files/doc/other/ed17-0138_-_he_-_glossy_budget_report_acc.pdf

Appadurai

(2004) The capacity to aspire: Culture and the terms of recognition. In: Rao V and Walton M (eds) Culture and public action. Stanford, CA: Stanford University Press.

Ball

(1993) What is policy? Texts, trajectories and toolboxes. Discourse: Studies in the Cultural Politics of Education 13(2), 10–17.

Ball

(2003) The teacher's soul and the terrors of performativity. Journal of Education Policy 18(2), 215–228.

Ball

(ed) (2013) Foucault and education: Disciplines and knowledge. London: Routledge.

10.

Bacchi

(2009) Analysing policy: What's the problem represented to be? Frenchs Forest: Pearson Education.

11.

Bennett

Hodges

Kavanagh

, et al. (2013). ‘Hard’ and ‘soft’ aspects of learning as investment: Opening up the neo-liberal view of a programme with ‘high’ levels of attrition. Widening Participation and Lifelong Learning 14(3), 141–156.

12.

Bennett

Naylor

Mellor

et al. (2015) Equity Initiatives in Australian Higher Education: A review of evidence of impact. Newcastle: Centre of Excellence for Equity in Higher Education, University of Newcastle. Available at: https://www.newcastle.edu.au/__data/assets/pdf_file/0016/261124/REPORT-FINAL.pdf

13.

Bennett

(2016) ‘One size does not fit all’: Theorising the measurement of equity initiatives to deconstruct limited spatio-temporalities of purpose and impact. Converging Concepts in Global Higher Education Research: Local, national and international perspectives. Newport: Society for Research into Higher Education (SRHE).

14.

Bennett

A, Motta SC

Hamilton

, et al. (2017) Enabling Pedagogies: A Participatory Conceptual Mapping of Practice. Report submitted to the Centre of Excellence for Equity in Higher Education (CEEHE), University of Newcastle, Australia. Available at: https://nova.newcastle.edu.au/vital/access/manager/Repository/uon:32947

15.

Bennett

Fagan

Harvey

(2017) Programs that prepare students for university study may no longer be free. The Conversation. Availablle at:http://theconversation.com/programs-that-prepare-students-for-university-study-mayno-longer-be-free-77851

16.

Bennett

Burke

(2017) Re/conceptualising time and temporality: an exploration of time in higher education. Discourse: Studies in the Cultural Politics of Education, 1–13.

17.

Biesta

(2010) Why ‘What Works’ still won’t work: From evidence-based education to value-based education. Studies in Philosophy and Education 29(5), 491–503.

18.

Bowden

Doughney

(2010) Socio-economic status, cultural diversity and the aspirations of secondary students in the western suburbs of Melbourne, Australia. Higher Education 59, 115–129.

19.

Boyask

Vigurs

K and

Lubienski

(2018) From critical research to policy. Policy Futures in Education 16(2): 127–129

20.

Bradley

Noonan

Nugent

, et al. (2008) Review of Australian Higher Education, Final Report . Canberra: Department of Education, Employment and Workplace Relations

21.

Burke

(2002) Towards a collaborative methodology: An ethnography of widening educational participation. The Australian Educational Researcher, 29(1), 115–136.

22.

Burke

Hayton

(2011) Is widening participation still ethical? Widening Participation and Lifelong Learning 13(1).

23.

Burke

(2012) The Right to Higher Education: Beyond Widening Participation. London and New York: Routledge.

24.

Burke

Bennett

Burgess

, et al. (2015). Capability, belonging and equity in higher education. Curtin University, Perth: National Centre for Student Equity in Higher Education. Available at: https://www.newcastle.edu.au/__data/assets/pdf_file/0011/243992/CAPABILITY-ONLINE.pdf

25.

Burke

(2017) Access to and Widening Participation in Higher Education. In: Shin JC and Teixeira P (eds) Springer Encyclopaedia of International HE Systems and Institutions. Dordrecht: Springer.

26.

Burke

Lumb

(2018) Researching and evaluating equity and widening participation: Praxis-based frameworks. In Burke, PJ, Hayton A and Stevenson J. Evaluating Equity and Widening Participation in Higher Education. London: Trentham Books.

27.

Burke

Hayton

Stevenson

(2018) Evaluating Equity and Widening Participation in Higher Education. London: IoE Press.

28.

Clegg

(2005) Evidence-based practice in educational research: A critical realist critique of systematic review. British Journal of Sociology of Education 26(3), 415–428. Retrieved from http://www.jstor.org/stable/30036076

29.

Commonwealth of Australia (2009) Transforming Australia's Higher Education System. Canberra: Commonwealth of Australia.

30.

Commonwealth of Australia (2015) Higher Education Funding in Australia. Canberra: Department of Education and Training.

31.

Commonwealth of Australia (2017) House of Representatives. Official Hansard. no. 14. Tuesday 12 September 2017. Avaialble at: http://parlinfo.aph.gov.au/parlInfo/download/chamber/hansardr/306b57b7-f915-4758-a2ac-52cd5c9ce557/toc_pdf/House%20of%20Representatives_2017_09_12_5470_Official.pdf;fileType=application%2Fpdf

32.

Cook

Scriven

Coryn

CLS

, et al. (2010) Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation 31(1) 105–117.

33.

Cowen

Virk

Mascarenhas-Keyes

, et al. (2017) Randomized controlled trials: How can we know “what works”? Critical Review 29:3, 265–292.

34.

Crawford

Dytham

S and

Naylor

(2017) Improving the evaluation of outreach. Proposed Standards of Evaluation Practice and Associated Guidance. The Sutton Trust, The University of Warwick, UK: Office for Fair Access & Department for Education. https://www.offa.org.uk/wp-content/uploads/2017/06/OFFA-Interview-report.pdf

35.

Dawkins

(1987) Higher Education: A policy discussion paper. Canberra, Australian Capital Territory: Australian Government Publishing Service.

36.

Dawkins

(1988) Higher Education: A policy statement. Canberra, Australian Capital Territory: Australian Government Publishing Service.

37.

Deaton

A and

Cartwright

(2017) Understanding and misunderstanding randomized controlled trials. Social Science & Medicine 210: 2–21

38.

Department of Education, Employment and Training (1990) A Fair Chance for All: national and institutional planning for equity in higher education. Canberra: Australian Government Publishing Service.

39.

Department of Education and Training (2017) Higher Education Participation and Partnerships Program 2017 National Priorities Pool Commissioned Project Evaluation framework for the HEPPP. Presentation and briefing. Canberra: Department of Education and Training, delivered 20 June 2017.

40.

Department of Education and Training (2017) Higher Education Participation and Partnerships Program 2017 National Priorities Pool Commissioned Project Evaluation framework for the HEPPP. Email message. Canberra: Department of Education and Training, delivered 8 September 2017.

41.

Deleuze

(1994) Difference and Repetition. New York, NY: Continuum.

42.

Deleuze

(2004) Desert Islands and Other Texts, trans. M Toarmina. Los Angeles: Semiotext(e).

43.

Fineman

(2000) Emotion in Organizations (2nd ed). London: SAGE.

44.

Foucault

(1978) The history of sexuality, volume I. New York: Vintage.

45.

Fraser

(1997) Justice interruptus: Critical reflections on the ‘Postsocialist’ condition. London and New York: Routledge.

46.

Gale

Sellar

Parker

, et al. (2010) Interventions early in school as a means to improve higher education outcomes for disadvantaged (particularly low SES) students. Underdale: National Centre Student Equity in Higher Education.

47.

Gale

Tranter

(2011) Social justice in Australian higher education policy: An historical and conceptual account of student participation. Critical Studies in Education 52(1), 29–46.

48.

Gale

(2017). What’s not to like about RCTs in education? In A. Childs & I. Menter (Eds.), Mobilising Teacher Researchers: Challenging Educational Inequality. London: Routledge, pp. 207–223.

49.

Ginsberg

Smith

(2016) Do randomized controlled trials meet the “gold standard”? A study of the usefulness of RCTs in the What Works Clearinghouse. American Enterprise Institute. http://www.aei.org/publication/do-randomized-controlled-trials-meet-the-gold-standard/

50.

Gorard

Adnett

May

, et al. (2007) Overcoming the Barriers to Higher Education. London: Trentham Books.

51.

Gore

Holmes

Smith

, et al. (2017) Unpacking the career aspirations of Australian school students: Towards an evidence base for university equity initiatives in schools. Higher Education Research and Development, 1–18.

52.

Harrison

(2012) The mismeasure of participation: How choosing the ‘wrong’ statistic helped seal the fate of Aimhigher. Higher Education Review 45, 1: 30–60. http://eprints.uwe.ac.uk/17900/

53.

Harrison

Waller

(2017) Success and impact in widening participation policy: What works and how do we know? Higher Education Policy 30(2), 141–160.

54.

Harvey

(2017) Translating academic research to higher education policy reform: The case of enabling programs. International Studies in Widening Participation (4) 1:7–17.

55.

Heimans

S and

Singh

(2018) Putting the steam back into critique? ‘Gathering’ for critical–dissensual collaborations in education policy research. Policy Futures in Education 16(2): 185–201

56.

Hodges

Bedford

Hartley

, et al. (2013). Enabling retention: processes and strategies for improving student retention in university-based enabling programs: Final report 2013. Canberra: Australian Government Office for Learning and Teaching.

57.

James

(2002) Socioeconomic background and higher education participation: An analysis of school students’ aspirations and expectations, Evaluations and Investigations Program. Canberra: Department of Education, Science and Training. http://www.voced.edu.au/content/ngv%3A2433

58.

Leigh

(2009) ‘What evidence should social policymakers use?’ Australian Treasury Economic Roundup 1:27–43.

59.

Lingenfelter

(2015) “Proof,” Policy, and Practice: Understanding the Role of Evidence in Improving Education. Sterling, VA: Stylus Publishing.

60.

Lumb

Roberts

(2017) The inedito viavel (untested feasibility) of practitioner imaginations: Reflections on the challenges and possibilities of dialogic praxis for equity and widening participation. International Studies in Widening Participation 4(1):18–33.

61.

Macqueen

(2018) Family and social capital for the success of non-traditional students in higher education. International Studies in Widening Participation. 5(1): 37–50.

62.

Martin

(1994) Equity and General Performance Indicators in Higher Education. Canberra: Australian Government Publishing Service.

63.

McNay

(2008) The trouble with recognition: Subjectivity, suffering, and agency. Sociological Theory 26: 271–296.

64.

Motta

Bennett

(2018). Pedagogies of care, care-full epistemological practice and ‘other’ caring subjectivities in enabling education, Teaching in Higher Education 23: 631–646.

65.

Morrison

(2001) Randomised controlled trials for evidence-based education: Some problems in judging ‘what works’. Evaluation and Research in Education 15 (2):69–83.

66.

Morrison

K and

van der Werf

(2017) Discourses of diversity in evidence-based educational research and policy making. Educational Research and Evaluation, 23:3–4, 75–77,

67.

Naidoo

Whitty

(2014) Students as consumers: Commodifying or democratising learning? International Journal of Chinese Education 2:212–240.

68.

Naylor

James

(2015) Systematic equity challenges: An overview of the role of Australian Universities in student equity and social inclusion. In Shah M, Bennett A and Southgate E. (2015). Widening higher education participation: A global perspective. Oxfordshire: Chandos Publishing.

69.

National Committee of Inquiry into Higher Education (1997) Higher Education in the Learning Society (Dearing Report). http://www.leeds.ac.uk/educol/ncihe/

70.

O'Shea

May

Stone

Delahunty

(2017) First-in-family students, university experience and family life: Motivations, transitions and participation. United Kingdom: Palgrave Macmillan.

71.

Pawson

(2006) Evidence-based policy: A realist perspective. London: SAGE.

72.

Peters

Tesar

(2016) The critical ontology of ourselves: Lessons from the philosophy of the subject. In: Peters

Tesar

(eds) Beyond the Philosophy of the Subject: An Educational Philosophy and Theory Post-structuralist Reader. London: Routledge.

73.

Pitman

Koshy

(2014) A Framework for Measuring Equity Performance in Australian Higher Education: Draft Framework Document V1.6. Bentley, WA: National Centre for Student Equity in Higher Education, Curtin University.

74.

Prosser

McCallum

Milroy

Comber

B and

Nixon

(2008) “I am smart and I am not joking”: Aiming high in the middle years of schooling. Australian Educational Researcher 35(2): 15–35.

75.

Rizvi

Lingard

(2010) Globalizing education policy. London: Routledge.

76.

Savage

Sellar

S and

Gorur

(2013) Equity and marketisation: Emerging policies and practices in Australian education. Discourse: Studies in the Cultural Politics of Education 34(2): 161–169.

77.

Sayer

(1992) Method in Social Science. London: Routledge.

78.

Sayer

(2011) Why Things Matter to People: Social Science, Values and Ethical Life. New York: Cambridge University Press.

79.

Sellar

Storan

(2013) ‘There was something about aspiration’: Widening participation policy affects in England and Australia. Journal of Adult and Continuing Education 19(2): 45–65.

80.

Simons

Olssen

Peters

(eds) (2009) Re-reading education policies. Boston: Sense Publishers

81.

Southgat

Bennett

(2014) Excavating widening participation policy in Australian higher education: Subject positions, representational effects, emotion. Creative Approaches to Research 7(1): 21.

82.

Spohrer

Stahl

G and

Bowers-Brown

(2018) Constituting neoliberal subjects? ‘Aspiration’ as technology of government in UK policy discourse. Journal of Education Policy 33(3):327–342.

83.

Springer

(2016) The Discourse of Neoliberalism: An Anatomy of a Powerful Idea. London: Rowman & Littlefield.

84.

St Clair

Benjamin

(2011) Performing desires: The dilemma of aspirations and educational attainment. British Educational Research Journal 37(3) 501–517.

85.

Taylor

(1994) The politics of recognition. In A. Gutmann (ed), Multiculturalism: Examining the Politics of Recognition. Princeton, NJ: Princeton University Press, pp. 25–73.

86.

Thompson

Cook

(2014) Education policy-making and time. Journal of Education Policy 29(5), 700–715.

87.

Tesar

(2016a) Policy and philosophy in the contemporary educational landscape. Policy Futures in Education 14(3):311–313.

88.

Tesar

(2016b) The future of Policy Futures in Education. Policy Futures in Education 14(2) 141–146.

89.

Tesar

(2016c). On ethics, policy and the philosophy of education Policy Futures in Education 14(6) 593–596.

90.

Tesar

M and

Arndt

(2017) Cross-cultural complexities of educational policies. Policy Futures in Education 15(6) 665–669.

91.

Webb

Gulson

(2012) Policy prolepsis in education: Encounters, becomings, and phantasms. Discourse: Studies in the Cultural Politics of Education 33(1), 87–99.

92.

Webb

Gulson

(2013) Policy intensions and the folds of the self. Educational Theory 63(1), 51–68.

93.

Webb

Gulson

(2015) Faciality enactments, schools of recognition and policies of difference (in-itself). Discourse: Studies in the Cultural Politics of Education 36:4, 515–532,

94.

Whitty

Clement

(2015) Getting into uni in England and Australia: Who you know, what you know or knowing the ropes? International Studies in Widening Participation 2(2), 44–55.

95.

Whitty

Hayton

Tang

(2015) Who you know, what you know and knowing the ropes: A review of evidence about access to higher education institutions in England. Review of Education 3(1): 27–67.