Abstract
This article explores the relationship between education, training and the single market, focusing on the market for the production of policy evaluations in the areas of education and training, culture and youth of the European Commission. Two questions are addressed: the first question relates to the geographical distribution of the organisations that deliver policy evaluation services to the European Commission (‘Commission’) in those areas; and the second relates to the nature of the ideas for policy development put forward in the evaluations examined. Based on information gathered from 23 evaluations carried out between 2012 and 2016 (in particular, the circa 300 recommendations they included), the analysis reveals that although the Commission relied on competitive processes for the award of those evaluations, competition was somewhat restricted: there is a marked dominance of a limited number of countries as the powerhouses for the Commission’s education policy evaluation. In relation to the second question, and by contrast to other policy spaces, the analysis provided little evidence of unfettered penetration of private sector ‘ideological repertoires’, lexicons and sensitivities into the European policy evaluation space.
Keywords
Introduction
The free movement of goods, services, capital and people, introduced by the Treaty of Rome in 1958 and strengthened in the Lisbon Treaty signed in 2007, is an essential component of the European Union. Although the practical achievement of these freedoms is work in progress, European institutions present these freedoms as the ‘cornerstone’ (European Commission, 2014: 3) of a European single market that has brought about economic growth to the benefit of businesses and the over 500 million ‘consumers’ within it (European Parliament, 2017). Some of the most visible recent actions of the EU with regards to the single market are in the telecommunications area, with the recent abolition of ‘roaming charges’ – surcharges applied by operators when a phone is used outside its home country – and in the integration of the business energy sector, financial intermediation and transport (European Parliament, 2017).
The relationship among education, training and the single market is less visible to the public, but has been central in the thinking of many Europeanists, such as Jacques Delors (Corbett, 2005). There are various ways in which education relates to the fundamental freedoms of the single market. The EU is keen on increasing the number of Europeans that move cross-border to study and work (European Commission, 2014; Souto-Otero et al., 2013). Mutual recognition of qualifications across EU countries is central for the functioning of the single market, and in particular for the mobility of workers (Monti, 2010). European education policies and programmes aim to stimulate transparency, and to some extent convergence, across European education and training systems (Westerheiden et al., 2010), in order to facilitate the mobility of people. There are, in addition, various ways in which education has become commoditised and subject to liberalisation and trade in itself, in the EU and globally (Robertson, 2010).
In this article, I focus on a less researched topic in the study of European education and training policies: the market for the production of policy and programme evaluations. Policy is a key terrain as it can open up the way to new kinds of privatisation through regulatory change. The ‘privatisation of policy’ refers to the production of policy related texts, such as evaluation, monitoring and policy reports, by consultancies, think-tanks and other (semi)private organisations (Ball, 2009; Mahony, Menter and Hextall, 2004; Souto-Otero, 2015a). Although often operating underneath the radar of the public, such work has significant consequences for the future shape of education policy and practice. Given the ‘creeping’ activity of the EU to expand its education-related remit (Amaral and Neave, 2009), this is an aspect that deserves urgent exploration. The article explores, specifically, two forms of ‘travel’ of evaluation services’ provision, from particular Member States to the collective of the EU, and of ideas around education policy, from the private to the public spheres.
Education policy evaluation and the four freedoms in the EU: Crossing borders
Ball (2009: 89) refers to the privatisation of policy as …[the] export of “statework” to private providers and “agencies” and the formation and dissemination of new policy discourses arising out of the participation of these companies in report writing, evaluations, advice, consultancy and recommendations. In other words, the representatives of the private sector operate inside of government and are part of the policy creation community.
There are multiple ways in which policy privatisation is facilitated or made more commercially attractive thanks to the mobility of goods and capital. This article focuses on the mobility of services (under specific public procurement rules 1 ) and the mobility of workers. Previous studies have looked at the role of experts in the governing of EU education policy (Grek, 2013; Souto-Otero, 2015a) and its standardisation (Lawn, 2011), where the ‘evaluative state‘ has a key role ‘in the construction of the European space of education’ (Landri, 2014: 29). Under current trends towards the ‘privatisation of policy’ (Mahony et al., 2004), the production, monitoring and evaluation of education policies increasingly rely on externally produced goods and services sold in markets for ‘state-work’ outsourcing (Ball, 2009; Sassen, 2009). The production of advisory policy texts and evaluations could be understood as the production of services or the production of ‘goods’ – a tangible output of the provision of services – production of. The distinction between both is blurred and contested in the case of policy texts, as with standards texts: whereas books are considered goods, and France and Luxembourg saw e-books as books for tax purposes, the European Court of Justice defined e-books as ‘digital services’ instead in 2015 (Brantley, 2015).
Previous studies show that there is a preponderance of private organisations as providers of education policy research and evaluation services in countries like the UK. The production of policy texts by these private policy actors is seen to embody the defence of new discourses, ideas and sensitivities in education. Private sector vocabulary, ways of working and ideas as to how policy should operate penetrate the public sector through their actions (Olssen and Peters, 2005). New public management (NPM) tools, incentive systems, increased autonomy and accountability regimes, state retrenchment and deregulation adopt central roles in the construction of the ‘reform templates’ (van der Wende, 2011) provided by private contractors. Indeed, NPM can be understood as a way of managing (through performance measurement, targeting, the use of market incentives, etc.), but is also associated with specific actions (de-regulation, stimulation of public-private partnerships and of quasi-markets in education, etc.) and narratives (competition, market, value for money, rolling-back of the state, etc.) in education (Hall et al., 2015).
Private consultancies’ work in the European education space is largely a black box, but is seen to relate to specific sensibilities, associated with NPM. Anderson (2016: xv) argues that NPM ‘entered education in many countries through World Bank and OECD [Organisation for Economic Co-operation and Development] reports and private consultancies that too few educational researchers study’. Although the production of research, evaluation and consultancy services at the national level has been the subject of analysis (Ball, 2007; 2012; Grimaldi, 2013; Mahony et al., 2004), their production within the EU has received much less attention (Souto-Otero, 2013; 2015b), which raises questions regarding their nature. We should not assume that things are the same everywhere. There is also a risk that assumptions around the market-state divide prevent us from looking at how actual mechanisms and actors cross this divide, how commercial actors become part of the governmental assemblage of education and reconfigure market-state relations into ‘heterarchical relationships’, where often the state is perceived as ‘only first amongst equals and new private agencies aid government in designing policy’ (Grimaldi, 2013: 441; Simons et al., 2013).
The economic volume of public procurement in Europe is vast, amounting to approximately 14% of the gross domestic product (GDP) per year of the EU (European Commission, 2017). Organisations taking part in public procurement procedures are most commonly based in the same country as the public authorities that award the contracts; but not necessarily. Beyond certain value thresholds, specific European rules combat discrimination against businesses registered in other EU countries, aim to guarantee information about the contracts for tender and guarantee the acceptance of supporting documents (such as certificates and diplomas) issued by another EU country in public procurement procedures. Below those values, although national rules apply, procedures still have to respect the general principles of EU law. Openly defended by EU institutions, cross-border procurement has grown to over 20% of the value of procurement in the EU, ‘creating new opportunities for businesses’ (European Commission, 2017: 1), although a large majority of this cross-border procurement is indirect (‘where the successful bidder is based in the same country as the contracting authority but is a subsidiary of a foreign company’ rather than direct ‘where the successful bidder is not located in the same country as the contracting authority and is not domestically owned’ (VVA Consulting, London Economics and Joint Institute for Innovation Policy, 2017: 24).
It should be noted, in order to understand the range of organisations involved in EU education policy evaluation (a point that will be developed later on), that there are various types of procurement procedures available to public authorities, including open, restricted, negotiated and competitive dialogue procedures (Directive 2014/24/EU) – see Soliño and Gago de Santos (2010) for details. There are also various special forms of procurement. ‘Framework agreements’ or ‘Framework contracts’ are particularly relevant in the case under review. Under this form of procurement, the public authority enters into an agreement with one or various contractors. This agreement outlines the conditions (for example, day-rates to be applied or reimbursement rules for travel and subsistence) for subsequent contracts tendered under that framework agreement. Only those contractors selected to operate under the framework are invited to submit proposals for specific contracts under it. Thus, these agreements are not procurement contracts by themselves, but establish the terms of such contracts in advance and for a predefined period of time, which usually does not exceed four years.
The private and public actors that provide cross-border research and consultancy services to the Commission are by and large based in the EU, although some of their headquarters are located outside of the EU. They benefit from a single market that allows, under public procurement regulations, the free movement of the services that they sell. They also benefit from the free movement of workers, as their workforces are internationalised. Many assignments are completed in a networked fashion, with consultancies developing (regional) skill webs (Brown, Lauder and Ashton, 2008), forming teams that may only come together during direct interactions with clients in Brussels, restricted to the team leaders. The Commission has repeatedly noted that there is room for improvement regarding competition for procurement in Europe, as more than a third of all public contracts are awarded ‘without meaningful competition, either through negotiation or because only one bid was received’ (European Commission, 2017: 2). It wants to further encourage cross-border procurement whenever it can provide better value for money in the provision of goods and services to the public sector.
Over half of procurement procedures within the EU Member States use the ‘lowest price’ criterion as the only award criterion (European Commission, 2017). According to the Commission, competition should not only be on price. Instead, more complex criteria, such as ‘best price quality ratio’, uptake of innovation, green and social procurement should be used. The disposition to consider quality as well as price as relevant criteria is relevant in the context of EU procurement, given large differences in labour costs across geographic areas. As the article will show, the Commission’s use of multiple criteria has resulted in companies with high labour costs competing in the EU education policy market. This is important for the Commission in order to show that it aims to gather high quality evidence through high quality projects.
There is a long-standing debate on the (lack of) use of evaluations by policy-makers (see, for instance Sanderson, 2002; Weiss, 1979; Weiss, Murphy-Graham and Birkeland, 2005). Given its specific interest in presenting itself as an evidence-driven ‘expert’ organisation (Souto-Otero, Flekenstein and Dacombe, 2008) the EU takes policy evaluation seriously and makes much of the evaluation work it commissions work and associated internal assessments of the quality of each of those evaluations undertaken by its evaluation team, public. Although recommendations from the evaluations undertaken may not be accepted, they must be followed-up and acted upon: under its ‘smart regulation’ approach, ‘at the end of the evaluations, appropriate follow up actions must be identified and fed into the decision[-]making cycle’ (European Commission, 2015). In this article, however, the primary interest lies in the nature of the propositions made in the recommendations, rather than their use.
Research questions and methodology
The article addresses two questions, associated with the travel of policy services (across geographic areas) and ideas (from the private to the public sector). The first question relates to the geographical distribution of those organisations that deliver policy evaluation services to the Commission in the area of education and related areas. The article explores three associated sub-questions: What private and public actors are involved? What is the ‘division of labour’ between them, in terms of types of assignments or area of work? Where are they located? Following the Commission’s logic arguing for cross-border competition in the provision of services we expect that the services are delivered by a wide range of providers across European locations, and that these providers include different types of organisations (private sector organisations, universities) given that these different types will have particular expertise profiles. An analysis of the division of labour in the production of these services speaks to a narrative that sees competition and diversity as pathways to the enhancement of the quality of ‘products’. This set of questions also relates to the study of ‘concealed geographies’ 2 in the privatisation of European education policy production processes.
For the analysis of these questions, the lead evaluation company for each assignment was taken as the point of reference for methodological and practical reasons. Methodologically because in these evaluations the lead company often plays a fundamental role in the direction of projects, with associated companies playing a secondary role in data collection and providing limited inputs into the analysis. In fact, in around two-thirds of the contracts examined, lead companies are identified as the only contractor, undertaking the assignment without partners. From a practical point of view, there were some instances in which it was not possible to identify from the evaluation documents whether different consortium partners in framework contracts had played a role in the evaluation or only the lead company had; for example: one evaluation ‘is submitted by the Evaluation Partnership (TEP) to the European Commission on behalf of a consortium led by Economisti Associati s.r.l. and including Navreme Boheme, Amitié and Ipsos-Mori’ (The Evaluation Partnership, 2014: 16).
The second question relates to the nature of the ideas put forward in those policy evaluations, and in particular the extent to which the crystallisation of those ideas into policy recommendations reflect positions associated with ‘neo-liberal ideologies’, NPM and the logics of privatisation, commercial interest and profit-making. The article explores two associated sub-questions: What is the nature of those recommendations for change? To what extent do they create new spaces for influence and profit for education-related businesses and ‘import’ private sector logics into the public sector?
The analysis of recommendations reviewed their nature by looking at two dimensions in particular. First, the order of change proposed: from ‘first-order’ changes in the calibration of levels or settings of policy instruments, to ‘second-order’ changes, or changes that affect the policy instruments employed to achieve certain policy aims and ‘third-order’ changes, or changes that also affect policy objectives (Hall, 1993). It should be noted that, by third-order change, Hall referred not to changes in the specific objectives of a specific policy, but to a change in the objectives or the paradigm (the ‘template’) that guides policy in an area, for example the change from Keynesianism to monetarism as the guide to macroeconomic policy in Britain in the late 1970s and early 1980s. It could be expected that most evaluations would suggest some type of first-order change and a very small number second-order change. Third-order change could be expected to be marginal because the actors involved are frequently politicians together with media and other actors who participate in wide societal debates (Hall, 1993), rather than government officials such as those managing evaluation services, and because the proposition of such fundamental changes will normally be outside of the remit for the evaluations reviewed. Exploring the order of change of the suggestions made in policy evaluations is important in terms of the exploration of their role in the origination of policy change. In terms of accountability, it is important as it indicates the extent to which direction of policy is steered by public policy evaluation consultancies. The analysis in these respects offered here is only partial, as it refers to documents openly available. It should be complemented with an analysis of other ways to influence policy-making through what could be called ‘backstage politics’, and its informal practices and contacts (Souto-Otero, 2017; Wodak, 2015). For example, firms like McKinsey, known to be influential in the area of education (Coffield, 2012), do not perform policy evaluations but use other means to influence the education agenda and bring about paradigmatic change. But analysing policy evaluations as part of the ‘front stage’ offers a useful way to start, and helps us to understand the relationship between evaluation services and policy-making (Vedung, 2017).
Second, the analysis looks at the nature of the recommendations made, in particular at proposed renegotiations of the boundaries of the public and the private. Only main recommendations, identified as such in the evaluation documents, were examined. On average, evaluations offered around 12 recommendations each, resulting in the analysis of circa 300 recommendations. As noted, it could be expected that private companies privilege recommendations that protect or enhance their business. This could take the form of recommendations that argue for further privatisations and business-like methods (Ball, 2009; the ‘pro-privatisation’ hypothesis). Paradoxically, private companies providing services to the public sector may expand their business not by proposing privatisation in new areas and less state-work, but increasing the activities of the public sector, and then benefitting from the public procurement associated with this - for example in terms of additional policy advice and evaluation. Competences and actions in European education policy are not fixed, and there are ample examples in which the Commission has acted in entrepreneurial ways to expand its activities (Corbett, 2005; Souto-Otero, Huisman, Beerkens, et al., 2008). Thus in the EU context, an alternative expectation to the pro-privatisation hypothesis is that, in order to retain or increase their business, the organisations involved in the provision of evaluation services for the Commission aim to grow European competences and actions in education and training (the ‘pro-state action’ hypothesis, whereby business expansion would come indirectly, as a result of the increase of EU actions). In addition, the analysis reviews the use of the ‘ideological repertoires’ (Dale, 1989: 105) of the market in the recommendations made. This includes reference to the virtues of competition between education providers and the creation of quasi-markets, of the expected outputs of education being related to the creation of good workers or consumers; and the maximisation of returns on investment from education, notions of efficiency, choice and individualism, concentration of resources on the most able and a view of inequalities as fair (Ball, 2009; Dale, 1989; Olssen, Codd and O’Neill, 2004).
The data to explore these questions comes from policy evaluation services provided to the Directorate General for Education, Youth, Sport and Culture (previously called the ‘Directorate General for Education And Culture [DG EAC]’), 3 published – as opposed to contracted or undertaken – during the five-year period 2012–2016. Given that NPM and privatisation are often presented as increasingly forceful trends, this (the most recent) period is where stronger messages could be expected. The analysis focused on evaluations undertaken for the DG EAC; education-related evaluations undertaken for other DGs (for example, with regards to international development or humanitarian aid actions) were not included in the analysis. This resulted in 23 evaluations being analysed – a list of the evaluations included in the analysis is provided in Appendix 1.4
The evaluations reviewed were collected from the publicly available (no longer updated) DG Education and Culture website and the Commission’s Smart Regulation website. The DG Education and Culture website only contains evaluations related to the areas of work of that DG. The Smart Regulation website ‘Database of Evaluation files’ has a broader remit. The searches on that website were performed selecting the ’Education and Culture’ policy area. The evaluations selected and incorporated into the study referred to the main areas of activity of the DG: education and training, youth, and culture.5 Although this DG is also responsible for sports, this policy area is less prominent in its activities. No evaluations related specifically to sports were identified for the period under analysis. While the above are the most comprehensive publicly available sources of the evaluation-related work commissioned by DG EAC they are still an incomplete source, as not all evaluation related services are published in these websites. This should be kept in mind when interpreting the results.
Findings
This section reviews the nature of the evaluations examined, the procedures followed for their award and the profile of the companies leading on the evaluation assignments. Second, it analyses ‘geographical travel’, examining the actors involved in the delivery of those assignments. Third, it explores ‘ideational travel’ in the evaluations reviewed, by looking the boundaries and connections that they establish between the public and the private in the recommendations that they present.
Research consortia and concealed geographies
The documents reviewed included different types of interim and ex-post evaluations (and one meta-evaluation) related to education, training, culture, and youth:
Institutional evaluations (such as evaluations of the DG EAC Executive Agency, the European Centre for the development of Vocational Training, the European Training Foundation, the European Commission Library and e-Resources Centre, and the European Commission Traineeship programme);
Policy/strategy evaluations (such as evaluations of the EU Youth Strategy, the Strategic Framework for European Cooperation in Education and Training 2020 (ET2020), and the Open Method of Coordination (OMC));
EU tools (such as evaluations of the European Credit System for Vocational Training, the European Quality Assurance Reference Framework for Vocational Education and Training, the European Qualifications Framework, and the Europass); and
Community actions, operating grants and funding programmes (such as evaluations of the European Capitals of Culture, the European Commission-European Youth Forum Operating Grant, and the FP7 Marie Curie Actions).
The number of evaluations varied per year, as they follow Commission evaluation plans and are also dependent on programme cycles. They ranged from two evaluations in the year 2015 to 11 evaluations in 2013.
The Directorate General for Education, Youth, Sport and Culture made use of framework contracts for evaluation and related services throughout the period analysed. Although this is a competitive procedure, once the preferred contractors to form the framework contract are identified, it reduces the number of competitors for contracts. The Commission selected a single contractor for the period 2006–2010: ECOTEC Research and Consulting (a UK private company). It subsequently tendered a ‘Framework contract for evaluation, evaluation-related services and support for impact assessment’ (contract EAC/50/2009) for which it selected four consortia and then again in 2013 (EAC/22/2013), for which it selected four consortia too.
The consortia selected following the 2009 call were led by four private companies (see Figure 1): ECOTEC Research and Consulting (UK, that would later become ECORYS UK Ltd.; see below for further details), Economisti Associati (Italy); GHK (UK/Belgium, that would later become ICF Consulting Services) and The Public Policy and Management Institute (PPMI) (Lithuania). The consortia selected following the 2013 call were led by three of the previously selected companies (ECORYS UK, ICF, and PPMI) and a new company based in the UK: SQW Ltd., which replaced Economisti Associati. No universities or public research institutes were represented as lead contractors. There is, thus, a high degree of continuity in the leading organisations managing evaluation and related services in the period from 2009, or even from 2006, and 2016. What is also striking is the presence, over that decade and in particular in the last period, of a limited number of countries where these organisations are based. The UK, a country with a well-established tradition of policy evaluation, was without a doubt the leading provider. However, following the results of the UK’s EU membership referendum of 2016, which resulted in a majority (52%) of voters voting to leave the EU and started a process for the UK to leave the EU, ICF intensified its bidding from its Belgian entity, and other companies such as ECORYS could be expected to be doing the same. The Brexit process, as such, has begun to alter the UK’s leading position as the provider of evaluations, which speaks to the importance of having guaranteed access to the single market in shaping companies’ competitive strategies.
Number of evaluation assignments reviewed, by contractor (2012-2016).
The companies leading in the winning consortia in the current framework contract have offices internationally, but only to a moderate degree – much less than global firms such as generalist management consultancy firms that also provide policy advice and evaluation like E&Y, PWC or Deloitte. They are rather ‘nationally based’. Only two had offices in more than one European country: ECORYS and ICF. SQW and PPMI do not have international offices, and consequently have no offices in Belgium.6 PPMI from Lithuania – a country that joined the EU in 2004 – is a relatively new company, having operated since 2001, in contrast to the over 30 years of history of all other ‘leads’.
ECORYS claims to work in more than 100 countries (ECORYS, 2017). However, most of its European education policy evaluation work is undertaken by the UK office – and to a lesser extent the Netherlands. In effect all companies operated from one or two countries. They create ‘skills-webs’ of support linking with other private consultancies and individuals: being ‘A global consulting Provider’ (ICF, 2017) or through ‘partner organisations in all EU countries’ (PPMI, 2017). Their sizes vary significantly: while ICF (NASDAQ ICFI) reports to have almost 6000 professionals worldwide (around 150 in its European policy team), PPMI reports to have ‘over 50 in-house analysts’ (PPMI, 2017).
There were significant changes within consortium leads between the 2009 and 2013. In this short period of time, two of the four leading companies in 2009 changed their name, revealing the active market and investor activity in the sector: ECOTEC Research and Consulting (founded in 1982 as a spin-off from Aston University in Birmingham) became ECORYS UK. This was a moderate change since ECOTEC had merged with the Dutch company NEI Kolpron to create ECORYS in 2000, although ECOTEC was only renamed in 2010 (ECORYS, 2017). GHK underwent a more significant transformation as the company (also founded in the UK in the 1970s) was acquired by ICF, a large American consultancy firm, in 2012.
These companies are, nevertheless, specialist public policy consultancies. In no frame work contract generalist business consultancy companies, main actors in evaluation services at the national level in countries like the UK (see Ball, 2009), were selected. This indicates a difference in the ethos of the companies undertaking such services in these different polities – a point to which I return in below. This expectation is reinforced by the self-presentation that companies make on their websites. The values of these companies are presented as being different from the business-oriented management consultancies, which focus on the creation of economic value for their clients. The PPMI website (PPMI, 2017) illustrates this point clearly when it presents its values as: professionalism, collaboration, curiosity and ‘a happy office’. They emphasise theoretical knowledge as opposed to the delivery of practical solutions that tends to characterise management consultants: ‘We aspire for quality in solving public policy problems by applying state of the art theoretical knowledge & expertise and following the principals (sic) of transparency, openness & responsibility to society’ (PPMI, 2017). SQW provides very little corporate information on its website, only presenting its directors and content related to its assignments, rather than specific marketing materials.
The membership of the consortia was somewhat broader and included organisations from a number of locations other than those mentioned so far. Although I was not able to find data in relation to the members of the consortia selected in 2014, the 2009 call winning consortia included organisations from five additional countries: France, Germany, Czech Republic, Netherlands and Belgium.7 Yet, the organisations delivering EU policy evaluations in education, thus, represent a relatively restricted number of countries. The majority of companies are from more economically developed countries and tend to come from countries that have been part of the EU for extended periods of time.
The evaluations reviewed were often delivered through the cross-border supply of services (across countries) although there were also cases of commercial presence in Brussels. As already mentioned, the fact that the Commission uses selection criteria that rank both the quality and the price of proposals helps to ensure that companies from high-labour-cost countries are not priced out of the competition. The country from which a highest number of assignments were led was the UK (9), followed by Lithuania (5), Belgium (3), Netherlands (3) and France (1). Two assignments did not specify a location, but given that these two assignments were delivered by ECORYS, they were most likely delivered from either the UK or NL.
When more than one organisation participated in the delivery of the evaluation, members of these consortia were often based in the same country as the lead organisation. This can be due to a number of reasons, such as proximity in terms of work culture and networks, but what it clearly shows is that there is no ‘flat world’ in policy evaluation where location is unimportant, despite the four freedoms and the single market.
Distributing policy areas
The actors involved in the evaluations reviewed reflect to a large extent the existence of the Framework Contracts (FWC) discussed: contract form determines actors.
Assignments were unevenly distributed across the selected contractors, with a single contractor (ECORYS) being the lead in over half of the evaluations reviewed. This included large as well as a range of smaller evaluations. Although framework contracts provide a convenient way for the Commission to tender evaluation contracts as they reduce the burden of the administrative procedures to follow and the number of proposals to examine, other means to tender contracts can also be used, in particular for specialist assignments outside of the areas of expertise of the consortia selected in FWCs. The organisations not leading in a framework contract led on a single assignment each in the period under review.
An analysis of evaluations by policy area shows that there is a certain degree of specialisation in the division of labour between contractors. In some instances the contract could be seen as covering more than one of the areas outlined in Figure 2. In those cases the classification of the evaluation was done selecting the main area of the work, based on a qualitative assessment its content. Figure 2 shows that the predominance of ECORYS as a contractor is based on its monopoly over the area of culture (4 assignments) and cross-sectoral evaluations (four assignments) and strong presence in the evaluation of European agencies and networks. PPMI, on the other hand, based its strength in evaluations related to education and training, whereas ICF was represented in this area, and in the area of Youth.
Number of evaluations reviewed, by policy area and contractor.
There are, as such, a relatively limited range of contractors and countries represented as evaluation leads. If we select the three assignments with the highest political profile (the evaluations of the Youth Strategy, Education and Training 2020 and the OMC) this is further restricted, as all those assignments were led from the UK – one by ICF and two by ECORYS. There was limited/no travel of services and providers from/to Southern Europe or Nordic countries. The absence of German companies as lead is also surprising. Given arguments around differences in models and sensibilities around education, training, and youth issues across Europe, this could be considered as an issue of concern. It should be noted that many of the ‘lead’ companies have at least some international staff, from a variety of EU (and in some cases non-EU) countries - which should attenuate the effects of the above concentration to some extent -, although a review of their staff pages suggests that perhaps to a lesser extent than might be expected. The most internationalised offices are the ECORYS and ICF ‘Brussels office’.
Policy ideas
The second question that this article aims to address refers to the nature of the recommendations put forward in policy evaluations, their order of change and the extent to which they speak directly to dispositions generally associated with ‘neo-liberal ideologies’, NPM and privatisation.
Order of change proposed. By and large, the recommendations made by consultants referred to changes in the calibration of settings or administrative procedures in the policy interventions they reviewed. Much of the proposed change covered the following elements:
• Making links between policy objectives and strategies, on the one hand, and activities, outcomes and funding mechanisms, on the other, more explicit;
• Reducing the administrative burden;
• Simplifying the structures/actions of programmes and their administrative arrangements, increase of their user-friendliness and mainstreaming;
• Harmonising systems to achieve economies of scale and efficiency;
• Increasing the transparency of decisions and clarity of procedures;
• Setting up or improving the monitoring systems of the intervention by making better use of available information within the programme or that can be generated by related initiatives (so that the performance of the programme can be examined continuously and evaluations can concentrate more on analysis and less on data collection);
• Standardising concepts and vocabulary (e.g. define effectiveness, or notions such as ‘outreach’, the expected outcomes of a programme, etc.), or data collection instruments to measure performance;
• Benefitting from peer-learning by improving knowledge-sharing systems, and creating databases, to facilitate the spread of ‘lessons learnt’ and good-practice between (in particular EU) agencies/institutions and between programmes – with regards to programme management, utilisation of project outputs, etc.;
• Improving the communication and dissemination strategies of the interventions, to reach their public/target groups more effectively;
• Enhancing dialogue and engagement with civil society;
• Improving coordination, which takes various forms, such as: (a) horizontal coordination between EU strategies (e.g. ET2020, Europe 2020 and the European Semester), institutions, initiatives, and instruments (e.g. European Qualifications Framework [EQF], the European Credit Transfer and accumulation System [ECTS], European Credit system for Vocational Education and Training [ECVET], European Quality Assurance Reference Framework for Vocational Education and Training [EQAVET]) being implemented in related fields; and (b) vertical coordination with Member States, regional agencies and participants; and
• Related to the previous point, ensuring that the division of responsibilities between different bodies is clear and ensuring that appropriate staff attend European activities and ‘the right people are sent to the right meetings’ (ECORYS, 2014: 138).
The changes suggested, therefore, clustered around the prioritisation of objectives (without detailing what objectives should be prioritised), simplification of management and administrative procedures, improvement of data collection and monitoring mechanisms and communication with various stakeholders – internal and external. These changes are more procedural and incremental than substantive and radical in terms of what to do or not do. There is little to suggest that evaluations provide new policy directions or advocate public sector retrenchment.
A small number of evaluations made reference to changes to the objectives of the policy, for example in the Evaluation of the EU Youth Strategy and the Council Recommendation on the mobility of Young Volunteers across the EU or the Evaluation of the Commission’s Traineeship programme. When this happened, consultants almost invariably suggested that the objectives of the initiative should be reduced and formulated in a more specific way, in order for the interventions to gain clarity and focus. There seems to be here some acknowledgement of the limits of the initiatives, given their budget, and the need to incorporate realistic objectives relaxing the demands on an ‘overloaded’ public sector (Ball, 2009). This is consistent with recommendations to reduce the onus of the coordinating role of the Commission in some instances (ECORYS, 2014).
One evaluation referred to the need to formulate more ambitious objectives (ICF, 2016), and to modify the wording of objectives to reflect changes that had already happened in practice with regards to the activities of an EU Agency (PPMI, 2013b). The Evaluation Partnership (2014) proposed to readjust the objectives of an intervention in order to avoid contradictions between its objectives – a recommendation that required generic knowledge rather than engagement with the area. ECORYS’ (2014) ET2020 evaluation suggests that the objectives of ET2020 should respond to urgent social and economic priorities such as those already expressed through Europe 2020 and the European Semester, rather than suggest new objectives. It was a general pattern that consultants did not go beyond comments on the ‘form’ of the objectives selected by the Commission to suggest that certain objectives should be added or removed from the intervention. The above patterns could be criticised for associating complexity with number of objectives, and perhaps for providing a simple formulaic ‘solution’. However, this tendency was also associated with a desire to acknowledge the importance of politics in the policy process: ‘The decision as to whether the EU Youth Strategy should focus on core or youth issues or remain focused on mainstreaming is a political one’ (ICF, 2016: 111), over purely technocratic conceptions.
No evaluation recommended that the Commission should replace the policy instruments under examination by alternative instruments. However, there were frequent requests to produce additional rules for the use of EU tools (e.g. ECVET), practical user-guides (on financial issues, best practices, evaluation of impact, application procedures), reports, websites and social media platforms with an explanatory function before Member States, sectoral organisations and practitioners. These documents were required to explain what the EU is doing, how its actions relate to its strategies and to the actions of other institutions. But they also ask EU institutions to go beyond this, into guidance, for example to provide individuals with ‘tips on how to select and include only relevant information in CV [Curriculum Vitae]’ (PPMI, 2013a: 75). The use of simple language should accompany these explanations, as for most stakeholders EU-speak and highly technical tools are non-comprehensible. This is the case even for Member States, which would seem unable to interpret what the Commission is asking them to do (see ECORYS, 2013a; ICF-GHK 2013a). The vision portrayed in the recommendations, therefore, is one of an ‘explanatory’ and to some extent ‘self-promoting’ state – even though the EU is not truly one – that needs to be continuously self-reflective and communicate its activities to multiple constituencies. These recommendations imply a need for the Commission to present its actions in a simpler way and to further reflect on the connections that exist between the different aspects of what it does. In other words, the recommendations ask the Commission to make what it does ‘familiar’.
This kind of familiarity is easier to achieve for Member States, which touch the lives of their citizens in visible ways on a day-to-day basis, than for the Commission. Member States have larger civil services than the EU, so that the explanatory function can also be diffused between its different structures. The Commission lacks similar support structures and has a much larger constituency to reach. These peculiarities in the governance system of the European educational space increase the importance of detailed explanation from the core to organisations and individuals. But the success of this approach, in turn, relies on the users of Commission policies being endowed with resources (such as time and cognitive resources) to ‘make sense’ of the explanations it may provide. It also requires the existence of incentives to become active users of Commission policies, tools and actions, which partly explains the Commission’s emphasis on evaluation, the identification of good practices and dissemination of results.
Preference for incremental change could be due to the terms of reference for the evaluations, which often direct consultants to the analysis of the extent to which the intervention has achieved its objectives, rather than to the questioning of those objectives. This, in turn, may be related to the stage of the policy cycle of Commission’s policies during the period analysed, as the analysis of the objectives of interventions tends to be more detailed in ex-ante evaluations (DG Budget, 2004). It could also be asked whether the preference for incremental change was the result of the EU education policy arena already having accentuated NPM strategies. Although NPM techniques are undoubtedly present in EU policy-making in education (see Lange and Alexiadou, 2007) this explanation does not seem altogether likely due to the current nature of its policies (Gornitzka, 2017: 57; Souto-Otero, 2017) and because NPM is a relentless process, which is not ‘over’ (Christense and Laegreid, 2017) nor finished.
Shifting boundaries between the public and the private? Recommendations with regards to the shifting of boundaries between the public and the private were expected to be disproportionally favourable to the reallocation of tasks and resources to the private sector, and the creation of new areas for privatisation.
However, the shifting of boundaries from the public to the private was a largely absent theme in the recommendations of the evaluations. The recommendations tended to provide ‘areas for thought’ rather than being prescriptive with regards to future private sector involvement in European education policy; many recommendations related to the affirmation of what was being done and its continuation. When such involvement was mentioned in the recommendations, the pro-state (rather than the pro-privatisation) hypothesis prevailed: companies tried to maintain or increase their market by doing a bit more of the same, rather than by trying to open up whole new areas for private sector involvement. Recommendations in a small number of evaluations referred to the need for further studies (ECORYS, 2012a, 2013b), for example with regards to the use of learning outcomes, the implementation of the EQF recommendation (ICF-GHK 2013b), or to the need for ‘research capacity’ enhancement with regards to the OMC ‘either through research support from within the DG EAC or via a service contract with external providers’ (ECORYS, 2013a: 85).
In terms of the adoption of private-sector sensibilities, vocabularies and ways of understanding, references to the need to focus on excellence (ECORYS, 2014), or to enhance industry academia cooperation (ECORYS, 2012a) were present but scarce. There were a small number of recommendations around target-setting, creation of league tables, and benchmarking, which have been referred to as neo-liberal technologies of government (Bruno, 2009; Lawn, 2011). In one occasion, success was explicitly related to the market criterion of ‘sales of translated works’, and in two there were references to the need to attract private sector funding (into Arts and Culture; Europass initiative). However, mentions of the above themes were rather limited. Even the notion of meritocracy was questioned, as the Commission should, with reference to its internship programme ‘consider the balance between equity of access and excellence of trainees’ and acknowledge that whereas ‘by prioritising the former, the Commission services would aim to maximise the trainees’ contribution to their day-to-day work (…)’ prioritising the latter ‘would better serve the objective of increased employment prospects for trainees in exchange for a reduced (but still substantial) benefit for the institution’ (The Evaluation Partnership, 2014: 99).
Moreover, such mentions were counter-balanced by pro-public sector arguments in the same evaluations. While they included criticisms of some aspects of the initiatives under evaluation (inefficiencies, lack of transparency), none of the 23 evaluations examined suggested that the intervention should be discontinued or drastically reduced. There were no suggestions that the grass will be greener on the other side; recommendations around subcontracting to the private sector, for example, with regards to the management of programmes, were absent. The interventions examined are presented as beneficial, perhaps un-problematically beneficial; for example there is little account in the recommendations of lively debates in the literature around the benefits and problems of the learning-outcomes approach or qualifications frameworks.
The recommendations included some defence of public subsidies (Eureval, 2012), the need to improve accessibility to EU tools for disadvantaged groups (PPMI, 2013a) and to transfer knowledge between more and less developed European regions (ECORYS, 2013b). When discussing the availability of resources, evaluations tended to ask for more rather than fewer resources or ‘efficiency gains’. This was not always in relation to having more resources allocated to the European Commission initiatives directly (however, see The Evaluation Partnership, 2014), but to the networks that support the implementation of its initiatives at national level – such as National Contact Points for these initiatives (see PPMI, 2013a). It is also telling that most of the evaluations do not provide recommendations regarding the sustainability of the initiatives. These are seen to be, largely, depending on public funding and kept away from market mechanisms and pressures.
On the whole, the private sector was not presented as a model to follow. Recommendations address the Commission (a public sector bureaucracy) asking it to do things differently, rather than to stop doing them or to transfer tasks to the private sector. Improvement is associated to peer learning rather than cross-sector (from the private to the public) learning. The potential for improvement resides internally in the public sector, in cooperation with the European Parliament, DG Employment, and the European Education, Youth, Culture and Sport Council, in identifying well-working practices and in facilitating the spread of lessons learnt, rather than importing private sector concepts and practices.
Conclusions
This article set out to examine the production of policy evaluations in the areas of education and training, culture and youth undertaken on behalf of the Commission and its relationship with the single market. The four freedoms that characterise the single market are interconnected. Sometimes, as we have reviewed, they are difficult to disentangle. Freedom in the movement of people and capital, for example, facilitate freedom in the provision of services. Even the practical distinction between categories, in particular goods and services, can be challenging, as this article has discussed. Notwithstanding these caveats, the article focused particularly on two of the four freedoms: the mobility of workers and the mobility of services.
Twenty-three evaluations and almost 300 recommendations produced during the five-year period 2012–2016 were analysed. The article focused, in particular, on the actors involved in those evaluations, and the outputs of the evaluations, in the form of the recommendations that they present. More specifically, the article looked at two questions. First, it examined ‘concealed geographies’ and the geographical distribution of the organisations that deliver policy evaluation services to the European Commission in the areas previously mentioned. Following the Commission’s logic arguing for cross-border competition in the provision of services, we expected evaluation services to have been delivered by a wide range of providers across European locations, and by different types of organisations (private sector organisations, universities). In this respect, the findings reveal that the Commission relies on competitive processes for the award of evaluation services but that this competition is somewhat restricted. The type of contract most often employed during the period analysed were framework contracts, implying a limited number of potential contractors. This has resulted in private companies, rather than a more varied range of research organisations, being lead contractors. Moreover, the study reveals that the evaluation world is not flat, even in a ‘single market’: a limited number of countries dominate this market, as the powerhouses of education policy evaluation. The field is, however, dynamic: the Brexit process shows the importance of the single market in shaping companies’ competitive strategies, as it has begun to alter the UK’s leading position as the provider of evaluations, with major companies in the sector having already moved their leading bidding entity away from the UK.
The second question related to the nature of the ideas put forward in the recommendations of the evaluations examined regarding the balance between the private and the public. I advanced two possible ways in which this could be brought about. One was through a defence of the privatisation of functions previously undertaken by the public sector (a pro-market route), or the creation of new areas of work. A second way was through a defence or expansion of public sector activity (a pro-state or pro-EU, in this case, action route) that could lead to further business for evaluation companies without significant public sector retrenchment. In contrast to the situation in the UK or Italy, there was little evidence of the penetration of private sector ideas and ‘ideological repertoires’, lexicons and sensitivities into the public sector, and little in terms of suggestions for public sector retrenchment or expansion of private sector activities, except in the form of further studies and evaluations. Rather than replacing its vocabulary with market-friendly conceptions around individualism, free choice and competition, the evaluators asked the Commission to become an explanatory state, explaining its initiatives in more direct ways to its various constituencies.
Incremental adjustment, rather than dismantlement, was the prevalent form of the changes recommended. The main concern of the recommendations was to bring European policy into practice, rather than change or roll it back. When needs for improvement were identified, often the point of reference presented by evaluators was to be found in the public sector, rather than in the private sector. As warned by Simons et al. (2013), there is a risk that notions of privatisation, marketisation and neo-liberalism trap and blind scholarly work in its own discourse. These findings underline the need to study different ‘polities’ when looking at the extent and shape of privatisation in education policy-making. The analysis could be further tested by expanding the period covered, in order to incorporate a greater number of cases, stages in the policy cycle, and types of intervention – for example, including evaluations of spending programmes, such as the Lifelong Learning programme. Similar analyses in other countries and sectors would help to more systematically unpack differences and commonalities across political spaces and policy areas.
Footnotes
Appendix 1. List of evaluations reviewed
Declaration of Conflicting Interest
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article:: The author has worked for and with two of the main providers of evaluation services for DG EAC. He was not involved in any of the evaluations reviewed.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.