Five blindspots in reform studies of early childhood education and care (ECEC) policy

Abstract

Policy reform evaluations are an important source of knowledge regarding the (in)effectiveness of policies and have become increasingly influential in informing policymaking. A key advantage of these studies lies in their careful description of a single reform, observing outcomes before and after the reform was implemented, and making precise statements about the effectiveness of the policy reform among its beneficiaries. However, the very nature of reform studies also entails a number of limitations that are important to reflect on – particularly given their popularity in the context of evidence-based policymaking. We highlight five blind spots relevant to evidence-based policymaking, by critically reflecting on a focused literature review of reform studies of early childhood education and care (ECEC) policies published between 2000 and 2021, conducted by the authors and commissioned by the European Commission. The critical reflection highlights: (a) the context-specific nature of reform studies; (b) the limited focus on reducing inequality in the use and benefits of ECEC; (c) a focus on short-term outcomes; (d) a focus on individual-level rather than macro-level outcomes; and (e) various forms of publication bias.

Introduction

Policy reform evaluations are an important source of knowledge of the (in)effectiveness of policies and play an essential role in evidence-based policymaking (Cairney, 2016; Head, 2010). A key advantage to these studies relates to causal inference: precise statements can be made about the effectiveness of a policy reform among its beneficiaries. As such, it is not surprising that reform studies (and in particular randomized trials) have become increasingly influential in informing policymaking (Ravallion, 2020). Yet, the very nature of reform studies also comes with a number of limitations that are important to reflect on in the context of evidence-based policymaking. This short article presents a critical reflection on a focused literature review conducted by the authors and commissioned by the European Commission on early childhood education and care (ECEC) policy reform, covering studies published between 2000 and 2021. This literature review was explicitly commissioned by the European Commission's Directorate-General for Justice and Consumers to outline what could be learned about ECEC policies by focusing solely on evidence from reform studies. Here, we focus on the blind spots such a narrow focus creates and what this means for ECEC policy developed using this evidence base.

Background: Revising the Barcelona targets

In 2002, at a time when women's (and particularly mother's) employment rates were lower than today, the European Council concluded that EU member states needed to do more to increase ECEC attendance because insufficient ECEC availability was seen as a key barrier to maternal employment. Known as ‘the Barcelona targets’, the Council agreed on two objectives: 33% of children younger than three-years-old and 90% of children aged between three and compulsory-school age were to be attending formal childcare by 2010.

The European Union expected the Barcelona targets to lead to an increase in maternal labour market participation, hence strengthening the economy (Council of the European Union, 2002). Indeed, many member states introduced policy reforms in response to the Barcelona targets and ECEC attendance rose from 28.8% to 35.3% for children younger than three and from 86.8% to 89.6% for older pre-school children. As the 20th anniversary of the targets neared, the Barcelona targets had been met on average, yet many member states still lagged behind the original targets. The European Commission therefore planned a revision of the Barcelona targets announced in 2018, explicitly aimed at ensuring upward convergence and an alignment between ECEC targets and recent EU initiatives.

The revision of the Barcelona targets in 2022 took place against a changed EU policy background. The focus in developing ECEC policy was no longer solely on maternal employment but now also encompasses the right of all children to have access to high-quality ECEC, the need for work-life balance, and the need for more targeted support for disadvantaged children and families.

Against this background, the European Parliament invited the EU Commission to revise the Barcelona targets, focused on three underlying issues. First, by 2016, the Barcelona targets had only partially been met, with considerable variation across EU member states (European Commission, 2018). Second, the Parliament saw room for improvement in relation to gender equality in employment and care. Third, gender inequalities were seen to hinder economic development, thus making ECEC reforms economically beneficial (European Parliament, 2018). The requested revision of the Barcelona targets thus occurred under the ongoing assumption that improvements in ECEC would predominantly improve low maternal employment rates, and by extension, would reduce gender inequalities in the division of childcare between men and women, while boosting the economy.

Evidence in the ECEC reform literature review

In preparation of the revision of the Barcelona targets, the European Commission's Directorate-General for Justice and Consumers commissioned the authors of this article to conduct a focused literature review of ECEC reform studies. The aim was to expand the evidence base for the revision of the Barcelona targets through the creation of a database of relevant publications and findings, and a report synthesizing the literature (Nieuwenhuis et al., 2022). In the commissioning request, the timing, scope, and focus of the review were relatively fixed. As researchers, we were limited to providing input that restricted the scope to peer-reviewed publications in order to maintain a workable review and to ensure the quality of included literature. Timelines were tight, both in the commissioning of the review and in its development that – with a due date of the final deliverable within 5 weeks of signing the contract – did not provide space to discuss the focus of the review and/or a potential broadening of the research questions. This led to important limitations that are discussed below.

The commissioning of the review placed a particular emphasis on formal ECEC (i.e. paid, non-parental care typically outside the parental home) and specifically on ECEC policy reforms since the introduction of the Barcelona targets (i.e. changes to ECEC policy rules relating to various aspects of policy design). The review was further focused explicitly on ECEC reforms related to five outcomes of interest to the European Commission: children's ECEC participation rates, mothers’ employment, working hours, earnings, and gender equality more broadly. In short, the commissioned review took the form of a ‘focused’ review, explicitly focused on publications studying the outcomes of ECEC policy reforms. As such, the contours of the desired evidence base were mostly pre-determined and took further shape through three methodological steps taken for the review.

The first step was the search for high-quality, (predominantly) peer-reviewed scientific publications. As no explicit definition was given of what constitutes a reform, and neither was a set of specific reforms specified, we had to rely on (analyses of) policy changes that were described as reforms. We thus selected publications based on search terms (searching in title, abstract and keywords) to limit the focus on ECEC reforms. Following a systematic search of six academic databases (Web of Science, Scopus, Ebsco, Worldcat, Jstor, and Stockholm University library), we downloaded 8031 articles.

The second step constituted the inclusion and exclusion criteria. Duplicate studies (3022) were removed. Studies were then included if they were (a) empirical; (b) published between 2000 and 2021 (in the years just before the implementation of the original Barcelona targets and nearly two decades following implementation); (c) examined the outcomes of policy reform that took place in an EU member state, Norway or Switzerland at either the national or regional level; (d) focused on outcomes related to the reform that pertained to women's/maternal employment, gender equality or parents’ use of/children's enrolment in childcare/ECEC. Based on these five criteria, we were left with a final sample of 44 studies (see Table 1). This number is low because the inclusion criterion of a focus on a policy reform from the Commission was strict: a reform entailed changes to the policy and/or rules regarding ECEC or childcare. As studies were selected based on keywords and the description of the policy changes as ‘reforms’ by the studies’ authors, the included studies cover policy changes that range from complete overhauls of ECEC systems to gradual expansions. Moreover, if potentially relevant policy changes were not described as a ‘reform’ they were not included here (for instance: Zoch, 2020). A full list of the publications included in this review is included as an Appendix at the end of this article.

Table 1.

Publication overview at each stage of the election procedure.

Description	Number of publications excluded	Total publications remaining
Total number of publications found		8031
Duplicates excluded	3022	5009
Publications not meeting inclusion criteria	4965	44
Final number of publications	44

The third step entailed organising the analysis and presentation of the included studies around three aspects of ECEC policy design: accessibility, affordability and quality of ECEC services. This step reflected our choice, as authors, to focus on these three aspects of policy reforms based on our expertise and a priori identification of potentially relevant aspects of ECEC services that could be conceptually distinguished. As noted in the ECEC literature, of course accessibility, affordability and quality are often intertwined in reality (e.g. Yerkes and Javornik, 2019).

The resulting literature database and our analysis of it as commissioned by the European Commission helped shape European policymakers’ understanding of ECEC issues and what was relevant in the revision of the Barcelona targets. Our analysis produced at least two key insights. First, ECEC reforms to improve accessibility (including availability), affordability and quality have indeed been successful at increasing ECEC enrolment as well as increasing maternal employment and mothers’ working hours. However, the effect on maternal earnings is more mixed, depending on mother's socioeconomic status and the influence of heteronormative work- and care norms. Second, while the literature consistently shows that ECEC reforms to improve accessibility, affordability and quality positively impact ECEC enrolment and maternal employment, the picture becomes more multifaceted at the group-level, with several studies suggesting some socioeconomic groups benefit more from ECEC reforms than others. The focus of this short article is, however, not on what was learned from ECEC reform studies. Rather, we focus here on the blind spots revealed by this commissioned, focused literature review.

The blind spots: Critical reflections on the evidence base

Five blind spots were identified through an inductive, iterative analysis of the literature database and discussions amongst all four authors, starting by summarising which countries were covered, which reforms were implemented, and which outcomes were discussed in each study. These analyses and discussions, when placed against the broader ECEC literature, highlighted five themes.

Context-specificity

The first blind spot in this evidence base consisting solely of ECEC reform studies is an absence of context-specificity. The research design of reform studies explicitly seeks to exclude the influence of ‘confounding’ or contextual factors. Consequently, such studies generally focus on a single country (see Table 2), with only three focusing on multiple countries (of which one compared France and Germany; another Sweden and Belgium, and one study compared 18 countries. Reform studies often seek to demonstrate the existence of a causal effect and are less focused on explaining why the reform (or specific features of the reform) was (in)effective. Moreover, reform studies generally cannot show that the effectiveness of ECEC for increasing maternal employment, working hours and earnings, and how to increase effectiveness depends on its interplay with other institutional and structural conditions. A few notable examples studied the interplay at sub-national/regional levels. One example was found in Italy, where an ECEC reform was found to be more effective in regions that had greater labour demands (Carta and Rizzica, 2018). Another example was the study of the different contexts provided by (former) East and West Germany, for instance showing that an expansion of low-cost public childcare led to higher likelihood of mothers returning to employment after having their second child – but in Western Germany only, which was previously characterized by a stronger traditional breadwinner model (Zoch and Hondralis, 2017). Studying regional variation is relevant in itself, and an important strategy to test hypotheses about contextual interplay. Yet, the policy learning inherent to the EU policy making process also requires an understanding of the extent to/conditions under which findings can be extrapolated to other country contexts. To this end, more extensive country-level contextualization is needed to understand the interaction of various elements of policy design and/or policy domains and their outcomes (Yerkes et al., 2022).

Table 2.

Countries studied in the reviewed ECEC literature.

		Number of studies	Percentage
North-West Europe	Belgium	1	2
	Finland	1	2
	Germany	13	30
	Netherlands	2	5
	Norway	8	18
	Sweden	5	11
	Subtotal	30	68
Southern Europe	France	2	5
	Italy	1	2
	Spain	2	5
	Subtotal	5	11
Central Europe	Austria	2	5
	Switzerland	1	2
	Subtotal	3	6
Eastern Europe	Czechia	1	2
	Hungary	1	2
	Poland	1	2
	Subtotal	3	6
Multiple countries		3	7
Total:		44	100

How to reduce ECEC inequalities

The second blind spot identified is on how to reduce inequality in ECEC use and benefits. It is well-established that parents with higher levels of education and income are more likely to enrol their children in ECEC (Van Lancker, 2018), a point confirmed by several reform studies included in the database. What these studies are silent on, however, is the degree of cross-country inequality in ECEC use, which would be an indication that the degree of inequality is in part affected by the institutional context (see Table 3). Consequently, which aspects of ECEC policy lead to an increase or decrease in inequality in ECEC use remains unknown. The broader ECEC literature (i.e. non-reform studies) is, in contrast, more informative. A key lesson from the broader literature is that for ECEC to be effective at reducing inequality, it needs to be accessible, affordable and of sufficient quality (Yerkes and Javornik, 2019) – sometimes called the ‘childcare triangle’ (Gambaro et al., 2015). Yet, while the studies in the ECEC reform database all evaluated one or more reforms related to accessibility, affordability, or quality, the potential interplay between these different aspects of ECEC policy were not studied. As recently highlighted by Dobrotić (2022), reforms aimed at creating more inclusive leave policies in post-Yugoslavian countries led to greater systematic exclusion of some parents from ECEC rights. Additionally, non-reform ECEC studies show that merely increasing public expenditure on ECEC (spending that goes towards services, ECEC regulation, vouchers for parents or tax deductions) does not reduce inequality in ECEC use (Van Lancker, 2018). Rather, inequality in ECEC use is lower in countries that have public or publicly-subsidized supply of ECEC services, ECEC guarantees, lower out-of-pocket fees, and highly favourable parental perceptions of ECEC quality (Nieuwenhuis et al., 2021; Van Lancker and Ghysels, 2016).

Table 3.

Outcomes noted in the 44 studies.

Outcomes		Number of publications	Percentage
Commissioned outcomes	Childcare usage	24	54.5
	Maternal employment rates	29	65.9
	Maternal earnings	5	11.4
	Maternal employment hours	17	38.6
Other outcomes	Attitudes to maternal employment	2	4.5
	Human capital of children and mothers	5	11.4
	Household outcomes (income, division of labour)	3	6.8
	Macro outcomes (fertility rates, marketization of childcare)	2	4.5
	How to reduce inequality in ECEC use	2	4.5

Multiple studies have examined more than one type of outcomes; hence the total number of outcomes exceeds 44.

Temporality and long-term outcomes

The third (partial) blind spot relates to long-term outcomes, which are overlooked in two ways. First, Table 4 shows the length of the post-reform observation window of each study. Inherent to the research design, a substantial share of studies in the ECEC reform database examined the immediate and short-term effects of policy changes (i.e. changes visible within 1–3 years), or medium-term (4–6 years). Yet, enrolment in ECEC is beneficial to children's social and cognitive development later in life, with particularly significant benefits for children from disadvantaged backgrounds occurring years later (Gambaro et al., 2015). ECEC can also play a role in protecting children from the consequences of growing up in poverty (Leseman and Slot, 2014), including short-term consequences, such as food poverty, but also long-term consequences, such as learning disadvantages (Drange et al., 2016). The second way in which long-term outcomes represent a (partial) blind spot, is that only a few reform studies included in the database have examined how long it takes for implemented policy reforms to have an effect. This also pertains to the studies in Table 4 that had a longer post-reform time-window, as these too were generally silent on temporality and potentially longer-term effects. There is a reasonable expectation that the effectiveness of ECEC policies may change over time. In France, the reform of both the parental leave scheme and ECEC had ‘negligible effects in the short term’, but after three years, lower educated mothers used the extended parental leave rather than ECEC (De Muizon, 2020: 735). Similarly, in Germany, the educational inequality in ECEC usage and maternal employment resulting from ECEC reforms were not found immediately, but emerged over time (Stahl and Schober, 2018).

Table 4.

Observation window (years between the studied reform and the outcome data).

Number of years	Number of publications	Percentage
1–3 years	16	36.4
4–6 years	8	18.2
7–10 years	12	27.3
Above 10 years	8	18.2
Total	44	100

Macro-level outcomes

A fourth blind spot in the evidence base of ECEC reform studies is on macro-level outcomes of importance. Reform studies focus on individual-level outcomes (see Table 3), which facilitates causal inference but overlooks higher-order outcomes and thus the relationship between ECEC and important societal developments. For example, beyond individual advantages, ECEC provision helps to protect families against poverty (Förster and Verbist, 2012) and to reduce gendered earnings inequality both within as well as between couples (Nieuwenhuis et al., 2019).

Publication biases

A final blind spot emerges in the form of various publication biases. First, nearly all studies report that the ECEC reform was effective in achieving one or more outcomes. One of the few exceptions is a study on ECEC expansion in Austria, which reported that maternal earnings did not increase more in municipalities where ECEC accessibility expanded across time compared to municipalities where accessibility had not expanded (Kleven et al., 2024). Another exception is the study of a reform that capped ECEC fees in Sweden, which produced no effect on maternal employment (Lundin et al., 2008). Thus, while the database and the ensuing analysis presented here show which reforms have been effective, they generally remain silent on ineffective reforms.

The second form of publication bias is more specific, in the sense that the published reform studies focus more on some groups than others. Most notably, only a handful of studies examined fathers. The few studies that focused on fathers found little to no effect of ECEC reforms on fathers’ employment (Ravazzini, 2018; Vikman, 2013) or on fathers’ attitudes to maternal employment (Zoch and Schober, 2018). One exception is a study of a Dutch reform, where fathers with a youngest child aged 0–3-years-old reduced their employment hours by 0.5 h per week (Bettendorf et al., 2015). Consequently, the potential benefits of – and changes in behaviour in relation to – ECEC for multiple groups like fathers, but also those with less power and privilege (see Dobrotić, 2022), remain unknown.

The third form of publication bias is that considerable geographical differences in where reforms are being studied exist. Central and Eastern European countries are underrepresented in reform studies (Dobrotić, 2022), which is also clear from Table 2. In contrast, Germany, Sweden and Norway are overrepresented. This geographical bias in the database may reflect an absence of ECEC reforms taking place, an absence of evaluations of these reforms, an absence of results published in English, or any combination of these factors. In short, the representativeness of the literature is limited, and countries known for extensive provision of ECEC are overrepresented. This means that the evidence base for the revision of the Barcelona targets might be the weakest for those countries that might be furthest away from achieving them.

Discussion: Consequences of five blind spots in reform studies of ECEC policy

We have critically reflected on the development and analysis of a focused review commissioned by the EU, which was part of the evidence base for the revision of the Barcelona ECEC targets. This critical reflection highlighted (a) the context-specificity of most reform studies; (b) the limited focus on how to reduce inequality in the use and benefits of ECEC; (c) a focus on short-term outcomes; (d) a focus on individual-level rather than macro-level outcomes; and (e) various forms of publication bias. The latter relates to the absence of knowledge on ineffective reforms, the very limited focus on fathers, and the paradox that the countries that might benefit the most from the Council Recommendation and the Barcelona targets, were least likely to be included in the underlying evidence base.

Given the time constraints, we accepted the research questions in the commissioning of the review. Alternatives were limited. By accepting the commission, we at least were in a position to make note of the blind spots in the review. Due to the strict inclusion criteria set by the Commission and the tight timelines, we may have erroneously excluded relevant studies. Of particular note here is that we could only include studies that used the term ‘reform’ to describe the policy changes they analyse. The future development of a database that attempts to systematically document all (ECEC) policy reforms (according to a set of pre-determined criteria) would reduce the need to sample from published studies, and substantially improve the representativeness of policy learning endeavours. Yet, as the nature of the blind spots is closely linked to methodologies common in reform evaluations, these blind spots seem relevant to a larger set of studies. Nevertheless, we would be remiss not to acknowledge our own role in perpetuating the blind spots evident in the database we created. At the time, we did not question the scope, thereby helping to produce and disseminate the evidence base. We simultaneously attempted to disrupt the evidence base (Bacchi, 2009) by clearly noting the limitations of the review, listing the blind spots and discussing these points in expert meetings preceding the revision of the targets by the European Commission.

It should be clear that the evidence base used by the Commission to develop the revised Barcelona targets was much broader than the focused review of reform studies alone. Yet, given how explicitly the commissioning request focused solely on reform studies, it is relevant to discuss what the five blind spots in ECEC reform studies could imply for ECEC policymaking. A common factor of all the blind spots identified here is that they reduce the scope of evidence and possible policy options. Consequently, specific evidence/policy options are prioritised over others (Bacchi, 2009). We highlight three examples.

First, in the original Barcelona targets, ECEC was predominantly seen as a labour market instrument. With the adoption of the European Pillar of Social Rights, the benefits of ECEC for children's development came more into focus. Yet, this perspective remained out of scope for the review given how the project was commissioned. Consequently, the focused review did not – or at most barely – represent differences in perspectives that mothers, fathers, children, workers and/or employers may have. Instead, ECEC was predominantly represented to be an economic issue, with a focus on labour market-oriented solutions.

Second, as the evidence base is based on policy reforms introduced since the implementation of the original Barcelona targets, it should be acknowledged that these reforms themselves were shaped by the 2002 Barcelona targets. This creates a form of path dependency: prior policy goals shaped ECEC reforms, which are now used to inform future policy goals. To the extent that the European Parliament's invitation to the Commission to revise the Barcelona targets reflects the intention to achieve new policy goals, this form of path dependency could slow down institutional change (Streeck and Thelen, 2005). Furthermore, as this path dependency risks excluding new knowledge, and the evidence base is weakest for those countries that might benefit the most from ECEC reforms, this self-reinforcing process might hamper effective policymaking more broadly (Head, 2010).

Third, it was noted how the focus on reform studies tends to isolate the consequences of changes in one specific policy, or even just one aspect of ECEC policy design. This is a strength of these research designs. But the absence of attention for the interplay of policy design can make it difficult to infer generalisations about the effectiveness of ECEC reforms. The silence of most studies in the database on the importance of the interplay of ECEC policy design (Dobrotić, 2022; Yerkes and Javornik, 2019) as well as its interplay with the broader institutional and structural context (Yerkes et al., 2022) can limit what policymakers deem to be relevant in the revision of the Barcelona targets.

Academics have multiple strategies at their disposal to counter or reduce the sources of bias resulting from blind spots, by fostering the methodological pluralism in social policy research (Yerkes et al., 2022). Rather than debating which methodology is supposedly ‘best’, a more fruitful scholarly debate focuses on how findings from different methodologies can complement each other. An evidence base centred on any single methodology, irrespective of its rigor and transparency, risks that biases and blind spots remain. Therefore, an appreciation that a variety of methodologies can highlight different aspects of ECEC policy outcomes, will strengthen academic communities to develop a more complete evidence base relevant for policymaking (Yerkes et al., 2022). For instance, insights from the comparative welfare state literature have been instrumental in the identification of the blind spots presented here, and some of the limitations represented by these blind spots are –at least partially– covered in the comparative literature. However, policymakers would also do well to be explicitly aware of the blind spots present in evidence bases, how these are influenced by their own decisions, and what actions they can take to reduce such biases. The EU has a historically comparative tradition of social policymaking, including the open method of coordination and the performance of peer reviews, whereby countries present and compare their policy approaches to a certain issue. Such comparative approaches naturally bring a diversity of perspectives. We recommend policymakers take a broad perspective when developing an evidence base. ‘Broad’ is to be understood here from the perspective of the policy goal(s). Related to this, we also recommend that academics (and potentially other societal stakeholders) become involved as early as possible in the process of creating an evidence base, whilst recognising that earlier involvement may not always be possible. The design of the focused review described here was commissioned with a narrow formulation of aims and tasks. This produced a review that has strong evidence on the outcomes of ECEC policy reforms, yet inherently came with a number of blind spots. Involvement of academics at an earlier stage (e.g. formulating the aims of the knowledge base) may avoid some of these blind spots, and as such increase the range of insights represented in the evidence base available to policymakers.