Abstract
In this article, the authors provide an update on what has happened over recent months with the Organisation for Economic Co-operation and Development’s proposal for an International Early Learning Study, and review responses to the proposed International Early Learning Study, including the concerns that have been raised about this new venture in international testing. The authors call on the Organisation for Economic Co-operation and Development and its member-state governments to enter into open discussion with the wider early childhood community about the future direction of comparative work on early childhood education and care.
In early autumn 2016, an article appeared in Contemporary Issues in Early Childhood (Moss et al., 2016) about a highly consequential project that was about to be implemented by the Organisation for Economic Co-operation and Development (OECD): the International Early Learning Study (IELS), a cross-national assessment of early learning outcomes involving the testing of five-year-old children in participating countries. The article reported how this project had been evolving since 2012, in a partnership between the OECD and some of its member states. At the time of writing, the OECD was assessing tenders for the position of ‘international contractor’ to take the lead in designing, developing and piloting this study, which was to measure a range of cognitive, social and emotional skills among samples of young children, the results from which would be contextualised with information on three areas: early childhood education and care (ECEC) experiences, the home learning environment and the child’s individual characteristics. The aim of the article was both to inform, since the authors believed that most members of the early childhood community were unaware of this project, and to raise a number of concerns about the proposed project. The aim of this article is to tell readers what has happened since, and to review the concerns that have been raised, both in the original article and in subsequent publications.
What happened next: The OECD
The OECD has forged ahead with its project since the autumn of 2016. It has selected a consortium to act as an international contractor for the IELS, consisting of the Australian Council for Educational Research, 1 the International Association for the Evaluation of Educational Achievement, 2 and cApStAn, 3 the last named being responsible for translation, domain instruments and questionnaires. It is also looking to recruit three to six countries to participate in the first stage of the IELS. At the time of writing this article, no country had publicly committed to participate, although we assume that the OECD is confident that it will receive the necessary support, else it would not have appointed an international contractor. We also know that some countries have said they will not participate, including Belgium, Canada, Denmark, Germany, Norway and Sweden. In the case of Germany, to give just one example, the government’s decision not to participate in the IELS was influenced by a statement written by a coalition of national organisations that includes service providers, trade unions, parents’ organisations and researchers; this brings together critical arguments against Germany’s participation in the IELS, building on the lack of recognition of children’s rights, diversity and sociocultural contextualisation of early childhood practices in the OECD approach (Urban and Swadener, 2016).
Regardless of the uncertainty about participating countries, the OECD has announced a timeline for the project. Following development work and a field trial in 2017, the main study will take place in 2018–2019, with a report in 2020 – or that at least is the plan.
Rather belatedly, the OECD has set up an extensive website, which provides a substantial amount of information about the project; 4 an earlier one-page website contained little information. The current website provides information about the background to the study and the study itself, and responses to frequently asked questions about the study. There is also a 24-page brochure called ‘Early learning matters’. We discover from the new website that the study has been renamed, now being the International Early Learning and Child Well-being Study (our emphasis), although the acronym remains IELS and only cursory reference to well-being is made on the website.
There is also another change from the original tender document. Whereas that referred to the implementation of the project starting with a design, development and piloting stage, there is now no reference to a pilot stage. Instead, following a short period of design and development, with an accompanying field trial, there will be what the new website refers to as a ‘main study’, across three to six countries, being undertaken in 2018 (‘Northern hemisphere’) and 2019 (‘Southern hemisphere’), followed by ‘quality control and analysis’ and a ‘report’ in 2019–2020. This suggests that the OECD now sees the initial round of assessments (in 2018–2019) as the first of a new agreed study, rather than as an exploration of feasibility prior to deciding whether or not to proceed further.
The website spells out the purposes of the IELS as being to:
Provide robust empirical data on children’s early learning through a broad scope of domains that comprise cognitive and social and emotional development.
Identify factors that foster and hinder children’s early learning, both at home and in early childhood education programmes.
Provide findings that will allow parents and caregivers to learn about interactions and learning activities that are most conductive to child development.
Inform early childhood education centres and schools about skill levels of children at this age as well as contextual factors related to them that they could use to make more informed decisions about curriculums and pedagogical [sic].
Provide researchers and educators in the field of early education with valid and comparable information on children’s early learning and characteristics obtained from a range of sources and accompanied by a broad scope of contextual variables.
The IELS, according to the website, will produce benefits for children, families, ECEC centres and schools, as well as for countries as a whole, including improving the early learning environment at home and in ECEC; improving parenting programmes and other support for parents; identifying ECEC settings that are most conducive to early learning; improving understanding of children’s needs in early learning; assessing children’s early learning outcomes through a wide range of critical domains; identifying key factors that drive (sic) or hinder early learning; and learning from each other by developing common frameworks and benchmarks.
We are also supplied with some details of how the IELS will be carried out. Four early learning ‘domains’ (emerging literacy, emerging numeracy, self-regulation, and empathy and trust) will be assessed using tablets, with each domain taking ‘approximately 15 minutes’ and spread over two days. In addition, ‘indirect assessment of children’s skills will be obtained from parents and staff through written and online questionnaires’. Additional information will be collected from the study ‘administrators’ [sic] observations’. The parents’ questionnaire will ‘collect information on children’s socio-demographic characteristics, parental background, home learning environment, early childhood education participation and community characteristics. The staff questionnaire will be used to gather basic information on staff and, where possible, the early childhood education participation of the child’.
The children’s perspectives will also be sought by asking if they liked the assessment activity, its content and different aspects. These debriefing sessions will be used to ensure children’s well-being during the assessment but also to provide valuable feedback about the assessment material and procedures. In addition, children will be asked about their favourite learning activities in different settings.
There is, however, no indication that children’s consent to participate in the IELS will be sought.
What happened next: Concerns about the IELS
In the last six months, a number of publications have appeared commenting on the proposed IELS. The initial article in Contemporary Issues in Early Childhood has been followed by two further pieces, both from New Zealand (Carr et al., 2016; Mackey et al., 2016), while a statement, ‘Democratic accountability and contextualised systemic evaluation’, has been issued on behalf of the Reconceptualising Early Childhood Education (RECE) group, and published in International Critical Childhood Policy Studies (Urban and Swadener, 2016). A number of articles and statements have appeared in national magazines, including in Belgium, Canada, France, Germany and the UK. These publications raise a wide range of concerns about the OECD’s project and its possible consequences, which we summarise below.
Technical concerns about standardised assessments applied across countries
There is, notes Gorur (2014: 59), a ‘vast literature that critiques aspects of [the Programme for International Student Assessment’s (PISA’s)] methodology’. The RECE statement cites ‘ample evidence of the low reliability and validity of standardized tests of children, especially in contexts of large-scale comparison’ (Urban and Swadener, 2016: 1). None of this literature and evidence is referred to, let alone addressed; nor is there any discussion of issues raised by computer-based assessment of five-year-olds, with the website simply asserting that ‘even children with no previous experience with tablets have no problems in understanding and following the procedures’. 5
The political reduced to the technical
The authors of the first Contemporary Issues in Early Childhood article (Moss et al., 2016: 346) argue that education is a political practice, raising profound political questions, each with alternative answers from which political choices need to be made. The OECD, however, completely ignores this political dimension. Instead, it treats early childhood education and the proposed study as if they are purely technical practices, adopting a vision of comparative education as a technical process modelled on industrial benchmarking, in which the outcomes have been determined, and the aim is simply to engage in the global war for talent by learning enough from [our] competitors to beat them at their own game. (Auld and Morris, 2016: 226)
The universal approach of the IELS cannot welcome or even accommodate the diversity of context, culture and purpose found in ECEC
The IELS, it is argued, is a blunt instrument, seeking to reduce the rich diversity and complexity of ECEC to a common standard, measure and outcome. Such contextual information as the OECD proposes to collect in the IELS is limited and narrow; among the many contextual omissions is the absence of any reference to inequality, which is known to be highly consequential (Moss et al., 2016). This is consistent with PISA, which has been criticised for ‘emphasising policy effects over history and structural factors’, and with the OECD’s overall education policy work, which ‘depends, to a significant extent, on stressing the importance of policy factors over the effects of cultural and social context’ (Sellar and Lingard, 2013: 721, 723).
In these circumstances, the RECE statement argues that the IELS findings will be ‘largely meaningless due to their disconnect with and disrespect for diverse, locally embedded approaches to early childhood education and care’, and that the focus on the early learning function of ECEC ignores the important contribution of ECEC services to a wide range of ‘societal and political challenges of our time’ (Urban and Swadener, 2016: 7, 8). At a national level, both articles from New Zealand doubt the ability of the IELS to do justice to the distinctive sociocultural approach at the heart of that country’s innovative curriculum, ‘shift[ing] the emphasis away from pedagogies which focus on that which is meaningful and relevant in children’s lives and their learning, to an emphasis on achieving assessment results that fit a universal framework’ (Mackey et al., 2016: 448). Margaret Carr and her colleagues (2016: 450) quote a fellow countryman, who contends that: To fairly and truly judge what a person can do, you need to know how the talent (skill, knowledge) you are assessing is situated in – placed within – the lived social practices of the person as well as his or her interpretations of those practices. … many a standardized test can be perfectly ‘scientific’ and useless at the same time; in a worst case scenario, it can be disastrous. (Gee, 2007: 364)
The dangers of soft power
Through its powerful ‘human technologies’, its creation of ‘epistemic communities of policy analysts, bureaucrats and politicians within the Organisation and in member countries’ (Sellar and Lingard, 2013: 712), and its global reach, the OECD exercises enormous ‘soft power’. Yet the responsibilities and dangers of this unaccountable power are not acknowledged. In particular, through the effects of league tables comparing national performance and the prospect (as the IELS website puts it) of providing ‘countries with a common language and framework’, there is the worrying prospect of ‘a growing standardisation and narrowing of early childhood education, as the IELS tail increasingly wags the early childhood dog’ (Moss et al., 2016: 349). Once again, the experience of PISA provides a warning: the simplest way to improve PISA scores is for nations to align their curricula more closely to what is measured by PISA … If countries do this and improve their scores, we will enter into a closed and self-fulfilling system in which nations teach according to test requirements and better scores create the illusion of improvement. (Morris, 2016: 26)
Developing this theme, New Zealand academics fear that the IELS will lead to ‘teaching to the OECD measures’ and a consequent ‘pedagogy of compliance’, as governments are tempted ‘to call on the apparent precision of numbers to prescribe and measure context-free and curriculum-free internationally developed and validated outcomes over time’. This would be to the detriment of ‘the Aotearoa New Zealand early childhood sociocultural and bicultural curriculum … [which] has established a set of priorities for teaching and learning that are different from most of the other OECD countries’ (Carr et al., 2016: 451, 453).
A naive belief in policy learning
Central to the rationale of the IELS is the notion of policy learning, with countries that perform less well on the standardised assessments being able to identify and apply the strengths of better-performing countries by adopting ‘a common language and framework’. But Robin Alexander, a leading figure in comparative education, has dismissed the feasibility of such a process, concluding that: national education systems are embedded in national culture … [so that] no educational policy or practice can be properly understood except by reference to the web of inherited ideas and values, habits and customs, institutions and world views, that make one country distinct from another. (Alexander, 2012: 5)
Meanwhile, Paul Morris has commented on the naivety of the OECD’s implied model of enlightened policymakers objectively and rationally applying lessons from other countries, since what is more apparent in practice is the wholly unsurprising tendency for policymakers to view such comparative data on pupil performance as an expedient resource, which serves a primarily symbolic role in the theatre of politics and provides a massive source of evidence, from which they can hunt for correlations to legitimize their own ideological preferences. (Morris, 2016: 11)
The potential for involvement by corporate interests
PISA, the IELS’s precursor, was initially undertaken by international consortia of professional organisations. This is now changing. In 2013, McGraw-Hill Education, the giant textbook and testing company, was awarded the contract for administering tests in the USA, while in 2014, the OECD awarded the contract for developing the frameworks for PISA 2018 to Pearson, the largest education company in the world (Unwin and Yandell, 2016). The RECE statement expresses concern that a similar process may occur with the IELS, opening the door for international for-profit corporations to extend their reach into early childhood: ‘if not by intention then by design, the current international initiatives for standardised assessment contribute to opening public education sectors to corporate profit interests and to channelling scarce resources from the public sphere to private, corporate profit’ (Urban and Swadener, 2016: 5).
Secrecy and dismissal of concerns
A final concern arises from the whole experience of trying to engage with the OECD on the subject of the IELS. The evolution of the IELS, going back to 2012, has been shrouded in secrecy; few people in the early childhood community knew of its existence, even when the project was on the verge of implementation. The OECD itself says that informing and consulting more widely about this project is the responsibility of member-state governments, yet few seem to have done so. Nor can the OECD entirely wash its hands of the matter. It now has a very detailed website, put up in February 2017. Could it not have done this much earlier, including, for example, notes of meetings with government representatives where the study was under discussion? The early childhood community is now faced with what is, in effect, a fait accompli – comments might be welcome on the details of the IELS, but it is too late to question the very concept of a cross-national standardised assessment of children’s performance on selected outcomes.
Furthermore, rather than ‘opening for debate and contestation’, the OECD has chosen to dismiss the concerns that have been expressed about the IELS in recent months by simply ignoring them. The OECD was invited to respond to the original article in Contemporary Issues in Early Childhood – but so far no response has been published. It could have chosen to respond on its new website – but has not done so. It could even have included critical articles and statements in the ‘publications’ link on its website – but, again, it has chosen not to do so, confining the selected list of publications to those from the OECD itself. This adds up to an apparent lack of interest on the part of the OECD, bordering on disdain, about the views of the wider early childhood community or the criticisms and worries voiced by some members of that community.
This behaviour reflects a general attitude on the part of the OECD towards differing and critical perspectives, highlighted in the first Contemporary Issues in Early Childhood article: The Organisation adopts a particular paradigmatic position which might be described as hyper-positivistic … the OECD is free to choose its position. However, it should be aware that it has made a choice and taken a particular perspective. It should also be aware that there are other choices and other perspectives. Yet on both counts it shows a total lack of self-awareness. (Moss et al., 2016: 346)
Seemingly unaware of its own paradigmatic position, the OECD seems to be similarly unaware of those working in other paradigmatic positions and of how ‘[o]ver the past 25 years reconceptualist scholars have contributed to a rapidly growing body of research and knowledge that offer alternative – postcolonial, critical, feminist, indigenous, transdisciplinary – understandings of what it means to educate and care for young children’ (Urban and Swadener, 2016: 2). Such research and knowledge is rendered invisible by the OECD, its existence not even acknowledged.
In conclusion
At a meeting with the OECD to discuss the IELS, we were asked if our criticisms of the IELS reflected an outright rejection of a quantitative approach. That is not so. We believe that there is a time and a place for quantitative methods, and have indeed used them ourselves; there are many things that need counting.
But we would add an important qualification: the findings produced by quantitative methods must always be treated critically and subjected to careful questioning and interpretation. Numbers, by themselves, tell us little, yet once set into a table, they readily acquire totemic status, being treated as if they were a true representation of reality. This is always the danger of projects such as PISA and the IELS – that what might be a useful source of documentation, to be carefully discussed and critically reflected upon alongside other forms of documentation, is reduced to a crude and unexamined tool of management. There are limits, also, to what quantitative methods can provide insight into; for example, we are not convinced that they can give sufficient insight into context and culture, the ‘web of inherited ideas and values, habits and customs, institutions and world views, that make one country distinct from another’, to which Alexander refers.
Our overriding concern, therefore, remains that the IELS will end up, in the words of Loris Malaguzzi, as ‘a ridiculous simplification of knowledge, and a robbing of meaning from individual histories’ (cited in Cagliari et al., 2016: 378). Or, as the RECE statement puts it: Instead of careful, culturally and contextually appropriate consideration of the achievements of early childhood systems in diverse countries, and of systemic evaluation of the actual outcomes for children, families and society, IELS appears to adopt a strategy that favours largely decontextualised comparison and measurement of narrowly defined predetermined outcomes. It is our concern that such an approach will not provide necessary or meaningful information for decision makers and early childhood leaders in participating countries and beyond. What it will do is draw early childhood education firmly into a global framework of standardised assessment across all tiers of the education system, from early childhood to higher education. (Urban and Swadener, 2016: 4)
The IELS, for us, is not just a concern; it is a wasted opportunity. We want to see further comparative studies of ECEC, but studies that adopt an approach which is respectful of diversity, welcoming of complexity, inclusive of the field’s multiple perspectives and provoking of thought. With Starting Strong I and Starting Strong II (Organisation for Economic Co-operation and Development, 2001, 2006), a landmark study of ECEC in 20 countries, using case-study methods that were sensitive to the diversity and complexity of systems and pedagogies, the OECD provided an example of just such an approach.
We think that it is probably impossible to halt the OECD’s current insistence on an initial comparative study of the performance of young children using standardised measures, albeit involving only a small number of countries. But it is not too late to think about what should follow after the conclusion of the initial study. With this in mind, we call on the early childhood community to develop and sustain a critical engagement with the IELS; and we call upon the OECD and its member-state governments to enter into open discussion with the early childhood community about the future direction of comparative work on ECEC, and to do so now, rather than in 2020.
Footnotes
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.