Abstract
In this paper we explore datafication through a specific kind of evaluative practice in Swedish early childhood education and care (ECEC): the creation of quality reports by individual preschools. A theoretical framework is developed based on Dahler-Larsen's descriptions of three ‘evaluation imaginaries’ (the modern, reflexive modernity and audit society), with elaboration of their respective characteristics and complementary notions from other sources. We then apply the framework, and a qualitative text analytical approach, to analyse eight quality reports from diverse Swedish preschools, purposefully selected to represent preschools that vary in terms of ownership, geographical location, size and pedagogical profile. We find that the reflexive modernity imaginary is most clearly influential in the ECEC quality reports, but there are emerging indications of the audit society imaginary's influence. We also find variations in prioritised focus in the reports among ECEC providers, underscoring the need for further exploration. The study calls for future research to delve into provider-specific nuances, expand the investigation to other kinds of evaluations and scrutinise the broader implications for ECEC operations and practices.
Introduction
A major feature of contemporary society is the extent and use of data, indicators and metrics to value, appraise and determine the quality of organisations, projects and programmes. In research literature this has been conceptualised as ‘governing by numbers’ (Grek, 2009; Rose, 1991), ‘databased governance’ (Roberts-Holmes and Bradbury, 2016) and ‘datafication’ (Bradbury, 2019; Hansen, 2015). Datafication refers to the process of making a phenomenon ‘in a quantified format so it can be tabulated and analyzed’ (Mayer-Schönberger and Cukier, 2013: 78). However, these processes are not neutral, as they have consequences for practices, values and subjectivities (Bradbury and Roberts-Holmes, 2017). Education is a particularly important domain for the study of data and its consequences. This is because there are diverse forms of datafication in education, reflecting the scale and diversity of education systems and practices, with correspondingly high potential to effect people, practices and our understanding of what education is and ought to be (Jarke and Breiter, 2019; Lingard and Sellar, 2013; Williamson et al., 2020).
While datafication and its consequences have gained increased attention in the scholarly literature in relation to schools and higher education, it remains understudied in the field of ECEC (Jarke and Breiter, 2019; Siippainen et al., 2023). However, several scholars have provided vivid examples of increasing datafication, use of standardised assessments, tests and intensification of numerical measurements and assessments permeating ECEC practices in England (Albin-Clark, 2021; Basford and Bath, 2014; Bradbury, 2019; Bradbury and Roberts-Holmes, 2017; Roberts-Holmes and Bradbury, 2016). This contrasts with important elements of a ‘Nordic model’ of ECEC (Trætteberg et al., 2023), rooted in a social democratic welfare model (Esping-Andersen, 1996). Nordic ECEC documentation practices have been described as visualising learning and learning opportunities, rather than assessing learning. According to Urban et al. (2023: 10), the ‘Nordic approach to evaluation and assessment’ in ECEC is based on ‘shared values and principles such as well-being, child-centredness, play, learning, professionalism, and reducing inequalities’. In Sweden, documentation and assessment practices have been described as focusing on children's learning through what is being offered rather than abilities, knowledge and behaviour of individual children (Alasuutari et al., 2014; Åsén and Vallberg Roth, 2012; Bjervås, 2011; Johansson, 2016; Lee-Hammond and Bjervås, 2021; Urban et al., 2023).
The cited studies indicate that datafication may not be prevalent in Swedish ECEC settings, if evaluations and assessments are not focusing on easily measured and quantified aspects, in accordance with the ‘Nordic model’. However, Swedish preschool teachers interviewed in 2011 and 2012 by Johansson (2016) expressed ambivalence towards assessment aspects of their work. They reportedly welcomed ‘An increased focus on children's learning and on mastery of specific assessment formats and discourses … as signs of being professional’ that increased their legitimacy. However, they also expressed perceptions that ‘the same features seem to conflict with the values and discourses that constitute preschool teachers’ professional identity’. Further, recent marketisation of the sector may have spurred development towards more measurements and ratings of various sorts, as reportedly observed globally on other education levels (Busch, 2017).
This suggests needs for more examination of whether, and if so how, datafication is manifested in Swedish ECEC settings. A potentially valuable notion for addressing these phenomena is an ‘imaginary’, as ‘not a set of ideas; rather it is what enables, through making sense of, the practices of a society’ (Taylor, 2002: 91). This can facilitate the visualisation and problematisation of contingent and contextual settings in which practices occur, in this case assessment and evaluation practices.
The increasing attention to various assessment practices, and their results, on a societal level has led to recognition of a so-called ‘evaluation society’ (Dahler-Larsen, 2012) or ‘audit society’ (Power, 1999). Thus, evaluation can be considered an umbrella concept for several activities involved in and associated with ‘systematic procedures, values, utilization and view of the evaluand’, such as auditing, accreditation and quality assurance (Dahler-Larsen, 2012: 11), or one aspect of ‘datafication’ (Williamson et al., 2020). A key problem associated with datafication is that evaluations are supposed to provide ‘objective’ information, but they can never be neutral as they are inevitably based on value judgements. What is rendered valuable and judged as good, poor, successful or unsuccessful is contested and context-specific, reflecting normative beliefs and values (Schwandt, 2015). Hence, in the modern ‘evaluation society’, various ‘evaluation imaginaries’ circulate that express what evaluation is and ought to be (Dahler-Larsen, 2014). Such evaluations also express what different practices (ECEC) are and ought to be in relation to the judgments being made.
The aim of the study is to provide information about manifestations of datafication in Swedish ECEC. For this we focus on a specific set of important elements; the quality reports (QRs) generated by individual preschools. Our study is guided by two research questions. First, How and on what basis are ECEC practices evaluated in the QRs? Second, How do evaluation imaginaries become manifested in QRs?
Contextualising quality reports in Swedish ECEC
In Sweden, all children have the right to attend preschool and be provided a place after the age of one. Since the mid 1970s when childcare became an explicit municipal responsibility, enrolment rates have increased, in line with political ambitions, and now cover 86% of 1- to 5-year-old children and 96% of 4- to 5-year-olds (NAE, 2023). ECEC is heavily subsidised through taxes, and only relatively low fees are allowed. The 290 Swedish municipalities can provide ECEC service themselves, or through private actors, and over time the share of privately operated preschools has increased (Trætteberg et al., 2023; Westberg and Larsson, 2020). Today, about 20% of the children attend a privately operated but publicly funded preschool, and local preschool quasi-markets have evolved (Carlbaum et al., 2023).
As in the other Nordic countries, ECEC assessment and evaluation regimes have strengthened in Sweden in recent decades (Alexiadou et al., 2022; Karila, 2012). The most recent curriculum emphasises requirements for monitoring, evaluation and development, with a stronger focus on quality work and assessment (NAE, 2018). The steering documents also stipulate that ECEC should be monitored and evaluated on several levels, including national, local, individual preschool and work-team levels. At the preschool level the results are often presented in the form of an annual written quality report. The QRs are thus results of evaluative practices conducted within a broader infrastructure embedded in a national legislative framework for systematic quality work in ECEC (Håkansson, 2019).
However, as evaluation is not a neutral activity, the ‘objectivity’, utility and functions of the QRs may substantially vary, depending on the motives and perspectives of both the producers and readers. They may be helpful for guiding the systematic quality work of specific preschool units. They could also contribute to the accountability of ECEC organisers (municipal or private) and/or marketing to parents and potential customers. Thus, in a marketised setting such as Swedish ECEC, documents such as QRs might foster performativity ideals in preschools as they need to appeal to potential customers. As various kinds of actors provide ECEC services, there are likely to be correspondingly diverse motivations related to evaluative practices. For example, a for-profit ECEC provider may have incentives to use resources to meet measurable quality criteria and display the results, while ignoring unobservable qualities that are difficult to measure. In contrast, employees and managers of nonprofit organisations such as personnel-run cooperatives may be devoted to the service mission instead of profit-making, resulting in other types of evaluative practices (Roomkin and Weisbrod, 1999). Thus, studying QRs can contribute to expressed needs to critically examine the quality concept, as well as the associated assessment and documentation procedures, from multiple perspectives (Vallberg Roth, 2015).
Approaching quality reports through evaluation imaginaries
Our theoretical approach is inspired by the notions of ‘the evaluation society’ generally, and evaluation imaginaries in particular (Dahler-Larsen, 2012), emanating from the work by Charles Taylor. His understanding of the term ‘social imaginary’ encompasses something ‘broader and deeper than the intellectual schemes people may entertain when they think about social reality in a disengaged mode’ (Taylor, 2004: 23). A social imaginary is thus ‘shared by large group of people’ and refers to ‘the way ordinary people “imagine” their social surroundings’ (Taylor, 2004: 23–24).
Similarly to Dahler-Larsen (2012), Schwandt (2009) has used Taylor's (2004) imaginaries concept in explorations of evaluation in contemporary society. Schwandt relates Taylor's social imaginary to what he calls a ‘western evaluation imaginary’. He regards the common understanding of an imaginary as moral-political: ‘moral in that it reflects ideas of what individuals and societies should do, not simply what they are likely to do, and political in that it concerns ways in which societies make decisions and govern themselves … It is by virtue of the intersubjective agreements we have in place in the imaginary, so to speak, that evaluation has social purpose and meaning and evaluation knowledge has currency’ (Schwandt, 2009: 22)
Dahler-Larsen's (2012) notion of evaluation imaginaries builds on these concepts and distinguishes between ‘modernity’, ‘reflexive modernity’ and ‘audit society’ evaluation imaginaries (described in the following section). Few authors have attempted to apply Dahler-Larsen's approach to empirical cases, possibly at least partly due to its relatively high level of abstraction, but for a notable exception see a study by Kunseler and Vasileiadou (2016) on environmental policy on two of the imaginaries. Here we intend to concretise Dahler-Larsen's (2012) evaluation imaginaries to assist critical empirical exploration of the datafication of Swedish ECEC.
Three evaluation imaginaries
Dahler-Larsen (2012) argues that in modern times the idea of evaluation has become such a mainstay of societal organisation that speaking of an evaluation society is warranted. However, as modern society changes, different evaluation imaginaries emerge. In this section we describe characteristics of the three mentioned imaginaries; modernity, reflexive modernity and audit society, presented by Dahler-Larsen (2012).
The principles of ‘modern’ society are rooted in Enlightenment ideals, particularly belief in progress guided by rationality and autonomy, rather than tradition, prejudice and religion (Dahler-Larsen, 2012). Hence, society should govern itself rather than being governed by transcendental forces, and autonomous, rational agents should analyse, interrogate and question the functions of society in the name of progress. These characteristics are particularly important in the modernity evaluation imaginary. The evaluator is imagined as a disengaged subject who can deal with the world instrumentally and the evaluand (the object of the evaluation) can be ‘divided and decomposed into inputs, activities, programmes, criteria, and outcomes so that it can be managed and reorganised’ (Dahler-Larsen, 2012: 101). The modernity evaluation imaginary's overall modus operandi is systematic investigation to rationally guide societal progress. This imaginary is associated with specific assumptions about what evaluation is, its consequences and what it can achieve. Important elements include objective (analytically value-neutral) evaluation of evaluands, use of evaluation methods related to the positivist paradigm and application of the results in rational decision-making. The evaluations are typically ad hoc, large-scale and intended to provide generalised knowledge that can be beneficial for progress of the entire society (Vedung, 2010).
Reflexive modernity is mainly a response to its predecessor, modernity. As optimism about progress-oriented modernity has faded, and undesirable side-effects of previous achievements have become more apparent, questions have arisen about what rational, disengaged and value-neutral evaluations can actually achieve, and new evaluation approaches have emerged (Dahler-Larsen, 2012). Large-scale ad hoc evaluations have been pushed aside for smaller evaluations that are more closely connected to specific organisations (Leeuw and Furubo, 2008). In addition, evaluations that embrace multiple perspectives and stakeholder involvement, with no claims of generalisability, have been increasingly widely used at the expense of ‘value-neutral evaluations’ with objective claims. Belief that democratic ideals should be realised in evaluation methods by involving stakeholders in determination of value is a central element of this imaginary. These evaluation activities also provide challenges to reflexive modernity's evaluation imaginary (Dahler-Larsen, 2012). Evaluations in this imaginary lack the ability to reduce complexity, which can be perceived as a hindrance to utilisation of the results in decision-making. The constant reflection associated with reflexive modernity's evaluation imaginary may also include effects such as ritualistic use of evaluations, that is, evaluations becoming routine elements of organisations’ activities, with no real impact on development or decision-making, and consequent lack of enthusiasm for evaluation activities.
Like the reflexive modernity evaluation imaginary, the audit society's evaluation imaginary is a reaction to previous imaginaries. The lack of complexity reduction and generalisability, as well as the ritualistic effects of reflexive modernity's evaluation imaginary, have engendered new beliefs about what evaluation can and should do. In the words of Dahler-Larsen (2012: 169), ‘Audit society has lost the courage and characteristics of progress-oriented modernity and the flexibility and curiosity of reflexive modernisation.’ A central component of this evaluation imaginary is neorigorism, which entails establishment of a number of values and principles as non-negotiable to counter undesirable characteristics of reflexive modernity, such as perspective pluralism, value-laden evaluations and contingency. Evaluation activities in the audit society imaginary provide ways to reduce complexity to counter such effects of the previous imaginary.
The audit society evaluation imaginary also includes a rise in organisations’ management-orientation, with associated increases in bureaucracy, administration and evaluation activities. This is clearest in increases in the number and reach of evaluation ‘machines’ (or systems). These machines have five main characteristics. First, they involve permanent and recurring evaluation activities. Second, they are closely related to the organisation, and in the absence of strong regulation they can provide organisations with legitimacy. Third, they are prospective in the sense that their outputs are timed to be available for use in budgetary processes or decision-making windows. Fourth, evaluation machines generally have distinct epistemological foundations and elements, including use of tolls such as scripts and handbooks to support their standardised operation. Finally, evaluation machines describe activities in ways that make the results comparable and generalisable (Dahler-Larsen, 2012). Through amassment of data and production of indicators they provide straightforward quality statements about complex evaluands. However, such evaluation machines have been shown to have undesirable effects, such as metric fixation (Bevan and Hood, 2006) and increases in documentation that may shift human and monetary resources away from an organisation's core assignments towards bureaucracy and fringe activities (Coulson, 2009).
Elements of all these imaginaries are present to varying degrees in society at large, and the presence of key elements of one does not exclude the presence of other imaginaries’ typical elements. However, in various fields such as education, and sub-fields such as ECEC, elements of one imaginary may predominate.
Methods and material
The study presented here was based on a qualitative text analysis, regarding texts as tools that significantly shape our understanding of the world (Ferraris, 2013). The scrutinised texts are so-called QRs created in preschools. We argue that they warrant attention as they express values, judgements and interpretations of what good ECEC is or ought to be, as well as aspects that were considered relevant, visible and indicative of ECEC practices in making such judgements as well as how those judgements were made. In searching for and selecting QRs to analyse for the study we strove to cover the maximal feasible variation (Ritchie et al. 2014), by including QRs from diverse ECEC providers (municipal and private, as well as various sorts of private providers, such as cooperatives and business-oriented companies) with equally diverse geographical locations, sizes (in terms of both municipality size and numbers of children enrolled in their preschools) and pedagogical profiles. We have striven to examine textual documents from various preschools operating under various conditions as different service providers might have different motives and approaches that influence both their judgments and rankings of values, as discussed for example by Roomkin and Weisbrod (1999). Due to the purposive selection strategy we applied, focusing on variation, the resulting sample of QRs cannot be considered as representative as those generated in Swedish ECEC settings, as it is both too limited and targeted.
However, we argue that it is suitable for the purpose of this paper. The material consists of eight QRs, ranging from eight to 35 pages, and their authors were principals and co-workers of the preschools when stated, but otherwise anonymous (see Table 1).
The material: Quality reports and contextual characteristics of the preschools.
Since the QRs are public material and can be found freely on websites of the preschools or other open platforms we did not seek formal ethical consent for the purpose of our study. However, we have only quoted short extracts from the QRs and anonymised them to avoid singling out a particular preschool, and followed the Swedish research council's ethical guidelines (Vetenskapsrådet, 2017).
The analysis was conducted in two steps, in line with a directed content analysis approach (Hsieh and Shannon, 2005). In the first step we focused on descriptive mapping of the QRs’ content. Following Schwandt's (2015) notion that what is rendered valuable and judged as good, poor, successful or unsuccessful is contested and context-specific, this part of the analysis was guided by the following questions. What is the content of the QR? What criteria were used? What data/material are mentioned? What was evaluated and what is referred to (e.g., curricula, policy documents, research)? In this phase we primarily addressed the first research question. In the second step, we primarily addressed the second research question, and used the developed framework (described below) to guide the analysis.
Analytical framework
To analyse how the evaluation imaginaries become manifest in the studied ECEC QRs we have developed an analytical framework by bringing together Dahler-Larsen's (2012) theorisations on evaluation imaginaries, Schwandt's (2015) understanding of values in evaluation and Donabedian's (1988) understanding of three main classes (structure, process and output) of quality. Schwandt (2015) and Donabedian's (1988) approaches provide the analytical framework with more practical descriptions of what different imaginaries may entail, often lacking in Dahler-Larsen's (2012) more abstract theory building. Schwandt's (2015) understanding of values and how they intersect with different epistemological strands provides a way to analyse the values included in the QRs. We use Donabedian's (1988) understanding of different classes of quality (structure, process and output) by linking them to the different evaluation imaginaries, which enables us to analyse the understanding of quality embedded in the QRs. Following the notion of social imaginaries in general (Taylor, 2004) and evaluation imaginaries in particular (Dahler-Larsen, 2012) we argue that in each evaluation imaginary the core evaluation characteristics (scope, form, approach, object and values) have typical features, forms and functions. Collectively, elements of our theoretical framework (summarised in Table 2) provide conceptualisations of the key characteristics of evaluation connected to each imaginary, and hence facilitate analysis of both evaluation activities and their relations to specific imaginaries.
Framework for analysing evaluation activities and their relations to evaluation imaginaries (freely based on Dahler-Larsen, 2012; Schwandt, 2015 and Donabedian, 1988).
The evaluator in the modernity imaginary is usually an external evaluation expert with few or no connections to the evaluand in order to secure objectivity (Dahler-Larsen, 2012). In reflexive modernity the evaluation is usually conducted by internal members of the focal organisations, for example, professionals such as pedagogues or preschool principals in ECEC settings. A common feature of evaluations in the audit society imaginary is the absence of evaluators since evaluations tend to be automatic and ongoing, based for example on recurring indicators (except perhaps commissioners or writers of applied algorithms). In terms of scope and form, evaluations in the modernity imaginary are typically large-scale, ad hoc and intended to provide both generalisable results and comprehensive assessments of programmes or projects (Dahler-Larsen, 2012). In reflexive modernity, evaluations tend to be small scale and recurring, to address local-level needs and encourage reflection. In the audit society imaginary, the scope is small scale in the sense that every organisational unit should be appraised, but the appraisal should be scalable (through complexity reduction) to enable comparison of units’ efficiency and promote competition in line with New Public Management ideals.
The evaluation approach in the modernity imaginary is based on production of objective knowledge to assist rational decision-making. In reflexive modernity, stakeholder involvement is a core element of the evaluation approach and in the audit society imaginary neorigorism (the construction of quantifiable metrics and indicators that enable comparisons within and between organisations) is a core element.
The evaluation object of quality can focus on structure, process or output in the different evaluation imaginaries. Structural quality refers to the degree to which structural features of an organisation, external conditions and available resources help or hinder achievement of the organisation's goals (Donabedian, 1988). In a preschool relevant factors may include financial resources, educational levels of the personnel, staff density and number of children enrolled (which are often governed by external forces). This could be found in the modernity imaginary. Process quality concerns the interactions between children and teachers, and educational activities, such as the staff's approaches, their way of working and what happens in the meetings between the pedagogues and children. Which would be in line with the reflexive modernity imaginary. Output quality is related to goal fulfilment and results, that is, the degree to which goals are achieved. It may focus on educational results, that is, what the children learn and the skills they develop, but the extent of parental satisfaction may also be considered. Such output focus would align with the audit society imaginary.
Finally, values in the three evaluation imaginaries also differ (cf. Schwandt, 2015). In modernity, scientific values and rationality are the most important, as exemplified by use of the hierarchy of evidence notion (Vedung, 2010) to assess the validity of claims and results of evaluations. In reflexive modernity's evaluation imaginary, the stakeholders and professionals’ values are at the centre of evaluation. In the audit society values of accountability and effectiveness strongly influence evaluation activities.
This framework led us to examine the characteristics of each imaginary and how they became manifested in the QRs. Specifically, we paid attention to the expressions related to scope, form, approach, object and values expressed in the evaluation.
Quality reports: Evaluative judgments and evaluation imaginaries
In this section we present our findings related to the two research questions. First, we address the question of how and on what basis ECEC practices are evaluated in the QRs. Subsequently, we analyse how this can be understood using a framework of evaluation imaginaries.
Evaluative aspects in the QRs
Our initial mapping of the QRs’ content was guided by our first research question, and hence focused on the characteristics of the QRs, on what basis judgments were made (mentioned data sources), the evaluative criteria applied, object of evaluation, and what was referred to (see also Table 3, Appendix). These aspects provide the structure for this section.
Our analysis shows that the QRs seem to be documents of a representative characteristic intended for an external audience as they all describe structural factors such as numbers of enrolled children, staffing and, in some cases, the buildings and surroundings. In some QRs there are also descriptions of the organisation's value base, thoughts on teaching and overall approach regarding their mission. The QRs’ are to a large extent descriptive in character as most of the content focuses on descriptions of projects, themes and what has been done, or the kinds of learning opportunities that have been provided during the year, as exemplified in one of the QRs as follows: The preschools’ project compilations tell four different stories about the activities the children have been offered. The stories show different impacts with the curriculum goal formulations as a basis. They show that the children's learning and skills have developed and changed.
As shown in the quote, the tone in the QR is representative, intended for an external reader in that it expresses how the reader should understand the text. It is also exemplary of another central aspect: that judgments are made but the reason for the judgments are not always clearly expressed.
Regarding the basis for judgments, various data sources are mentioned and drawn upon in both the QRs’ descriptions and assessments. Although not articulated, they seem to be based on the personnel's descriptions and statements. The explicitly mentioned data sources vary from just one (guardian survey) in QR5 to nine in QR1 including surveys, project summaries, analyses and interviews. Most QRs also mention views of guardians expressed in surveys, as in the following quotation: In order to meet the guardians who, according to the survey, do not fully agree with the statement ‘I receive continuous information about my child's life at preschool (e.g., well-being, development and learning)’, the educators will supplement what they write on the board in the entrance with a reminder to look at [the digital platform] to get more information.
Children’s perspectives are also mentioned in some QRs, but only in terms of provided activities and learning opportunities, rather than verbally or behaviourally expressed perceptions.
The criteria used are seldomly clearly expressed or defined (Table 3, Appendix). For example, one (QR4) expressed them as ‘goal fulfilment’ and another (QR8) explicitly mentions a focus on ‘children's development and learning’. However, most of the QRs described some kind of assessment regarding the ‘overall quality’ that the preschool provided, as exemplified by QR1: ‘We assess that our preschools, overall, offer the children a preschool of good quality.’ Most of the reports focus mainly on provided learning opportunities. In three of the QRs we discerned a different approach towards judgment of quality. In QR4 the extent of goal fulfilment is clearly expressed as the focus of the assessment, and the QR's structure is based on pre-set questions related to each specified goal: ‘Have you achieved the goals you set for the work and what influenced the results you obtained?’, ‘What measures have you taken?’, ‘What effects did you see?’ (QR4). QR8 states that the aim of reported assessment was to provide information about the children's development and learning based on a learning taxonomy ranging from remembering to understanding, applying, analysing, valuing and creating. The QR states that specified goals are assessed in terms of indicators linked to the taxonomy, as exemplified here: Effect goal: 1. Ability to function individually and in groups, cooperate, manage conflicts and understand rights and obligations and take responsibility for common rules. Development is ongoing when each child: Collaborates/interacts with others, both adults and children. Effect goal 2. Ability to explore, describe with different forms of expression, ask questions about and talk about science and technology. Development is ongoing when each child: Notices and explores differences between different sounds and their impact on the ear.
QR8 also describes how the personnel's observations using this approach could be focused, and how reflecting on associated questions guided their discussions on efforts to learn ‘where the children are situated in their learning’.
The external sources that the QRs refer to include statements of policy goals and rights at the international level (e.g., the United Nations Global goals and Convention on the Rights of the Child), national level (e.g., the Swedish Education Act and National Curriculum) and local level (e.g., documents generated by municipal or private organisers and preschool work plan). Several QRs also mention various books and researchers, as well as specific models and templates.
In conclusion, the QRs are predominantly descriptive. As have been shown, central aspects of evaluations such as the object of the evaluation, criteria applied and on what basis judgments are made are often implicit in the reports. However, the content of the QRs show that they tend to describe learning opportunities provided, in line with previous studies of documentation practices in Swedish preschools (Åsén and Vallberg Roth, 2012; Bjervås, 2011; Johansson, 2016; Roth, 2014; Urban et al., 2023; Vallberg Roth, 2015). However, we also found examples of other kinds of practices, focusing on the children's learning or level of learning in relation to various indicators, but also other aspects which received appraisals and were judged or considered important or indicative of the work undertaken at the preschool. These aspects will be further scrutinised in the next section through the developed framework of evaluation imaginaries.
Quality reports from the perspective of evaluation imaginaries
When applying our framework based on Dahler-Larsen’s imaginaries in analysis of the QRs we detected few indications of elements of the modernity's evaluation imaginary, apart from some elements associated with the object of evaluation. Introductory passages of most of the reports describe structural aspects of the preschool they concern, in line with modernity's evaluation imaginary, including the kind of preschool it is and important variables, such as the number of qualified preschool teachers and children. However, these are fact-stating or scene-setting passages that express no judgements of such aspects. Although structural outputs are described they are not evaluated in relation to components of modernity's evaluation imaginary, and there is no clear connection between the evaluator, scope, form, approach or values and corresponding elements of the modernity imaginary. For example, both the scope and form of the QRs differ from the large-scale, ad hoc evaluations associated with that imaginary. The quality reports are localised evaluations, recurrent and small scale. Moreover, the reported evaluations were conducted by pedagogues and principals themselves rather than external impartial or ‘objective’ evaluators.
The QRs displayed many characteristics of the reflexive modernity's evaluation imaginary. All were conducted internally by either the principal or the principal together with pedagogues. The QRs are also small scale and closely connected to the preschool activities, as exemplified by the kinds of content included in the reports. As outlined above, and in Table 3 (Appendix) this is largely information that is closely connected to the preschools’ activities and performance, with no comparative indications of the quality of the activities and performance relative to those of other preschools.
The QRs often focus on learning opportunities provided, but some explicitly address what the children learned or developed knowledge about. In two reports (QR7, QR8) this is expressed and assessed in relation to the taxonomy of learning presented by Bloom et al. (1956), which is a hierarchical classification of levels of complexity and specificity of educational learning objectives. Examples of such evaluative statements include: We can see that the children have become familiar with what plants need to grow … These children displayed familiarity with how funerals are conducted and could spread their new knowledge through conversations with their friends. (QR1)
The evaluation approach applied in creation of the QRs was often stakeholder-oriented, and multiple stakeholder perspectives were often included in analyses of the preschools’ quality. For example, responses of parents and guardians in surveys and documentation from the pedagogues were recurring sources of data for the quality appraisals.
One aspect that is clearly aligned with reflexive modernity's evaluation imaginary is that the pedagogues’ reflection on their own teaching is treated as a sign of quality in itself in some of the reports: The matrices [templates with pre-set questions] are tools used during the pedagogues’ reflection time. The teaching is followed-up, evaluated and analysed. The matrices help to deepen the analyses with pre-set questions … The matrices also help visualisation of the conditions and possibilities the pedagogues create for learning. (QR3)
Self-reflection by the staff on their own pedagogical practices is clearly treated as an important component of high quality in such cases. In addition, the matrices provide guidance and are intended to deepen the analysis and reflection. The tools applied are thus used to increase reflection by the pedagogues and the organisations generally.
Finally, some aspects of the QRs show traces of the audit society's evaluation imaginary. For example, the object of evaluation may include preschool staff's use of templates and matrices. Although probably not intentionally, applied matrices are strongly highlighted in some reports, and related to specific authors or researchers and their work. For example, in one report (QR3), ‘measurable goals’ for one of the stated overarching goals is use of specified matrices for both individual- and group-level assessments:
Measurable goals:
Are [name] matrices used for group-level assessment? 4 out of 4 units = 100% Are [name] matrices used for assessment at the individual level? 2 out of 4 units = 50% Does the mapping documentation develop the staff's professional language? 4 out of 4 units = 100% Do the pedagogues give examples of how the matrices have contributed to a professional language towards guardians and children and to other actors within the collective school activities? 2 out of 4 departments = 50%
The evaluation object in this case is the extent to which the units in the preschool used the matrices for assessment. In one QR a matrix was used to assess goal achievement through a numerical scale. It is not specified how the goal achievement was appraised, but the report focuses on how the appraisal changed over time and whether the goal achievement had risen (Q2).
Examples of neorigorism described in the reports include standardisation efforts and use of tools intended to facilitate streamlining and reduce complexity. In this sense the extensive use of matrices and other documentation tools could also be seen as signs of neorigorism, which may be partially true, but as already mentioned many of these matrices were used to enhance and encourage reflection rather than to provide comparable outputs. The use of matrices also seems to have other effects. It seems to encourage and foster ritualistic reflections. For example, many appraisals of quality in the QRs are based on pedagogues’ use of the reflection matrices rather than what the pedagogues actually reflected on when using the matrices.
A key element of the audit society's evaluation imaginary is that organisations must be auditable to be perceived as serious and legitimate. We detected signs of auditability in the reports, but only partial, manifested through statements of evaluative activities rather than generalised metrics or indicators that can be compared and validated. The only (potentially) generalisable metrics mentioned in the QRs are sick-leave rates and results of surveys of parents/guardians and employees. However, these kinds of data are not applied in any kind of ‘evaluation machine’ that enables numerical comparison to corresponding information from other preschool providers in the reports. Such comparison may have been conducted at a higher governing level but that is beyond the scope of this article. In summary, the QRs show partial and inconclusive traces of the audit society's evaluation imaginary, as they lack statements of quality from outside the organisation based on clearly constructed indicators. Thus, our analysis indicates that local level quality statements of Swedish ECEC are more strongly influenced by the reflexive modernity imaginary than the audit society imaginary.
Concluding discussion
Our exploration of datafication in Swedish ECEC, with an empirical focus on QRs in relation to three evaluation imaginaries was guided by the RQs: (1) how and on what basis are ECEC practices evaluated in the QRs and (2) how evaluation imaginaries become manifested in QRs. Regarding RQ1 our primary finding is that the QRs express an overall assessment of quality, but generally without explicit definitions of quality, the criteria applied or how it is judged. Further, we find tendencies that require more empirical exploration, of differences between ECEC providers, which must be treated cautiously due to our small sample size. However, there are indications that the values attached to quality assessments may depend on the characteristics of ECEC providers, as suggested by Roomkin and Weisbrod (1999). Our analysis indicates several such differences. Two are that the private providers’ QRs tended to be longer and more detailed than those of their municipal counterparts. In addition, the private providers’ QRs were all adapted to preschool settings and did not display traits of school-relevant indicators, which were found in some of the municipal QRs. These findings suggest needs for further exploration of evaluations in relation to marketised context as well as how national legislative frameworks are translated by ECEC providers with different profiles.
Concerning RQ2, we interpret the content and expressions in the QRs as being mostly in line with the reflexive modernity evaluation imaginary (Dahler-Larsen, 2012), particularly due to their local operational and functional nature. They are based on perspectives of the pedagogues and school leaders, and tend to focus on stakeholders’ participation, descriptions and experiences, with few or no claims to objectivity. They can thus be interpreted as intentionally focusing on aspects considered relevant for the preschools’ professionals, supporting reflection.
As the QRs were interpreted as aligning with the audit society's evaluation imaginary to a lesser degree, it could be concluded that datafication has not permeated or impacted Swedish ECEC as strongly as in educational settings of other countries and levels. However, as mentioned, some QRs and some aspects of the QRs did display traits of the audit society's evaluation imaginary and indications of datafication characteristics. One example was the use of a five-grade scale for quality self-assessments by the staff, and for determining if their ratings were higher or lower than in the previous year. This reflects one of the key aspects of the audit society's evaluation imaginary: reduction of complexity, manifested through use of a numerical metric to gauge quality, that is, datafication (Mayer-Schönberger and Cukier, 2013). Another structural element of the QRs connected to the audit society's evaluation imaginary is in the comprehensiveness of coverage. They are intended to provide a recurring account of preschools’ quality, regardless of the need for such comprehensive review. This is a key feature of evaluation machines (Dahler-Larsen, 2012), and a key element of the audit society's evaluation imaginary. A final aspect that could be interpreted as being in line with the audit society imaginary is their auditability function. This entails making the preschools auditable – a key feature of legitimate and serious organisations according to the audit society evaluation imaginary (Power, 2007) – through the process of writing and distributing QRs. As the QRs include overall assessments of the preschools’ quality, and many value-laden statements (e.g., that the ECEC service provided has high quality, and that the personnel and caregivers are satisfied), they can be interpreted as attempts to manifest quality to actors outside the organisation in a documented format. Thus, their production may be regarded, both within and outside the preschools, as essential for legitimacy. However, since some kind of documentation of quality appraisal is expressed as necessary in national policy documents, this is part of a national governing system (Håkansson, 2019) rather than being a specific QR trait.
Accordingly, we should emphasise that our analysis is based on QRs, which are just one type of evaluation of Swedish ECEC. Analysis of other evaluative activities might provide indications of other influential evaluation imaginaries and alignments of the intrinsic components. For example, the QRs are local evaluations, but at the national level in Sweden there are examples of evaluations that are more in line with modernity's evaluation imaginary, as shown for example by Aalto et al. (2019).
Finally, we argue that although this is not the main focus of this article, the framework could be useful for exploring effects of prevailing evaluation imaginaries on a deeper level. For example, according to Dahler-Larsen (2012), effects of the reflexive modernity's evaluation imaginary include lack of enthusiasm for evaluation and lack of complexity reduction. An analysis of QRs alone, as in this study, is not enough to determine the presence or absence of such enthusiasm. Although the ECEC QRs are mainly aligned with reflexive modernity's evaluation imaginary, there are indications in the form, evaluation approaches and objects in some of the QRs that the audit society's evaluation imaginary may be on the rise in Swedish ECEC. The different evaluation imaginaries are therefore best understood as layered and will manifest differently depending on the context's historical trajectory. One example is the manifestation of ritualistic elements found in the QRs. Manifestations that are potentially a product of how the evaluation imaginaries of reflexive modernity and audit society are layered and interlinked within evaluation practices in ECEC.
Footnotes
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The authors received no financial support for the research, authorship and/or publication of this article.
Notes
Author biographies
Appendix
See Table 3.