Abstract
This paper examines how predictive algorithms construct risk by calculating and anticipating children's uncertain futures. Theoretically, we analyze algorithmic risk construction by attending to (a) the problematizations justifying algorithmic prediction, (b) their underpinning data infrastructures, and (c) the configurations of agencies across humans and machines. Empirically, we examine two experiments in Danish child protection services that developed algorithmic models to predict children's maltreatment. Our analysis highlights how algorithmic predictions can create different notions of risk. The first case used predictive algorithms to supplement human risk assessments with data from child protection services, while the second case aimed to detect risk early by constructing parents as risk factors, requiring data from other welfare sectors. By comparing these cases, we highlight two distinct risk constructions: one that uses algorithmic prediction to manage uncertainty and another that seeks to eliminate undesired futures by preempting risk. These different constructions have implications for how the present is viewed as a moment of intervention and for how families are constructed as “risk objects.”
Introduction
We need to work with early detection, right. Especially since we know how the first three years are the most important in a child's life (…). So, we’re debating: how do we identify these families, already when their children are infants or when their mothers are pregnant? (Interview, Municipal manager, child protection services, June 2021)
Algorithmic systems for decision-making, or automated decision systems (ADM), are often introduced to “advance public welfare and social good” (Leslie et al., 2020: 4). Government agencies are increasingly attracted to the idea that welfare services can identify and prevent citizens’ problems before they become severe, complex, and expensive. To achieve this, they are adopting algorithmic scoring and calculation methods to allocate risk, resources, and services to citizens (Dencik et al., 2019; Henman, 2004). As the introductory quote illustrates, this was also the case in the context of child protection services in a Danish municipality. Taking us through a long history of failing early detection initiatives, the municipal manager explained their rationale for developing a predictive algorithm to manage risk. Working risk-based through algorithmic prediction, the manager argued, would allow the municipality to intervene with families in a gentler manner, anticipating and intercepting children's maltreatment before symptoms would even emerge—perhaps even before the child had been born. This showcases how promises of algorithmic prediction give rise to new forms of risk management, allowing governing bodies to anticipate and preempt unwanted futures in new ways.
Many scholars interrogate such ADM systems from a social justice, fairness, or harm perspective, highlighting their adverse effects such as biases or increased surveillance (e.g. Eubanks, 2018; Leslie et al., 2020; Redden, 2020). We are interested in exploring a different dimension of these systems, specifically how they construct “risk” and the implications arising from different algorithmic risk constructions. Predictive algorithms make it possible to relate to uncertain futures as problems of risk. However, predictive algorithms not only work to absorb uncertainty about the future but are performative and shape ideas about managing risk. As Amoore (2013) has noted: “[Risk technologies] hold out the promise of managing uncertainty and making an unknowable and indeterminate future knowable and calculable” (p. 7). In this view, predictive analytics are not simply technologies of governance used for population management but a reordering “of what the future could be” (Amoore, 2023: 22).
While several scholars focus on eliciting an overall temporal logic of predictive algorithms (e.g. Amoore, 2013; Cevolini and Esposito, 2020), we propose that the algorithmic construction of risky futures is no singular matter. This insight builds on science and technology studies’ (STS) long-standing concern with the situated and variable construction of socio-technical systems (e.g. Jaton, 2021). Conceptually, we elicit algorithmic constructions of risk by examining their problematizations (Foucault, 1980), data infrastructures, and human–machine configurations (Suchman, 2007). These entail distinct constructions of children's risky futures, resulting in different profiling of citizens as “risk objects” and different temporalities for when to manage such risks.
Empirically, we compare two Danish experiments developing algorithmic models to calculate children's probability of future maltreatment. While predictive algorithms are internationally on the rise in child protection services, they are highly controversial in a Danish context and are yet to make it into child protection services (Kristensen, 2022; Leslie et al., 2020; Ratner and Schrøder, in press; Redden, 2020). One of the projects, initiated by a Danish municipality, has been cancelled whereas the other, a research project, has been changed to the extent that the algorithmic model will not be tested on real cases. One might object that cancelled algorithmic systems may not teach us much about contemporary algorithmic risk constructions. On the contrary, we contend, together with Grant (2020), that cancelled algorithmic systems constitute a window into contemporary value struggles about what our shared undetermined futures might look like, and what role algorithms should play in our relationships to these futures. In other words, what is at stake is the ‘future politics' of government interventions (Amoore, 2013).
After a background section introducing scholarship about ADM systems in child protection services, we review extant literature discussing the temporality of predictive algorithms. Building on these studies, we outline our conceptual approach and analytical framework. We then analyze the “research project” and the “municipality.” Whereas the research project developed a predictive algorithm to support existing practices of risk assessment, using data from only their own administrative sector, the municipality endeavored to predict children's risk of maltreatment, ideally from the moment of conception, looking into parental “risk factors” and hence requiring data from other administrative systems. In the discussion, we compare the two algorithmic constructions of risk, discussing the difference between using algorithmic prediction to tame uncertainty in existing risk assessments and using algorithms to preempt undesired futures before they emerge. We also consider the possible implications of introducing algorithmic risk calculations, such as increased accountability and proceduralization of professional discretion. We conclude by discussing the value of analytically attending to multiple algorithmic risk constructions compared to an overall and singular algorithmic risk logic.
Background: Predictive algorithms in child protection services
There is a growing scholarly concern regarding the proliferation of ADM systems in the public sector, which aims to target, profile, and predict citizen behaviors and outcomes. Eubanks (2018), for example, has examined how ADM systems are used to manage social welfare programs in the USA, arguing that they are designed to prioritize efficiency and cost savings over the needs of the people they are supposed to serve. In her view, ADM systems are increasingly used to surveil and police the poor, subjecting them to invasive and often punitive forms of monitoring and control. Core to her study is the replacement of social work expertise with standardized algorithmic systems, which may shift decision-making power away from the caseworker's discretion. Similarly, Jørgensen and Nissen (2022) observed how roles and accountability relations were renegotiated with the introduction of an algorithmic decision support tool in a Danish social work context. Other scholars have pointed out that ADM systems may be built on flawed assumptions and biased data, leading to discriminatory outcomes for marginalized communities and raising questions of “algorithmic fairness” (Dencik et al., 2019; Gillingham, 2020; Körtner and Bonoli, Forthcoming; Redden, 2020). Finally, scholars have found that ADM systems may be related to austerity programs used to prioritize resource allocation and service availability and, thus, influencing citizens’ life chances and possibilities in adverse ways, and leading to a “technicisation of social work” (Leslie et al., 2020: 29). Some scholars even conclude outrightly that ADM systems “relating to predicting real life have a long way to go to being safe and trusted for use” (Waller and Waller, 2020).
These critiques raise important questions about ADM systems’ possible harmful and detrimental effects. While acknowledging these, we take a different approach. Inspired by Amoore (2013), we focus on how ADM systems, based on predictive analytics, prioritize capturing risk over other forms of knowledge. While ADM systems may generate new forms of calculated and data-intensive surveillance, more is at stake here than these systems’ accuracy and fairness (Gillingham, 2020). Instead, ADM systems are performative in nature and have world-making effects as they materialize ontologically different ‘citizen objects' (cf. Ruppert, 2012). This has consequences for how the state views the citizen as a risk object and for what counts as the “timely” intervention in the name of child protection. Rather than assessments of these systems according to their fairness or efficiency, we are here interested in their performative effects in terms of constructing citizens’ futures as objects of risk management. In doing this, we situate this article in dialogue with scholars examining the temporalities produced by predictive algorithmic systems.
Predictive algorithms as political technologies of the future
While technologies for governing the future have a long history within organizations and society (Flyverbom and Garsten, 2021), the advent of predictive algorithms has reinforced this ambition (Christin, 2017). Most authors agree that predictive algorithms render the future actionable in the present and work as a technique for mitigating its uncertainties. Below we visit some recent debates to elicit how we build on and contribute to these discussions.
According to Esposito (2013), algorithmic prediction represents a shift in modernity's concept of an open future, moving from an unknowable and uncertain future to one figured through predictability and probability. This resonates Sheehey's (2019) study of predictive policing where the future becomes less open for specific groups. For instance, racialized groups tend to be “relegated to a futureless past” in the sense that criminal pasts are assumed to be repeated in the future for predominantly minority populations. In contrast, white people are typically granted more open-ended futures. This is a preemptive temporality, Sheehey (2019) argues, where “the future has to acted on as if it will happen” (p. 54, emphasis in original). Sheehey concludes that predictive algorithms may generate a paranoid future that accentuates the politics of the past, thus becoming a “self-fulfilling” image of “a criminal past” that is projected onto future events (p. 50).
More fundamentally, predictive analytics reorder the role of uncertainty in how we approach the future. This reordering involves a shift from the previous use of probabilistic statistical techniques, calculating probable futures at the level of populations, to “the algorithmic arraying of possibilities such that they can be acted upon” (Amoore, 2013: 23). Amoore (2013) further suggests that we are witnessing a new government rationality that entails a shift from prevention to preemption. Preemption “acts not strictly to prevent the playing out of a particular course of events on the basis of past data tracked forward into probable futures but to preempt an unfolding and emergent event in relation to an array of possible projected futures” (p. 9). Traditional probabilistic models distribute uncertainty equally across a population, while algorithmic models differentiate risk at the individual level. This results in sorting population into individual profiles with varying degrees of risk (Cevolini and Esposito, 2020). In this way, predictive analytics materialize a form of governance that seeks to intervene with the population as unfolding risky futures, which can be known, differentiated, and preemptively acted upon.
Analytics for examining algorithmic risk constructions
The existing literature, discussed above, focuses on how an anticipatory logic of predictive algorithms changes the relationship between past, present, and future. These insights form an important backdrop for our analyses. At the same time, rather than eliciting an overall algorithmic temporal risk logic, we suggest that the algorithmic recasting of children's unknown futures into objects of anticipation is neither univocal nor singular. In making this argument, we follow scholars associated with STS who understand algorithms as multiple socio-material entities that cannot be reduced to one singular logic or computational artifact (Jaton, 2021; Seaver, 2017; Ziewitz, 2016). As Seaver (2017) notes, “algorithms are enacted by practices which do not heed a strong distinction between technical and non-technical concerns, but rather blend them together” (p. 5). We propose to analyze the predictive algorithms as part of (a) a wider governmental apparatus; (b) as contingent upon the datasets used for its development; and (c) as a configuration of human and machine agencies. By examining these three aspects, we can identify how risk constructions render uncertain futures governable in distinct ways.
First, we approach algorithms as part of a more comprehensible ensemble of a governmental apparatus, i.e. as “government schemes devised to both calculate and intervene in the performance of individuals and populations” (Ruppert, 2012: 120). In this understanding, a predictive algorithm is not isolated from governmental strategies but part of a broader landscape of regulations, procedures, instruments, and institutions. To examine this aspect, we will investigate how algorithmic technologies are accompanied by specific problematizations (cf. Foucault, 1980), which involve the identification and definition of a problem, as well as the development of a set of techniques to address it, hence pointing out a solution. Problematization, thus, does not simply regard the identification of a problem or issue but the way it is articulated and constructed within a particular governmental context. Hereby, we identify the risk constructions that arise when our two cases frame children’s uncertain futures as a problem that can be anticipated and acted upon with the aid of a predictive algorithm.
Second, we need a sensitivity to the data infrastructural arrangement entwined with these problematizations, here understood as datasets feeding the algorithm. Predictive algorithms infer futures from patterns and correlations found in datasets. Thus, the datasets used to train the algorithm are crucial regarding which variables are used to infer and attribute risky futures to individuals. As Amoore (2013) notes, “In isolation each data item offers little to the calculation of probable risks. Once associated together, though, what matters is the capacity to make inferences across the data, such that risk derivatives can be recognized, shared, and [made] actionable” (p. 63). To understand the algorithmic constructions of risk, i.e., how they assign risk to citizens, it is imperative to study how these constructions are achieved through the integration of datasets.
Finally, predictive algorithms generate new relational dynamics between humans and machines, and these are important to investigate as part of their risk construction. In her notion of human–machine configuration, Suchman (2007) abandons a strict separation between humans and machines—predictive algorithms in our case. Instead, she explores how their different agencies are “figured” together and how different tasks are delegated to these agencies. With a sensitivity to this figuration, we will examine the envisioned agency of the algorithm vis-à-vis the human. Specifically, we will look into which responsibilities are delegated to respectively human professionals and the predictive algorithm, and how these two agencies are imagined to collaborate. These configurations are closely entwined with problematizations (e.g. if human risk assessments are problematized) and, as our analysis will show, are essential in the construction of—and intervention in—risky futures.
Case material and methods
We analyze two empirical cases (the “research project” and the “municipality”) of algorithmic technologies within child protection services. Both cases aim to develop an algorithmic risk model to predict children's risk of maltreatment. Located in Denmark, they are subject to the same regulations and target the same field of expertise—caseworkers and social workers in child protection services—professionals that are typically considered to contain a high degree of relational expertise and discretionary authority (Ellis et al., 1999; Pakarinen and Huising, 2023). Furthermore, both cases share the distinct feature that they were cancelled or removed from casework. Officially, this was mainly due to regulatory matters, but both cases were also exposed to intense public scrutiny, including concerns of bias and automation, which challenged their legitimacy (c.f. Kristensen, 2022).
As other Nordic welfare states, the Danish social welfare system is based on principles of universal and free services, family orientation, and “least intrusive” interventions (Pösö et al., 2014). The Nordic welfare states’ child protection services have a long tradition of preventive and voluntary approaches targeting “at-risk” families (Gilbert et al., 2011). The ADM systems analyzed in this paper may be interpreted as techniques to amplify such preventive work and its corresponding risk logic. The Danish uptake of ADM systems may also be understood in a context of political calls for evidence-based practices as well as what Danish social work scholar Hestbæk describes as a “slow movement away from voluntary family and child welfare measures (…) towards a more interventionist child protection regime” (Hestbæk, 2011: 136). Finally, Danish child protection services are currently facing administrative challenges in managing a significant increase in notifications related to potential child maltreatment cases, with a reported 112% rise from 2014 to 2019 (Falster, 2020). This situation serves as a justification for the research project.
More generally, Denmark has been pushing public sector digitalization for years and is considered top-ranked in digital government (UN, 2022). Denmark has had consecutive digitalization strategies since the early 2000s, and in 2019, the Danish government decided on a national artificial intelligence (AI) strategy that should push AI in the public sector (Government, 2019). These developments have led scholars (Jørgensen, 2023) to suggest that the Danish public sector risks advancing “a digital technocracy that treats its citizens as data points suited for calculation and prediction rather than as individuals with agency and rights” (p. 123).
Cases: The research project and the municipality
The interdisciplinary “research project” commenced in 2017 and involved collaboration between the Centre for Child Research at Aarhus School of Business (BSS), the Department of Economics and Business Economics, and social work researchers at VIA University College (VIA). The research project has developed a predictive model that anticipates the risk of future maltreatment in children. It aims to assess whether such a model can serve as a decision support tool to enhance caseworkers’ risk assessments of notifications from statistical, social work, and family perspectives. This use of algorithmic risk prediction as decision support is internationally quite common in child protection and unemployment profiling (Körtner and Bonoli, Forthcoming; Leslie et al., 2020).
The predictive model was developed using supervised machine learning on administrative data from Statistics Denmark (containing information on all Danish residents). This register includes individual-level data from Danish child protection services on “(a) all notifications of concern regarding child maltreatment received by Danish municipalities between April 2014 and December 2018 (195,639 different notifications, representing 102,309 unique children); (b) all interventions implemented by CWS between 1977 and 2018; (c) all removals and subsequent placements from 1978 to 2018” (unpublished research paper, 2021). After testing various machine learning models, they chose the algorithm developed with a gradient-boosting machine learning technique. Using out-of-home placements as a proxy for maltreatment, the model was built to provide caseworkers with a risk score (1–10) on children notified to the child protection services. This is where the VIA researchers were to examine caseworkers’ and families’ experiences with the model. However, after a public contestation of the model's legality (Andersen, 2021), the researchers decided in June 2022 to only test the model on artificial cases.
Our second case, the municipality, began as a development project in 2016 in Gladsaxe municipality and was the first Danish attempt to develop predictive algorithms in child protection. They pursued this as part of a national program to de-bureaucratize public administration, which was termed the “free municipalities experiments” (internal document, B2). Here, they chose child protection services as an area of focus to support their 2016–2019 “Strategy for early and holistic intervention” and the “de-bureaucratization” aspect consisted in applying for exemption from data protection legislation. The purpose of this algorithmic model was to predict risk before notifications arrive, to detect children at risk as early as possible. While algorithmic prediction for early detection, rather than decision support, is unusual in child protection services, it is not unheard of (Putnam-Hornstein and Needell, 2011).
The model combined data points from multiple registers in the municipality, including (un)-employment histories, residential information, information about substance abuse, disabilities, dental care, healthcare data, special needs education, contact with child protection services, and daycare (internal document). If the data exemption were granted by the Ministry of Interior, a predictive algorithm would be developed using a decision-tree machine learning algorithm, also using out-of-home placements as proxy. The model would then flag high-risk families, which a social worker would contact and offer voluntary help. However, after a national media controversy, the government did not grant them the exemption and the municipality was not allowed to continue its algorithmic experiment as part of the free municipality application (Kristensen, 2022).
Data collection and analysis
Data collection was done in parallel on the two cases, and we draw on an empirical corpus containing multiple data sources, including documents written by the two project owners, media and news coverage, and interviews. Whereas the research project provides public access to several documents describing the model, we had to request such documents from Gladsaxe municipality (see the specification of document material in Table 1). Both cases have been highly debated in public media, including radio programs, news sites, blogs, and social media. We also examined these debates as part of the data.
Overview of documents
Analyzing available documents provided a context for conducting semi-structured interviews. We have conducted 15 interviews with a total of 9 people, including project leaders, municipal managers, social work researchers, statisticians, and data scientists. These interviews were all transcribed verbatim.
In conducting the analysis, we were inspired by abductive analysis (Tavory and Timmermans, 2014) and previous comparative studies of algorithms, discerning their similarities and differences (Christin, 2020). We conducted a comparative thematic coding following our analytical categories of problematization, data infrastructures, and human–machine configuration. Problematization involved registering discursive articulations such as problem definitions, critiques of existing practices, and arguments for developing a new predictive technology. For example, in coding problematizations, we paid attention to how risky futures were articulated—and thus constructed—as an object of concern and how the algorithm was imagined to (better) calculate risk. Data infrastructures involved registering the datasets sourced for developing (training and testing) the algorithm, looking into which datasets were used (described above), from which administrative sectors, and which citizens were constructed as risk objects, as well as the actors’ reasons for including or excluding specific datasets. Human–machine configurations focused on the distribution of human professionals and algorithmic systems. This involved noting down the responsibilities and tasks delegated to humans and algorithms and the sequencing of their respective agencies, as envisioned by the developers of the algorithms.
The research project: Using prediction to tame uncertainty in human risk assessments
Problematizing professional discretion in a context of limited information
The research project developed an algorithm to support existing practices of risk-assessing incoming notifications. Notifications, often sent by professionals such as teachers or doctors, communicate concerns regarding the well-being and development of a child or youth. Upon receiving a notification, a caseworker must make a risk assessment estimating its gravity and acuteness within 24 hours (Gjedde et al., 2017). The research project developed a predictive algorithm, which can attribute a risk score to the child being notified to child protection services. This risk score is to function as decision support. The data scientist working on the project explained before the research project changed: “[In a Danish context,] this is the first time someone tries to develop databased risk assessments of notifications (…). Potentially, it can help some children. We want to find out whether caseworkers also find it meaningful” (data scientist, October 2021).
A problematization of professional discretion in a context of high uncertainty was central in justifying the algorithmic model. This problematization builds on a previous qualitative study of caseworkers’ risk assessments, concluding that caseworkers’ risk assessments are fraught with complex dilemmas, deep uncertainties, and a lack of standardization (Villumsen and Søbjerg, 2023). For instance, a report documented significant variations in caseworkers’ practices when it comes to determining how much information to incorporate into their risk assessments: should a social worker only assess the notification or gather more information (Gjedde et al., 2017: 7)? In the report, most caseworkers characterized their assessments as involving “doubt” and agreed that “more and better-described categories” could help them navigate various dilemmas. Central to this problematization were questions of how to sort information (which to include and exclude) and a wish for better standards in categorizing risk. The report concluded that a “statistical tool can potentially support the assessment of how fast action is needed and potentially also guide caseworkers in the further collection of information, in relation to high-risk factors for serious maltreatment” (p. 3). This led to the current research project developing the algorithmic model.
Asked about the challenges facing caseworkers assessing notifications, a social work researcher further described these as problems of moving between “the general” and “the particular” in a situation where there often is little information: Using a nice word, we call (…) [caseworkers’ assessments] discretion, but I mean, in reality, it's guesswork. I mean, it's guesswork because there is nothing to guide them here. For example, of course, we know that it's not healthy to grow up in an alcohol-abusing environment – we do know that this is not good. But we do not know exactly what the risk factor of it is here in relation to this particular child, in these particular circumstances, in this exact institutional environment. (Researcher, Social work, May 2021)
The statement problematizes the absence of clear guidelines to assist caseworkers in using their general understanding of children's risk and protection factors in actual risk assessments. We thus see a problematization of the general predicament for social work expertise in the context of risk assessments: Their general knowledge of risk and protection factors is of little help due to the limited information and time they have at hand when assessing children's risk on the basis of a notification.
To this end, the algorithm was invoked as a potentially helpful element with its capacity to detect nonlinear patterns across different variables found in historical cases through machine learning techniques. As the data scientist explained: At least, I imagine that maltreatment cannot be reduced to individual factors (…). So, it's not whether – for example – you have seven siblings or whether you have moved eight times within two years. Instead, it could be the interplay between some of these inputs that are predictive of children's future maltreatment. (…) We can model complex relations in data (…) we can model non-linear relationships between variables. (Data scientist, October 2021)
As the quote testifies, the researchers consider machine learning techniques capable of mapping complex relationships in the risk factors, here reconfigured as data points found to have high predictive values in historical datasets. In these ways, we see how the research project problematized professional discretion, deemed as too uncertain for risk-assessing notifications.
Building on dataset infrastructures within an existing jurisdictional domain
To understand the research project’s proposed use of algorithmic prediction, we must consider the role of data infrastructures. The legality of the model was important from the outset, and the project description emphasizes that to “guarantee the privacy and data protection throughout the system's entire lifecycle, the information upon which the tool is based is already used in the municipalities, adhering to the GDPR rules and regulations.” In Denmark, municipalities are not allowed to merge data from different welfare sectors (e.g. across medical and social work) without consent. The model was, therefore, only trained on data that caseworkers can also access—and are legally allowed to use—within their own administrative systems. Hence, their domain of jurisdiction would remain the same, and the model was to predict the future by automatically pulling 66 variables from the child protection services’ information systems, assigning them with algorithmically calculated predictive powers, and translating their combination into a risk score on a scale from 1 to 10.
Even if caseworkers already had access to the information pulled by the algorithm, the automated assessment of this information from various data infrastructures was seen as an improvement in terms of efficiency and standardization. As the project description states: When a caseworker is to decide on a citizen, some cases will have a lot of information about the child and its family, and they need to access this information through different digital information systems. This can be a rather massive task and there's a risk of disregarding important information or that information is given different weight from case to case or caseworker to a caseworker. (Project description, 2020, p. 3)
The algorithmic approach promised efficiency by automatically collecting data from various IT systems, thereby standardizing information collection and assessment. This was expected to minimize the risk of neglecting critical information. The problem to be algorithmically solved was not one of missing information but rather an uncertain and nonstandardized handling and assessment of the information in a complicated data infrastructure within child protection services. The algorithm was thus developed as a tool to make human information processing more precise by leveraging fast and standardized information processing methods. In this way, the algorithmic tool was to tame the uncertainty pertaining to caseworkers’ discretionary risk assessments, standardizing the question of which information (initially) was deemed relevant, and how it was relevant in assigning risk.
Configuring algorithmic decision support by leveraging professional discretion
The problematization of the caseworker's discretion is mirrored in the predictive model's human–machine configuration. The setup maintains humans with the proactive ability to detect children's risk. The algorithm is to be invoked after a maltreatment concern has been detected by a human professional (e.g. a teacher) and notified to the child protection services. Further, it not only attributes the child in question with a risk score. The risk score is “followed by an information sheet listing all the model inputs” (project description). As the project description explains: “This means that the social workers will have exact knowledge about which type of information the model relies upon, and thereby make it easier to infer the degree of additional information they might have about a specific case compared to the tool.” In the project description, the algorithmic model is configured as a responsive assistant in assessing the notifications, providing human caseworkers with risk scores and an information sheet allowing them to interrogate or further confirm the data leading to the algorithm's risk score.
This supportive function was critical for the researchers who, in interviews, were keen to stress that the algorithmic prediction cannot stand alone: “The caseworker will always have access to much more detail about the family in question than what we can observe from a few databases. I mean, there will typically be many factors, right? Which we could never include in our model. I mean, [the data used to develop it] are very aggregated” (data scientist, October 2021). The risk assessment was thus conceived as a combined agency involving algorithmic risk scores and caseworkers’ professional expertise. For the caseworker, this approach included not only evaluating the risk of the notification itself but also assessing the accuracy of the algorithmic risk score and incorporating additional human-accessible data. This configuration thus at once challenges professional discretion and expertise while also affirming its value due to contextual knowledge and access to disaggregated data.
In conclusion, the predictive technology developed in the research project aimed to support professional discretion by standardizing the sorting and assessment of information. Additionally, it aimed to improve the precision of assessments by using complex nonlinear relationships in historical datasets to generate and infer rules for assigning risk scores to children. At the same time, the algorithmic model does not change how the present is enacted as a moment of intervention. It rather sustains existing procedures of quick risk assessments of already detected concerns, hoping to render these more precise through algorithmic prediction. While the algorithmic model does not change the existing governmental focus on risk, it intensifies the risk management logic through the production of risk scores. We will return to this point in the discussion and now turn to the municipality's algorithmic project, which gave rise to another risk management approach.
The municipality: Using prediction to preempt symptoms of maltreatment
Problematizing the late detection of children at risk
Whereas the research project wanted to predict children at risk of maltreatment in response to a notification, the municipality aimed to use prediction for purposes of detection, i.e., before notifications being written. This vision of intervention was in direct line with the municipal strategy, viewing parents as risk factors and resources to work with. The predictive algorithm was thus one of many initiatives through which Gladsaxe municipality wanted to focus on early interventions with “parents, as the responsible part, for generating good conditions for their children's lives and development” (Gladsaxe municipality, 2016, s. 1). As we will argue, this reflects a different entwinement of problematization, data infrastructures, and human–machine configuration than what we saw above, leading to a different construction of risk.
The municipality problematized their detection of children at risk for happening too late. As the project manager argued in their choice of building an algorithmic tool for early detection: Even though we could see that the child was exposed at a very early age – for example by being absent from the dentist, divorce, [parents with] mental illness etc. […] Then we did nothing until they were 8 years old [when a notification would tick in…] At the same time, we know (…) that the first three years of a child’s lifetime are the most important. We also know about what matters [in having a good childhood]. For example, what significance does your childhood home have for your ability to learn mathematics as a 16-year-old. We know all this now, but why do we not act on it? Why is it then that we do not change our approach to these vulnerable children? (Project manager, Interview, June 29, 2021)
This problematization of existing detection practices has clear repercussions to their municipal strategy, which, referencing James Heckman's economic studies on “childhood investments,” stated that “research has documented the high return investments with early childhood interventions” (Gladsaxe municipality, 2016: 3). Not surprisingly, this led to the ambition of intervening with children's maltreatment as early as possible. As the head of child protection services work explained: We want to act more proactively (…) on risk indicators in the parents before there are symptoms of maltreatment in the child, to ensure an earlier and more effective response to [children's] risk of maltreatment (…) We wanted to avoid that they even began showing symptoms of maltreatment that we could see, for example, that their language did not develop or that they were absent from school or similar […] we aimed to predict it before it got to this point. (Head of social work, Interview, May 6, 2021)
This idea of acting “pro-actively” through a predictive algorithm materializes a somewhat different relationship to the future than we saw with the research project. Rather than increasing precision in risk assessments of already discovered symptoms of maltreatment, we here see an impetus to intervene before there is a human concern. This reflects a risk logic that seeks to eliminate problems by preempting even the first symptom that this risk would become a future reality. As Amoore (2013: 9) notes, predictive algorithms “reorient risk to action on the very basis of the imagination of those possibilities,” regardless of whether they appear. Thus, their problematization involved questioning their existing detection practices (the notification), invoking a predictive algorithm as a reorientation towards risk and analyzing all citizens as potential risk objects.
Expanding the data infrastructure beyond jurisdictional domains
What does it take, data infrastructurally, to anticipate symptoms before they emerge? The head of social work described it as increased information sharing across different welfare departments: “Our employment services manager at some point says that, in reality, [they] (…) are the first to know when a long-term unemployed mother becomes pregnant (…) However, this information stays with the employment services because there is no ‘forward-button,’ which would turn it over to the child protection services who could start working with these mothers” (head of social work, interview, 6 May 2021). In another interview, the project manager described the approach as viewing all indicators together, even if they may seem insignificant as individual data points: Nowadays, you see the indicators individually and react individually. And this might mean that the individual professional does not react because the individual indicator alone is not sufficient to react. But the fact that we combine the data [with data from other systems which the caseworker cannot access] and can see that here is a child who is both absent from dental care, the parents are long-term unemployed and one indicator more - this constitutes a reason to contact [the family] and see if there is anything you can do to help. (Project manager, Interview, 29 June 2021)
As the excerpt illustrates, existing data infrastructures were central in their problematization. Departmental digital “silo” infrastructures kept data from circulating between departments, and, as a result, kept the “revelatory” combination of indicators from caseworkers. The algorithmic tool, in turn, was to transgress this infrastructural limit, connecting data from different welfare departments to generate a “complete” picture of risk. As a director in the municipality explained: “If we do nothing and if we do not use the data we already have (…) then I also think we have a problem as a public authority” (municipal director, interview, 21 April 2021).
With this problematization of data infrastructures, the municipality approached the issue of detecting children at risk by algorithmically merging data across multiple departmental data infrastructures, what they referred to as making a “systematic linking of data” across different jurisdictional welfare areas (internal document, application for free municipality, 2017). As they described in an internal document: “Data are obtained from relevant subject systems on both the group of vulnerable children and our control group as well as parents. Forty-four potential risk indicators are currently being operated across nine data sources” (internal document, meeting minutes, November 2018). Indicators could be “missing the dentist,” “parent status,” “employment status,” “earlier provided services” to the parents, “data on abuse,” “health data,” “address,” etc. Some of these data points could be extracted even before the child was born. Moreover, data perceived to allow early detection were emphasized. For example, the importance of including health data was promoted as an opportunity to “strengthen the model as it is data at the level of the child recorded from birth and onwards” (internal document, meeting minutes, 2018).
Configuring algorithmic decision generation
The imagined ability to predict uncertain futures established a human–machine relationship with a distinct temporal orientation. As the predictive algorithm was to detect children and preempt notifications, the predictive algorithm had a proactive role, framing families as possible sites of future maltreatment. As the municipality explained in the application for getting exempted from data protection legislation, “Data is collected to assist the identification of children or young people in the municipality who are located in the risk zone” (internal document, application for free municipality). The municipality emphasized that professional discretion would still be necessary: “We therefore want the model to be the basis for early attention where a professional assessment will always decide on further treatment” (internal document, public presentation, June 2019). Similarly, the municipal director explained in an interview: “the output is intended to be used by professionals before contacting the families […] In our choice of method, we wanted to ensure that the model was transparent so that the professionals could understand the output and use it in their work.” Instead of having a predictive algorithm support professional decisions after a notification was filed, the municipality aimed for a decision-generating system understood as generating decisions about initiating contact.
Discussion
The two cases illustrate how predictive algorithms construct risky futures as objects of intervention and how these risk constructions organize relations to citizens as risk objects. Arguably, both cases of predictive algorithms make risky futures into an object of governance and decision-making. Also, instead of changing governmental logics, they exacerbate and intensify existing risk logics. This is both the case with the research project's focus on risk assessments of notifications and the municipality's strategy for early intervention focusing on parents as risk. In this sense, the predictive algorithms cannot be understood independently of the broader ensemble of governmental strategies, procedures, or institutions surrounding their development. However, there are important differences in how their algorithmic models construct risk. These differences are summed up in Table 2.
Comparison of predictive algorithms.
The research project envisions predictive algorithms to support human risk assessments. Here, the algorithm is to sort, select, and evaluate data from child protection services’ information registers, already accessible to caseworkers and within the jurisdictional domain of child protection. With the ambition of rendering existing risk assessments of notifications more precise, the algorithm attributes children with a risk score, calculating their probability of future maltreatment. The Municipality, in contrast, endeavored to predict children's risk of maltreatment as early as possible, from the moment of their birth or even conception. In a sense, this disrupts the existing political initiative of enhancing preventive measures through a system of notifications. The very idea that symptoms of maltreatment can be preempted by algorithmically detecting children in a “risk zone” organizes a relationship to pure risk and mere possibility, severed from early indications of actual maltreatment.
This vision entails attributing risk to parents, as little or no information would exist on the child, requiring collection and merging of data about parents across different welfare sectors and allowing them to discover children that have not yet acquired existence as a “case” or a “notification” in the child protection services. We here see a contrast in the role delegated to the algorithm, i.e. taming the uncertainty in existing risk assessments (the research project) versus detecting (and possibly eliminating) risk by preempting maltreatment before its symptoms emerge. There are thus important differences in how the predictive algorithms, and the datasets that feed them, construct “risk.”
This difference is mirrored in the two different human–machine configurations. With the research project, the algorithm is reactive, to be used as decision support after a human concern has materialized. In contrast, the municipality's algorithm is proactive, profiling citizens as risk objects in advance of a professional concern. We may say that the two cases reveal different forms of anticipatory governance, understood as the casting of potential futures to guide organizational decisions (Flyverbom and Garsten, 2021; Guston, 2014). Whereas the municipality reveals the ambition to manage risk “while such management is still [perceived] possible” (Guston, 2014: 218), the research project seeks to intercept (possible) maltreatment in the making.
The algorithmic configuration of risk is neither apolitical nor neutral. As already noted, the municipality intensifies risk by constructing novel “risky” citizen objects. This “folds future possibilities into present decisions, making present the future consequences of events that may or may not come to pass” (Amoore, 2013: 75) in a much more radical way than the research project. The municipality's algorithmic profiling of children at risk thus expands the plane of risk governance by inventing new administrative practices and widening the scope of which citizens caseworkers can consider as “risky.” This model also has a different relationship to the past, as data about citizens automatically are pulled, regardless of the child protection services being notified, hence constructing all citizens as potential risk objects.
For both projects, turning professional and political discourse about risk management into algorithmic prediction has implications. Following Power (in Munro, 2010: 1145), the notion of risk “implies a domain for decision making about the future and a corresponding allocation of responsibility for that decision.” However, Munro (2010) suggests that child maltreatment and well-being are complex phenomena that cannot be predicted. Sometimes good interventions lead to poor outcomes and vice versa. Thus, it is a complex field with no causal relationship between decisions and outcomes. Yet, the language of risk turns poor child intervention outcomes into a “failure of risk management” (Dekker in Munro, 2010: 1142). In the UK, this perception has led to a political focus on standardized “best practices,” often implemented with the help of digital information systems. This development, however, leads to overvaluing standardized and measurable aspects of practice, while undervaluing professional expertise and judgment and may thus paradoxically hinder organizational learning and improvement of child protection (Munro, 2010). This speaks to Eubanks (2018) who documented algorithms shifted decision-making authority away from caseworkers and toward the technology or its developers. Thus, even if the research project emphasizes the decision support as an augmentation of professional expertise, if implemented, it may lead to redistribution of accountability and an amplified focus on risk management. According to Munro (2010), this may promote excessive control and proceduralization of professional practice at the expense of professional learning.
Conclusion
In this article, we have compared two predictive algorithms in child protection services, eliciting their constructions of risk by examining their problematizations, data infrastructures, and human–machine configurations. While algorithmic risk constructions mobilize distinct relationships to the future, this relationship is contingent upon the datasets analyzed to profile citizens at risk. Indeed, how an algorithm constructs risk as an object of intervention, and what kind of human–machine configuration it compels, differs with respect to both problematizations and choice of datasets. As summarized above, our analysis elicits two rather different algorithmic risk constructions, respectively, as a mode of uncertainty absorption (the research project) and as a new technique for risk detection (the municipality).
The predictive algorithms’ risk constructions are political: they organize populations, risk categories, institutions, data infrastructures, and administrative procedures by promising to know uncertain futures. As nonimplemented, and to some extent, failed experiments, our two cases are interesting in a contemporary debate about what a governance of citizens’ uncertain futures can look like. They reveal two different imaginations of how public sector authorities are to relate to citizens through algorithmic prediction. Is the role of the state to preempt and anticipate problems such as children's maltreatment before they emerge and to use algorithmic power to intervene in children's lives perhaps even before they are born? As Amoore (2013) notes, such a government rationality is “indifferent to whether a particular event occurs or not. What matters instead is the capacity to act in the face of uncertainty, to render the gaps in what can be collected or known actionable” (p. 62). Or is the role of the state to deploy predictive algorithms to support existing risk assessments? Is the “timely intervention” to occur as early as possible, even before symptoms of maltreatment in the child have materialized? Is “timely intervention” the legally defined timeframe for assessing notifications, representing the concerns discovered by other welfare professionals, but also problematized as arriving much too late? These specificities come into view through detailed comparison. They also speak to broader questions of the possible horizons of governing risk enabled by predictive algorithms. Political questions about how or whether the state should use algorithms to turn uncertain futures into an object of preemption and management are also a question of future politics, of what relationships we should have to our uncertain futures. In this respect, it is important to examine whether predictive algorithms, with their promise of knowing the future, also increase accountability and proceduralization, at the cost of professional discretion and learning.
As extant literature has shown, predictive algorithms close an open future when patterns found in past data are used to anticipate future events. Our comparative case study echoes these findings, as machine learning elicitation of patterns in historical datasets was imagined to manage risk by predicting the future. At the same time, they also demonstrate a varying landscape with different configurations of risk as a contrast between preemptive detection and decision support. Thus, there is a difference in scope when risk is inferred from limited datasets on children notified to the child protection services compared to profiling all citizens residing in a municipality. Our point, then, is that algorithmic prediction, and their constructions of risk, needs to be studied contextually rather than in an abstract form. We have proposed attending to problematizations, infrastructural data arrangements, and human–machine configurations to elicit such specificities. At a point in time where modern governments are reinventing themselves in the face of proliferating AI techniques such as predictive algorithms, it is crucial to explore the particularities and situated contexts of those anticipatory logics. To understand how predictive algorithms and other technologies of risk become part of public administration, we need to be open to the coexistence of multiple algorithmic risk constructions. By acknowledging and disentangling these issues, we, as researchers and citizens, hope to better participate in ongoing experiments regarding the extent to which we are willing to live with and be analyzed by algorithms. This includes fundamental choices such as keeping our futures open versus understanding them as algorithmically constructed risks.
Footnotes
Acknowledgments
We wish to thank the editor and reviewers for helpful comments in improving this article. We are grateful to Minna Ruckenstein and her research group as well as Jakob Laage-Thomsen, Ida Schrøder, and Louise Jørring for helpful comments. We also thank our interlocutors in the two cases who participated in and made the research leading to this article possible.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding from the VELUX Foundations (Algorithms, Data & Democracy Jubilee Grant) supported the writing of this article (co-principal investigator, Helene Friis Ratner, Aarhus University).
