AI failure loops in devalued work: The confluence of overconfidence in AI and underconfidence in worker expertise

Abstract

A growing body of literature has focused on understanding and addressing workplace artificial intelligence (AI) design failures. However, past work has largely overlooked the role of the devaluation of worker expertise in shaping the dynamics of AI development and deployment. In this paper, we examine the case of feminized labor: a class of devalued occupations historically misnomered as “women’s work,” such as social work, K-12 teaching, and home healthcare. Drawing on literature on AI deployments in feminized labor contexts, we conceptualize AI Failure Loops: a set of interwoven, sociotechnical failure modes that help explain how the systemic devaluation of workers’ expertise negatively impacts, and is impacted by, AI design, evaluation, and governance practices. These failures demonstrate how misjudgments on the automatability of workers’ skills can lead to AI deployments that fail to bring value to workers and, instead, further diminish the visibility of workers’ expertise. We discuss research and design implications for workplace AI, especially for devalued occupations.

Keywords

Devalued work AI design and evaluation feminized labor workplace AI technology and labor AI failures

Introduction

Workplace artificial intelligence (AI) systems are often touted by developers and deployers as tools to augment workers’ abilities and improve their work experience (Brynjolfsson et al., 2025). In practice, however, these systems have frequently fallen short of meaningfully supporting workers (Acemoglu et al., 2023). In social services, AI-based decision support tools have been introduced for high-stakes decisions, only to be withdrawn following backlash from frontline workers (Samant et al., 2021). In healthcare, clinicians ignore AI tools designed to assist in clinical decision-making (Elwyn et al., 2013; Lebovitz et al., 2022). In education, AI tutoring systems have added to teachers workloads and compromised their pedagogical autonomy—despite promises to reduce labor and enhance teaching (Holmes, 2023; Holstein et al., 2017; Modén et al., 2021). Across these sectors, AI systems fail to complement the situated, social, and tacit knowledge that underpins workers’ expertise (Lebovitz et al., 2021).

Prior research in human–computer interaction (HCI) and science and technology studies (STS) offers several explanations for these failures. Scholars have pointed to issues of poor contextual fit, where AI systems conflict with existing organizational structures and work practices (Suchman, 1987; Yang et al., 2019). Others have highlighted flawed assumptions about the nature of work, misaligned values, or inadequate governance, emphasizing how these design shortcomings intersect with broader labor and power dynamics (e.g. Forsythe, 1993; Kawakami et al., 2024a; Star, 1999; Suchman, 1993). However, one crucial dimension remains underexplored: occupational devaluation, the systemic undermining of worker expertise, particularly within professions predominantly occupied by women and people of color (Flores-Robles and Gantman, 2024). In these professions, expertise is often not fully understood or explicitly recognized by others (Huising et al., 2025), and devalued both in status and pay even when accounting for education and experience levels (Flores-Robles and Gantman, 2024).

How does the devaluation of worker expertise shape AI design, evaluation, and deployment? Current literature offers little insight into how this dynamic contributes to the repeated failure of workplace AI to genuinely enhance labor. This paper addresses this gap by placing occupational devaluation at the center of our analysis. We conduct a focused review of academic and gray literature on AI deployments across three domains of feminized labor in the United States: K-12 education, home healthcare, and social work.

Feminized labor has long been devalued in the United States. Such work often involves emotional and care-oriented labor (England, 2005), is historically labeled as “women’s work,” and continues to be predominantly performed by women and people of color (e.g. Connacher, 2023; Council on Social Work Education, 2023; Taie and Lewis, 2022). These jobs tend to be low status within organizations, with limited worker influence over decision-making (Cotter et al., 2001). Workers in these fields frequently report that their experience is misunderstood by those in more powerful roles (e.g. York, 2022).

We argue that workers engaged in devalued labor are especially vulnerable to flawed AI deployments, which often rest on reductive understandings of their work. These systems not only fail to reflect the complexity of frontline labor but can also further obscure and erode the visibility of workers’ expertise, creating a negative feedback loop that deepens occupational devaluation. Through a comparative analysis (Schwandt and Gates, 2018) of past scholarship on AI deployments across three feminized labor contexts (social work, home healthcare, K-12 teaching), we articulate why and how this dynamic emerges.

To do so, we introduce the concept of AI Failure Loops: recurring patterns of failure in the design, evaluation, and deployment of workplace AI systems, in which the devaluation of worker expertise leads to ineffective AI deployments—and those failed deployments, in turn, further erode recognition of workers’ skills. These loops are sustained across the AI development lifecycle, fueled by persistent over-estimation of what AI can do and under-recognition of the complexity and value of worker expertise.

Some of these challenges are familiar: they echo Forsythe’s observations of expert systems nearly three decades ago (Forsythe, 1993). When developing expert systems, AI developers excluded social and maintenance activities in their conceptualizations of “work,” leading to AI deployments that failed to reflect experts’ real-world work (Forsythe, 1993). Our paper examines how similar challenges have persisted with modern AI, with a focus on the role of occupational devaluation.

Today, with recent waves of both AI hype and actual increases in AI capabilities, developers are targeting a rapidly expanding range of socially complex tasks associated with feminized occupations. Drawing lessons from past AI deployments in these contexts, we can expect this will further accelerate the gross underestimations of worker expertise and overestimations of AI capabilities that has shaped these occupations in the past three decades. This paper responds to calls to redirect this trajectory toward pro-worker futures (e.g. Acemoglu et al., 2023), where worker expertise is celebrated within AI practice and society more broadly, and workplace AI systems are designed to truly complement worker capabilities. By centering occupational devaluation in our analysis, we extend these calls, highlighting the need for AI systems that not only avoid harm but actively contribute to the dignity and visibility of labor.

Addressing challenges in the deployment of workplace technologies

Researchers in STS, computer-supported cooperative work (CSCW), and organizational science have long studied challenges in the deployment of workplace technologies. Workplace technologies are often designed or introduced by managers in ways that fail to account for the social contexts within which they are deployed (Suchman, 1987). For instance, those in hypercompetitive workplaces (Orlikowski, 1992) may lack incentives to use collaborative technologies (Star, 1999). Computing systems can also fail to be used in practice due to workers’ limited visibility into systems and the system’s limited visibility into workers’ contexts (Suchman, 1987). The design of workplace technologies is also highly political—it can prescribe certain ways of working, thus decreasing worker autonomy (Suchman, 1993). In other words, workplace technologies can, in the Foucauldian sense (Foucault, 2023), discipline and punish (Suchman, 1993).

A growing body of literature has examined how so-called “AI” systems extend these trends and present new challenges (e.g. Holmes, 2023; Saxena et al., 2021). As modern AI systems expand the scope of tasks subject to automation, frontline workers find themselves performing new forms of hidden integration labor or “patchwork” (Fox et al., 2023) to bridge the gaps between what workplace AI systems purport to do and what they are actually able to do (Fox et al., 2023; Mateescu and Elish, 2019; Sendak et al., 2020). Prior literature has discussed how workers across occupations, from social workers and judges to radiologists, frequently ignore AI-based decision support tools due to inconsistencies with their decision-making practice and challenges around interpretability (Kawakami et al., 2022a; Lebovitz et al., 2022; Pruss, 2023). These challenges are further exacerbated by flawed evaluations, where workplace AI systems intended to augment or automate complex human decision-making are built upon imperfect “ground truth” labels, leading to overclaims of model performance (Lebovitz et al., 2021).

AI researchers are increasingly invoking the language of “participation” to support more responsible technology creation (Sloane et al., 2022). Such calls have roots in Scandinavian efforts to democratize workplace technology development in the late 20th century (Muller and Kuhn, 1993). These participatory design engagements were facilitated by strong labor protections and partnerships between labor unionists and Marxist computer scientists (Spinuzzi, 2002). However, worker participation in AI development remains rare in practice, particularly within the United States where the sociopolitical conditions differ from the Nordic model of industrial democracy. Even when stakeholders are involved, workers are rarely empowered to substantially shape the design process, and familiar challenges often recur once technologies are deployed (Delgado et al., 2023; Kawakami et al., 2024a).

We build on existing literature on workplace technology challenges, using the systemic devaluation of worker expertise as a lens to understand why some AI deployments fail to truly support workers, even in cases where developers intend to design worker-complementary AI.

Devaluation of feminized labor

Notions of what counts as “work” and, relatedly, expertise are socially situated and have evolved across time (Star and Strauss, 1999; Suchman, 1995). These notions are used to justify the societal and economic (de)valuation of human labor. Feminized labor—which often involves care-oriented labor like teaching, home healthcare, or childcare—provides a particularly striking example. Many forms of feminized labor were historically not viewed as “work,” but as acts of love (Flores-Robles and Gantman, 2024). Even as feminized labor became formalized into paid occupations through social movements, workers struggled to gain legitimacy. Sociologists have theorized several explanations: Employers may justify low pay by conceptualizing feminized labor as “acts of love” provided out of “goodwill” (the “prisoners of love” framework; England, 2005). The association of feminized labor with women may also lead to its devaluation precisely because women themselves are devalued in society (the devaluation theory) (England, 2005).

Even when accounting for factors like workers’ education level, occupations with a higher percentage of women pay less and have lower social status (Flores-Robles and Gantman, 2024). The most female-dominated occupations in the United States still tend to involve care-based work, including home care (85%), social work (85%), and teaching at the primary and secondary levels (77%) (Connacher, 2023; Council on Social Work Education, 2023; Taie and Lewis, 2022). Workers in feminized occupations tend to have low status in their workplace, with little say in organizational decisions impacting their day-to-day work—challenges which are exacerbated by a “glass ceiling” on career advancements (Cotter et al., 2001). These occupations often also face systemic resource shortages and high rates of worker burnout, all amid growing labor demands (England, 2005).

In the context of AI in feminized occupations, prior work has quantitatively analyzed labor automation trends, and found that many feminized roles—particularly those involving emotional and care-oriented labor—are projected to be among the most “immune” to future automation (e.g. Filippi et al., 2023; Gilman, 2018). Yet, these studies leave open how day-to-day labor dynamics shape these trends, and what they mean for worker practices and AI. Separately, a growing body of work documents AI failures in specific work contexts (see Section “Addressing challenges in the deployment of workplace technologies”). Our paper bridges these lines of inquiry, focusing on AI in feminized occupations in the United States. Through case studies across multiple fields of feminized labor, we introduce the concept of AI failure loops to trace how the systemic devaluation of worker expertise both drives and is intensified by AI practices.

Methods

To understand how the devaluation of worker expertise shapes AI practices, and vice versa, we conducted a focused review of academic and gray literature on AI deployments across three feminized occupations (social work, home healthcare, K-12 teaching), with analyses informed by our team’s direct empirical experiences. In this section, we elaborate on how this experience informed our case selection (Subsection “Team experience and case selection”) and the literature we reviewed in our analysis (Subsection “Literature review and analysis”).

Team experience and case selection

We examine case studies of AI deployments across social work, home healthcare, and K-12 teaching, selected because our team has direct experiences collaborating with workers in each occupation. This experience—collectively including over 10 years of work observing worker practices, designing interventions with workers, and speaking with leadership and managers to facilitate these collaborations—allowed us to ground our analysis of relevant literature in our own empirical material (e.g. field notes). For example, our social work case study was informed by our prior findings from field research conducted across five US public sector agencies (Kawakami et al., 2022a, 2024a). This includes experiences observing workers’ use of AI-based risk assessment tools (Kawakami et al., 2022a), speaking with agency leadership and in-house AI developers in social work agencies (Kawakami et al., 2024a, 2024b), and designing interventions with social workers (Kawakami et al., 2022b). Our healthcare case study was informed by our experiences conducting interviews and design activities with home health aides, aide coordinators, and nurses to understand their perspectives around workplace AI (e.g. Bartle et al., 2022). Our K-12 teaching case study was informed by our prior field research studying K-12 teachers’ adoption and use of AI-based tutoring systems across hundreds of US-based schools and classrooms (e.g. Holstein et al., 2017, 2018, 2019).

Literature review and analysis

Our analysis is based on prior literature on AI deployments across the three case studies (social work, home healthcare, and K-12 teaching), each drawn from prior academic and gray literature (including but not limited to prior research from our team). Specifically, as we conducted a comparative case analysis (Schwandt and Gates, 2018) of AI deployments across these occupations, we began to notice common trends, which we explored further through a focused review of academic and gray literature. We gathered and analyzed literature from computing (e.g. HCI, robotics, AI in education (AIED)) and domain-specific (e.g. healthcare, social work, and education) research venues. We searched literature on field deployments of AI systems in workplace contexts (e.g. Jørgensen and Nissen, 2022), literature reviews on technologies deployed in specific workplaces (e.g. Kuo et al., 2022), and syntheses of trends on the impacts of AI on workers (e.g. Holmes, 2023). Our analysis was not limited to research on challenges from AI deployments. Through our initial comparative case analysis of findings across the authors, and this subsequent review of the literature, we iteratively developed the AI failure loops concept.

We next present the concept of AI failure loops (Section “AI failure loops”), then describe our three case studies (Section “Case studies”) to illustrate how AI failure loops unfold in practice.

AI failure loops

AI failure loops arise at the confluence of two dynamics: overconfidence in AI capabilities on the one hand, and underconfidence in worker capabilities on the other. We conceptualize AI failure loops as negative feedback loops that occur when the devaluation of worker expertise introduces or reinforces threats to the creation of useful worker-facing AI tools, in turn leading to the deployment of worker-facing AI tools that further amplify harms from worker devaluation. Based on prior literature discussed in the Section “Case studies,” we identify six interwoven AI failure modes that shed light into how this can happen:

Expertise Misunderstanding: AI tools in devalued occupations are often motivated by misunderstandings of workers’ expertise. When designing, evaluating, or deploying AI, this may manifest in workers having their expertise questioned.

Managerial Over Worker Needs: AI tools in devalued occupations are often designed to meet broader organizational goals and targets, at the expense of worker needs.

Design Exclusion: Workers in devalued occupations tend to be excluded from the design process. If they are involved as one of multiple stakeholders in a design process, their perspectives may be considered a second priority to other stakeholders.

Inappropriate Evaluation: Evaluations of AI tools in devalued occupations tend to be designed in ways that advantage AI performance and disadvantage workers, further promoting misunderstandings of their expertise and the nature of their work.

Forced Use: Workers in devalued occupations often do not have the power to refuse technology use.

Unwarranted Blame: Workers in devalued occupations tend to receive unwarranted blame or responsibility for shortcomings or harms observed from AI systems during deployment.

Figure 1 provides a high-level visual overview of AI failure loops. As illustrated by the web, the six failure modes are mutually reinforcing, where the presence of one failure mode can amplify the likelihood of others. As we will see in the Case Studies Section, devaluation and misunderstanding of worker expertise are at the core of these dynamics. In our first case, we will show how misunderstandings about how social workers make child maltreatment screening decisions (expertise misunderstanding) led to the use of AI evaluation measures that were fundamentally misaligned with their decision-making goals (inappropriate evaluation). Then, we will describe how home health aides are systematically excluded from the design of new technologies (design exclusion), in part, because they are made invisible by organizational leaders who did not perceive them to be an integral part of the healthcare system (expertise misunderstanding). Instead, “higher-status” healthcare professionals’ and institutional needs and goals were prioritized in design efforts (managerial over worker needs). Finally, we will show how K-12 teachers’ exclusion from adoption decisions of new AIED technologies (forced use) disincentivized AI developers from understanding teachers’ work practices, limiting their ability to design AI tools that effectively complement teachers (expertise misunderstanding). When deployed classroom technologies fall short of achieving their envisioned potential, AI developers and school districts often attribute these shortcomings to teachers’ purported incompetence (unwarranted blame), with inadequate consideration for potential technology design issues.

Figure 1.

A visual overview of AI Failure Loops. The six failure modes (in circles) that contribute to the dynamics of the AI Failure Loops exist within a web to illustrate the inter-connected relationship amongst the failure modes.

Case studies

To illustrate the concept of AI failure loops, we present three case studies of AI deployments in feminized labor, each emphasizing different failure modes: AI-based risk assessment tools in social work, AI for home healthcare, and AI tutoring systems in K-12 teaching. We provide additional context on each occupation in Table 1.

Table 1.

Additional historical and social context on the three feminized occupations presented in our case studies (social work, home health, and K-12 teaching in the United States), drawn from existing academic and gray literature.

Origins	20th century	Today
Social work
Founded in 1880s by women organizing settlement house movement, a reformist social movement dedicated to bridging resource gaps between rich and poor families (York, 2022).	Formalized into profession, with requirements for Master’s training and state licensing to practice in most US states (York, 2022).	Social workers continue to be predominantly women and disproportionately Black or Hispanic (Council on Social Work Education, 2023). Continued worker devaluation: Paid relatively little compared to other fields requiring similar levels of education (soc, 2022); workers voice they feel their expertise is misunderstood and devalued in their workplaces and society (York, 2022).
Home health
Considered part of domestic life for most families in early 19th century, while professional home care was reserved for wealthy families. By late 19th century, women began organizing efforts to send nurses to homes of poorer families (Buhler-Wilkerson, 2003).	The Fair Labor Standards Act of 1938 outlaws exploitative practices and instituted an 8-h workday but home health aides are explicitly excluded (Migda, 2015).	Home health aides continue to be predominantly women and disproportionately Black or Hispanic (Connacher, 2023). Continued worker devaluation: Majority of workers lack benefits like paid time off, sick leave, or employer-sponsored health insurance (Banerjee et al., 2025); majority of workers live below poverty line (Connacher, 2023); workers often not seen as part of healthcare team, despite being integral to healthcare system (Osterman, 2018).
K-12 teaching
Originally conceived as male-dominated occupation that men partook in during farming off-seasons or prior to moving into more well-established professions. Women begin to enter the profession in the late 1880s (Rury, 1991).	Profession is predominantly occupied by women, except among administrators (Smith, 2022). Profession becomes increasingly formalized with state licensing and university education program requirements (tea, 2018).	K-12 teachers continue to be predominantly women, except in leadership positions (e.g. principals, superintendents) which continue to be male-dominated (Kocon, 2017). Continued worker devaluation: Paid relatively little compared to other fields requiring similar levels of education (Flores-Robles and Gantman, 2024); workers report high levels of stress and burnout (Steiner and Woo, 2021).

AI-based risk assessment tools have amplified misconceptions of social workers’ expertise

Context

To address widespread staffing and resource shortages, social work agencies across US state and local governments have started to deploy AI-based risk assessment tools in areas like housing allocation for unhoused individuals and screening for child maltreatment cases. One prominent example of such a tool is the Allegheny Family Screening Tool (AFST), a predictive optimization-based (Wang et al., 2024) tool intended to assist social workers in making screening decisions around potential child maltreatment cases. The model is trained on historical administrative datasets to predict the likelihood of a child being mistreated, using the likelihood a child will be removed from their home within 2 years as an imperfect proxy. Given administrative data on a particular family, the AFST outputs predictions in the form of a risk score between 1 and 20. This class of AI-based risk assessment tool has been adopted across other agencies in the United States, New Zealand, and Denmark, and applied to other social work decision-making tasks (e.g. foster care placements, temporary lodging placements). In initial years of deployment, the social work agency received both widespread praise and criticism for its deployment of the AFST. On the one hand, the agency was lauded for publishing public-facing evaluations on the ethical implications and technical performance of the model (e.g. Vaithianathan et al., 2017). On the other hand, academics, advocates, and journalists heavily criticized the agency for deploying an AI model with biased performance against Black and disabled communities (AP News, 2023; Gerchick et al., 2023) and poor construct validity (Coston et al., 2023), with some calling to end the use of such tools.

AI failures in social work

We turn our attention to how the kinds of AI design and deployment challenges discussed in these prior critiques may have been influenced by how workers in this domain are perceived and treated by agency leadership and AI developers. This section revisits prior literature on the design and use of the AFST, using AI failure loop as a lens.

AI tool design fundamentally misunderstands social workers’ expertise and overlooks their needs

The social work profession has a “glass ceiling,” where workers have minimal opportunities to advance into organizational decision-making roles. As a consequence, there are often large silos between agency leaders and social workers, so those making decisions impacting social workers may have limited procedural understanding of the expertise and on-the-ground knowledge social work requires (Holten Møller et al., 2020; Kawakami et al., 2022a; Saxena et al., 2021).

In the AFST case, the agency issued an open call to create a new tool to support child maltreatment call screeners in making more efficient decisions. Despite social workers being the main users of the tool, the tool creators did not treat them as stakeholders in the design process. For example, social workers’ perspectives on the AI tool were collected only after the model was already developed, using narrowly focused surveys confirming their comprehension of a brief training on the tool, while limiting broader feedback on the tool’s overall problem formulation and design (Kawakami et al., 2022a).

As a result, both researchers and workers have since pointed to fundamental flaws in the AFST’s design, which may have been avoided had workers been meaningfully involved in the design process (Gerchick et al., 2023; Kawakami et al., 2022a). For example, the longer-term outcomes this AI tool was trained to predict (risk of out-of-home placement over the next 2 years) fundamentally conflicted with how workers were trained and legally-required to make decisions (i.e. based on assessments of immediate risks to the child). Therefore, once deployed, workers expressed concerns about the tool’s validity and uncertainty around how they were expected to interpret and use its outputs (Cheng et al., 2022; Kawakami et al., 2022a, 2022b).

More broadly, workers questioned the validity and usefulness of the tool’s overall formulation: statistical risk prediction based on government administrative records. Instead of presenting opaque predictions derived from the shallow information available in administrative data, workers believed it would have been more helpful to have AI support them in making sense of available evidence—leveraging their on-the-ground knowledge and qualitative understanding of case-specific context as human experts (Kawakami et al., 2022b).

Developers interpret worker complaints around AI as signals of worker deficits

As illustrated above, social workers are often excluded from predeployment design and deployment decisions of AI tools that intend to support their work (Kawakami et al., 2024a). When workers are given opportunities to provide feedback on AI tools postdeployment, their complaints are often dismissed as an inability to understand a complex technology. For example, in the AFST case, one worker described the types of responses they would receive after submitting feedback forms on the AI tool’s risk score as follows:

The input does not feel like a two-way street. [We] are told why we’re wrong, and [that] we just don’t actually understand algorithms versus, like, maybe you [the workers] are observing something [our algorithm] missed, you know? [ $\dots$ ] In the past, it’s felt like when we’ve taken advantage of that function [to give feedback on the AFST score], it has kind of resulted in just explaining to you why the score is actually right. (Kawakami et al., 2022a)

Social workers are forced to use AI, even when they disagree

Social workers cannot simply refuse technology use (Kawakami et al., 2022a, 2024a; Saxena et al., 2021). Organizational policies govern how often they can apply discretion with the model. For example, social workers were required to use the AFST and were expected not to disagree “too often.” Workers feared that violating this expectation could jeopardize their employment, so they felt pressure to agree with the AI model, often against their best judgment (Kawakami et al., 2022a). Nonetheless, workers still exercised their ability to disagree with the AI on a “budget,” to try to do right by families under these constraints (Cheng et al., 2022).

Similar AI-based risk assessment tools have been deployed in criminal justice. For example, COMPAS assists judges in making recidivism risk assessments (Dressel and Farid, 2018). Despite some similarities between the risk assessment tools in social work versus criminal justice (e.g. excluding workers from the design process), we argue there are some notable differences. Given judges’ relatively higher-status and organizational power, they have greater ability to refuse technology use by “ghosting the machine” (Pruss, 2023), ignoring risk scores at their discretion or discarding the entire risk assessment model, without fear of employment consequences.

“AI for Healthcare” has left behind devalued healthcare experts

Context

Governments and private investors have thrown billions of dollars into healthcare AI innovations—and this number is only increasing each year (Esposito et al., 2016; Yang et al., 2019; Yildirim et al., 2023). Many of these innovations target home healthcare. Yet, today, there exist few, if any, technical innovations that actually support home health aides (Kuo et al., 2022)—workers, the majority of whom are women and people of color, who support elderly people and people with disabilities in everyday personal care (e.g. bathing, washing, monitoring health challenges) and household responsibilities (e.g. cooking, cleaning, shopping). Instead, existing technologies in home health have tended to automate worker skills (e.g. care robots; Esposito et al., 2016), manage worker practices (e.g. shift matching; Solano-Kamaiko et al., 2025), or provide support to other stakeholders like patients (e.g. interactive voice assistants; Bartle et al., 2022). For example, to help address the immense care work shortage, several countries—across Europe, North America, and East Asia—have heavily invested in exploring the potential of care robots (Esposito et al., 2016). In fact, care robots for home healthcare applications are among the main investments in robotics applications (Esposito et al., 2016). However, a growing body of work has recognized that, if not designed carefully, these systems may disempower workers and amplify challenges they face already (Fox et al., 2019; Persson et al., 2022; Tseng et al., 2020). These tendencies to decenter home health aides in AI innovation efforts—in home health and healthcare more broadly—has occurred amid increasing acknowledgment that home health aides are the cornerstone to systems of care, that demands for their labor are increasing, and that institutional support for their wellbeing and labor has remained stagnant (Connacher, 2023).

AI failures in healthcare

This failure to deploy technical innovations that support home health aides can be understood as a failure of existing efforts to consider the impacts of and on the devaluation of home health aides. This section revisits why research and design communities have failed to design AI technologies that genuinely support home health aides, borrowing AI failure loop as a lens.

Home health aides are rarely designed for

In the past decade, “AI for healthcare” has grown substantially as both an area of academic research and capitalist speculation. These efforts often center ways to better meet doctors’ needs. However, nurses and home health aides—who also play critical roles in healthcare—are rarely designed for or with (Kuo et al., 2022).

While healthcare services rely on collaboration among various professions—including home health aides, nurses, and doctors—most existing efforts to deploy AI technologies overlook this. Instead, the majority of AI for healthcare efforts in the past few decades focused on supporting doctors (e.g. physicians, radiologists, surgeons), for example, through new clinical decision support tools (Sendak et al., 2020; Yang et al., 2019, 2016). We argue that the decentering of home health aides in “AI for healthcare” efforts has happened, in part, due to misconceptions of the nature of their labor and organizational dynamics that further entrenched these misconceptions.

While home health aides provide many of the same services as nurses in healthcare institutions (Hittle et al., 2016), their labor is performed within their clients’ homes rather than in institutions. Prior literature has described this physical isolation from healthcare institutions has made their labor less visible to other healthcare professionals and policymakers, and created associations with informal home care work (Cho et al., 2023), challenging the profession’s perceived legitimacy along the way.

These misconceptions of home health aides’ expertise, and along with it, their systemic devaluation, are also observed through how home health aides are treated within healthcare institutions. In Improving Long-Term Care by Finally Respecting Home-Care Aides (Osterman, 2018), Osterman recalls a conversation with a nurse: “no one pays attention to home-care aides $\dots$ . When something changes they notice but are not taken seriously.” These misjudgments may not be explicitly acknowledged nor understood. For example, Osterman recalls conversations with other healthcare professionals and stakeholders, observing:

other medical professionals, home care agency executives, directors of nursing homes, insurance company executives, and state and federal policy makers, I consistently heard comments that began with a bow to what wonderful people direct care workers are, and then went on to demean their skills and potential. (Osterman, 2019)

Even as decision-makers in healthcare (e.g. doctors, hospitals, state regulators, insurance companies) verbally acknowledge home care aides, they “typically do not think that home care aides can be real members of a care team” (Osterman, 2019). Osterman describes this happens, in part, because home care aides’ skills are overlooked, instead misunderstood as that of “unskilled companions [and] glorified babysitters” (Osterman, 2019). Ming et al. observe that home care aides recognize these misjudgments: “Some people see us as a maid [ $\dots$ ] We’re not just here to clean your house [ $\dots$ ] Maid are different from an aide” (Ming et al., 2023). Ming et al. (2023) further caution that failing to account for the invisibility of aides’ work may contribute to the deployment of AI systems that enforce compliance to impoverished notions of “care”—which, in turn, may further devalue and make invisible home health aides’ work.

In fact, most existing technologies intending to support home care agencies have geared toward supporting higher-status workers in home health agencies (e.g. doctors, nurses), rather than directly helping home health aides. A literature survey from 2022 found only nine studies contributed technologies to support home health aides (Kuo et al., 2022). Of these, the majority (7/9 studies) were technologies where home health aides were only one of several potential users. The authors found no studies with a long-term evaluation on how the technology impacted home health workers.

Existing technology “for” home health aides have failed to emulate or enhance their work

Researchers and policymakers have long looked for ways to sustain the growing population of elderly people—much of which is currently supported through home health aides (Bedaf et al., 2015; Hendry and Wu, 2004). While there has been limited research on technologies to support home health aides, there is a comparatively large number of papers exploring ways to automate home-based tasks needed by elderly people, including tasks home health aides usually complete (Esposito et al., 2016). Many of these technical deployments have come in the form of care robots. However, despite the massive amounts of funding dedicated toward advancing the design of care robots, actual deployments of care robots have often been deemed failures (e.g. Persson et al., 2022; Sharkey and Wood, 2014). Most of this literature has discussed failures with respect to meeting elderly people’s care needs.

While care robots may have been intended to help alleviate the increasing demand for home care workers, the few studies that account for impacts on care workers have observed impacts in the opposite direction. For example, in a scoping review of 27 papers on the deployment of care robots, researchers concluded the effectiveness of care robots is “mixed” at best. It further observed how robots can instead increase caregivers’ workload: care robots “seldom work as a shortcut to increased efficiency or time effectiveness” as workers must undergo the labor of maintaining and operating the robot (Persson et al., 2022). Another study found care workers experienced emotional demands, grappled with new ethical challenges, and felt decreased job satisfaction when care robots were deployed in their workplace (Wright, 2018, 2019).

Beyond care robots, Solano-Kamaiko et al. (2025) discuss similar challenges with other forms of home health technologies, like AI-based shift management technologies that match home health aides with patients. Home health aides reported feeling concerned about how matching algorithms fail to account for the human dimensions of their work, like personality, compassion, and empathy, that impact the quality of matching decisions (Solano-Kamaiko et al., 2025). Others described how AI-based shift management technologies may further increase the invisibility of home health aides’ work by taking away the already minimal amount of contact they have with home health agencies (Solano-Kamaiko et al., 2025).

Institutional power imbalances incentivize involvement of higher-status workers in AI design

Home health aides are rarely consulted during the design process, even for technologies directly impacting them (Kuo et al., 2022). This creates a stark contrast with the development of technologies that support doctors. In this section, we briefly discuss how having organizational power as a “high-status” worker can create incentives to avoid the AI failure loop. In doing so, we shed light onto the power and labor dynamics at play in low-status occupations, that lead to deviations in how they are impacted by AI deployments.

The HCI, machine learning and medical research and development communities have long been interested in innovating new AI tools to support medical decision-making. Even with decades of attempts, successfully deploying AI tools into real-world healthcare contexts has remained a challenging task (Yildirim et al., 2023). Once deployed, the majority of AI innovations have failed to bring value to doctors (Yang et al., 2016). Much like in social work, early versions of decision support tools were often created in isolation from the professional decision-makers themselves: medical experts were not consulted during the design process. Academics and hospitals found that, even after spending millions of dollars and decades of research innovating on medical decision support tools, doctors were simply ignoring them once they were deployed in practice (Yang et al., 2019). This ability to ignore the AI tool—to have full agency over decisions to ignore or disagree with an AI tool—incentivized researchers and developers to consult doctors in future AI innovation attempts.

We argue that doctors’ refusal to use AI innovations incentivized researchers and developers to more closely examine why doctors were not adopting them (e.g. Wyatt and Altman, 1995; Yang et al., 2019). To ensure they were creating AI tools doctors would actually use once deployed, the academic and developer communities may have been more incentivized to account for doctors’ needs and desires (as evidenced by the gradual surge of academic papers documenting medical experts’ needs (e.g. Cai et al., 2019; Yang et al., 2016) and involvement in AI creation (e.g. Sendak et al., 2020; Yang et al., 2019, 2016)).

In comparison, home health aides, like social workers, are not as well-respected in the healthcare system, making it less likely for them to have agency over technology deployments like the ability to refuse technology use. Moreover, because postdeployment evaluations of home health aides’ perceptions of AI tools are comparatively scarce (e.g. Kuo et al., 2022), their negative impacts on workers may also be more likely to go unnoticed.

Persistent visions of automation in education have further devalued teachers’ expertise

Context

Researchers and developers have long explored ways to introduce new technologies into K-12 education, with hopes to transform the schooling experience. The vision behind AIED research is that every student will be able to work at their own pace and get the personalized support they need (Holmes, 2023; Watters, 2015). However, these technologies have often been designed with a focus on supporting student learning, without considering the needs and practical constraints of the teachers who are asked to use them in their classrooms (Baker, 2016; Holmes, 2023; Holstein et al., 2017; Roll and Wylie, 2016).

AI failures in K-12 teaching

As in the prior two case studies, we see that AI deployments in education have been designed and evaluated at the confluence of two persistent dynamics: policymakers, companies, and school districts have long lauded the potential of AIED technologies to radically transform K-12 education, while, at the same time, these groups have devalued and misunderstood K-12 teachers’ expertise. In this section, we examine how AI design, evaluation, and deployment practices have been impacted by these dynamics.

Designing to help students learn, at the expense of helping teachers teach

There is a long history of AI development in education guided by the ultimate vision of automating away the role of human teachers (Selwyn, 2016; Watters, 2015). Over the past 40 years, research and practice in AIED has largely centered students’ needs, often without explicit consideration for teachers’ needs and constraints in the classroom (Baker, 2016; Holmes, 2023; Holstein et al., 2017; Roll and Wylie, 2016). Holmes refers to these deployments as “student-focused AIED”—AI technologies aiming to automate one or more of teachers’ tasks for the stated goal of helping students learn, for example, including tutoring systems, virtual writing assistants, automatic writing evaluation, and chatbots. By contrast, Holmes finds that “teacher-focused AIED” has remained largely speculative, confined to academic research prototypes rather than commercial systems (Holmes, 2023). Beyond the research community, commercialization practices help explain the prevalence of student-focused AIED technologies at the expense of teacher-focused AIED. Companies sell AIED technologies to schools and districts, and past research has found that teachers may be given little say in whether or not to adopt a given technology (Holstein et al., 2017; Modén et al., 2021).

By reducing K-12 teachers to mere service providers responsible for operating technology and maintaining the classroom (Holmes, 2023), many commercialized AIED deployments require teachers to “accommodate” to the technology. Indeed, prior work examining AI deployments observed how teachers often struggle to make AIED technologies work in their classroom, without having to adjust their practices in unreasonable ways or take on new responsibilities along the way (Enyedy, 2014; Modén et al., 2021). Observing math teachers using an AI tutoring systems in their classrooms, Holstein et al. and Utterberg Moden et al. found that teachers were forced to grapple with numerous contradictions designed into the AIED system, eventually causing them to abandon the system. For instance, while the AI tutoring system aimed to help students work at their own pace, this ran counter to pedagogical practices around collaborative learning and discussion: teachers had a responsibility to maintain a cohesive classroom environment, where students were roughly synchronized in the content they were learning (Holstein et al., 2017; Modén et al., 2021).

Moreover, these failures to support teachers may be challenging to capture postdeployment because existing AI evaluation measures focus on improving student learning and achievement outcomes with little regard for teacher experiences. As was the case with AI deployments for social workers and home health aides, the emotional and physical labor involved in integrating and operating AI technologies into the workplace are not considered in assessments of new AI deployments.

“Teacher bashing”: Teachers receive unwarranted blame for failed AIED deployments

Despite the labor and skill required to make AIED technologies work in their classroom, teachers are disproportionately blamed for limited adoption and use of AIED technologies in classrooms (Holmes, 2023; Selwyn, 2016; Watters, 2015). Indeed, there is often an assumption that the reason AIED tools are not well-implemented is because teachers are too disinterested or unskilled to use the technology effectively. In their book, “Education and Technology: Key Issues and Debates,” Selwyn reflects:

As is often the case with debates over the “failures” of education systems, “blame” for the restricted use of technology in schools, colleges, and universities has tended to be attributed most readily to the perceived shortcomings of teachers [ $\dots$ ] teachers have been deemed to be too old, disinterested, or incompetent to integrate digital technology [ $\dots$ ] It is all too easy for enthusiastic academic commentators to indulge in “teacher bashing” and portray teachers as outmoded, obstructive, or ignorant. (Selwyn, 2016)

Limited incentive to change AIED development practices

Prior work has documented instances in which teachers were able to abandon AIED deployments that failed to bring them value (e.g. Holstein et al., 2017; Modén et al., 2021). However, even when teachers unite to refuse technology use, they are not guaranteed an escape from AI failure loops longer term. As discussed in Sections “Social workers are forced to use AI, even when they disagree” (judges) and “Institutional power imbalances incentivize involvement of higher-status workers in AI design” (doctors), higher-status workers may be able to simply ignore the outputs of an AI tool, creating natural incentives for future AI developers to account for their needs and desires in future AI creation attempts. On the other hand, lower-status workers like teachers may manage to escape from an instance of a failed AI deployment, but these escapes alone are unlikely to prompt meaningful improvements in AI development practices. As discussed above, school districts and developers may still disproportionately blame teachers for low adoption and use of AIED technologies missing opportunities to identify ways to design AI technologies that genuinely support teachers.

Discussion: toward pro-worker AI

In this section, we discuss implications for researchers and practitioners to support the study and development of pro-worker AI. We begin with three critical questions (Sections “(How) can design processes manipulate power?”, “What happens after worker-centered design?”) to guide future research, policy, and practice toward a pro-worker future for AI and society. Each question raises implications to help speak back against currents that devalue worker expertise through AI practices. Drawing on lessons from our case studies, we explore how AI failure loops might be avoided by attending to the ways design processes account for power dynamics (Subsection “(How) can design processes manipulate power?”), the ways measurement can be used to make misleading claims about human versus AI performance (Subsection “(Why) do we need to measure human performance against AI?”), and opportunities to bridge the gaps between pro-worker AI in theory and in practice (Subsection “What happens after worker-centered design?”). We end with a discussion of unique considerations for the case of feminized labor (Section “The case of feminized labor”).

(How) can design processes manipulate power?

Our findings demonstrate how the design of workplace AI can reproduce and exacerbate occupational devaluation. The everyday work of devalued occupations, including but not limited to feminized labor, is often obscured from those in comparatively “high-status” positions in the same organizations (Blomberg et al., 1996). Yet, across our cases, AI developers routinely privileged the knowledges of “experts” to speak for devalued workers. Rather than involving social workers themselves, developers consulted social work academics (Section “AI-based risk assessment tools have amplified misconceptions of social workers’ expertise”); rather than involving teachers, developers consulted school district administrators (Section “Persistent visions of automation in education have further devalued teachers’ expertise”). In doing so, AI developers further perpetuated the invisibilization of feminized labor and the devaluation of workers’ expertise (Haraway, 2013). In failing to include—let alone center—social workers and teachers in the design process, developers created tools that did not work for these frontline workers.

Centering historically devalued workers in the design and research of AI systems poses unique challenges. Echoing long-standing concerns regarding gatekeepers in participatory design (Spiel et al., 2020), relying on managers to access workers may exert undue influence on research and design decisions. Instead of expecting leaders to open a pathway to involve workers, we encourage researchers and practitioners to buy workers’ time and form collaborations with existing labor unions and worker advocates (Spektor et al., 2023). Researchers should also critically examine worker participation in AI development. Workers may feel forced to participate (Suchman, 2011) or be included (Hoffmann, 2021) in AI design processes with which they disagree but cannot reject (Delgado et al., 2023). If devalued workers are involved as one of multiple stakeholders, treating their voice as “equal” to other “high-status” stakeholders may reproduce workplace power imbalances (Fox et al., 2019). Finally, we caution that managers and developers may recuperate the language of “participation” by using minor worker involvement to justify AI deployments, also known as participation washing (Sloane et al., 2022).

(Why) do we need to measure human performance against AI?

The measurements used to evaluate AI models are but one of many representations of the world (Boyd and Crawford, 2012). Our case studies demonstrate how the design of workplace AI evaluation metrics can further marginalize devalued workers. As we described in Section “AI-based risk assessment tools have amplified misconceptions of social workers’ expertise,” managers measured worker versus AI performance according to AI developers’ flawed predictive target. Even though this predictive target misaligned with workers’ underlying decision-making goals (Cheng et al., 2022), this comparative analysis was used to justify the AI tools’ deployment. Moreover, once AI tools were deployed, these flawed measurements were used to track and surveil workers’ compliance with AI outputs (Kawakami et al., 2022a).

Our case studies also strongly suggest that we should question the value of “measuring” workers in the first place. As noted by CSCW scholars, the question of what work to make (in)visible through datafication can be politically fraught (Star and Strauss, 1999). Researchers and developers may find that some work cannot or should not be quantifiable, leading to the development of alternative evaluation standards (e.g. worker satisfaction). However, even in occupations where there has been decades of research and development, as is the case with AIED technologies in K-12 education, the vast majority of evaluations for AI deployments have focused on understanding impacts on other stakeholders (e.g. students), without consideration for frontline workers (e.g. teachers’) experiences and perspectives (Holmes, 2023).

In contrast to current workplace measurement approaches, future research could explore the design of new approaches with workers that help make visible the (currently invisible) impacts of AI deployments on their labor. In these efforts, ensuring worker ownership over data collection and disclosure decisions is critical to ensuring that such measurements can actually enable workers to advocate for better technologies and working conditions (or the abandonment of technologies), rather than further surveil (Ming et al., 2023).

What happens after worker-centered design?

Our case studies also demonstrate a tension between the theory and practice of pro-worker AI research and design. As illustrated in the education case study (Section “Persistent visions of automation in education have further devalued teachers’ expertise”), institutions may be less incentivized to adopt or commercialize AI technologies that center devalued workers’ needs. This has led teacher-centered AIED tools to remain largely “speculative” (Holmes, 2023), where they exist as research prototypes but not as actual tools used in the classroom. Worker-centered AI tools that do get adopted by real-world organizations may still inadvertently reinforce worker devaluation through organizations’ governance practices. As AI failure loops explains, workers in devalued positions tend to receive unwarranted blame for unanticipated issues and harms that arise from AI postdeployment, even if they have minimal power to prevent them. If AI harms are only visible to workers, they may also not have power to refuse to use the technology. Even in efforts to deploy and, when appropriate, commercialize AI technologies, researchers and developers must continue to remain wary of—and actively correct for—the impacts of occupational devaluation.

Researchers can also support pro-worker AI through policy advocacy aimed at empowering workers to govern AI use and maintenance. For example, future work could design new approaches to involve workers in the creation of organizational policies that govern technology use—including to empower worker involvement in decisions around whether and when to decommission a technology. Future work could also explore ways to expose policymakers to the (often overlooked) impacts of technology on workers, to inform the creation of sector-specific worker-protecting regulations (e.g. Zhang et al., 2024). Finally, future work should continue to formalize mechanisms for worker collective action by, for example, designing new technologies that allow workers to organize around and share experiences with technology use (e.g. Irani and Silberman, 2013).

Finally, our case studies demonstrate the importance of critical research elevating workers’ voices and questioning the design of workplace AI systems. For example, the AFST research we described in Section “AI-based risk assessment tools have amplified misconceptions of social workers’ expertise” partially contributed to some US states stopping the use of similar child welfare screening algorithms in other US states (June, 2022). Likewise, the development of a child welfare risk assessment tool in Denmark was canceled due in part to public scrutiny fueled by critical scholarship (Ratner and Elmholdt, 2023). While researchers may not always have the power to ensure that workers are centered in the design of workplace AI systems, they can play a crucial role in contesting the proliferation of workplace AI systems that reinforce worker devaluation.

The case of feminized labor

Redirecting toward a pro-worker future for AI may involve some unique considerations in the context of feminized labor. For instance, many feminized occupations center on emotional and care-oriented labor that may be difficult or impossible to quantify, increasing risks of misquantifications in AI development. Moreover, workers in service-oriented positions (i.e. most feminized positions) are often trained to decenter themselves, becoming a “non-person” in their workplace and to employers (Forsythe, 1993; Ming et al., 2023). Even if AI developers manage to form genuine design partnerships with workers, they may not—without explicit encouragement—advocate for designing technologies that meet their own needs. In past efforts to design technologies with feminized workers (e.g. home health aides, K-12 teachers), these workers often centered clients’ needs and de-centered their own needs by default during design engagements, unless nudged by designers to do so (Bartle et al., 2022; Holstein et al., 2019). We anticipate that, with time, redirecting this trajectory will only increase in importance for AI in feminized occupations. When one considers that many of the most devalued occupations are dominated by women, the majority of whom are women of color, it becomes necessary to recognize the presence and prevalence of AI failure loops as a racial, gender, and social justice issue.

Conclusion

Visions for “transformative” workplace AI technologies that undermine workers are not new. Technologists have long envisioned that AI technologies would “fix” staff shortages in feminized occupations. The challenges described in this paper—the development of workplace technology without a grounded understanding of workers and their work—are reminiscent of observations made by Forsythe nearly three decades ago (Forsythe, 1993). Forsythe found that AI developers building expert systems held impoverished understandings of “work,” disregarding social and maintenance activities. Thus, these expert systems failed to model experts’ real-world work practices. Despite the shift from rule-based to machine learning systems, many of Forsythe’s observations persist today.

Since Forsythe’s writing, the scope and breadth of worker tasks that AI developers target have substantially expanded, from systems attempting to provide emotional support for K-12 students to predicting whether a child is at risk of future abuse or neglect. As this trend continues, our case studies provide a cautionary tale: historically devalued occupations are particularly vulnerable to AI failure loops. We expect recent increases in both AI hype and actual capabilities will exacerbate the overestimations of what AI can do and underestimations of workers’ expertise.

Critically, nothing about the future of AI and work is inevitable. To overcome AI failure loops, we must learn from past developments that perpetuated the devaluation of feminized labor and contest the societal and organizational devaluation of certain work. Re-imagining AI development practices to empower devalued workers may even help foster such cultural shifts. Just as AI failure loops are self-reinforced through negative feedback loops, reversing these trends may cultivate positive feedback loops between pro-worker AI practices and worker valuation. This paper extends discussions of how researchers, policymakers, and practitioners can help redirect toward a pro-worker future for AI and society, by drawing lessons learned from how worker devaluation has shaped AI practices, and vice versa.

Footnotes

Acknowledgments

We would like to thank the reviewers for their excellent reviews that helped strengthen our paper. We are also grateful for the thoughtful feedback shared by Alicia DeVrio, Jodi Forlizzi, Nari Johnson, Joy Ming, Tina Park, and David Widder.

ORCID iDs

Anna Kawakami

Kenneth Holstein

Funding

This work was supported by the National Science Foundation (NSF) Award No. 2442153, the NSF Graduate Research Fellowship (GRFP), the National Institute of Mental Health (NIMH) Award No. R01MH139114, the K&L Gates Presidential Fellowship, and the Google Academic Research Award.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Acemoglu

Autor

Johnson

(2023) Can we have pro-worker AI? CEPR.

AP News (2023) Child welfare algorithm faces justice department scrutiny. Available at: https://apnews.com/article/justice-scrutinizes-pittsburgh-child-welfare-ai-tool-4f61f45bfc3245fd2556e886c2da988b.

Baker

(2016) Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education 26(2): 600–614.

Banerjee

Gould

Sawo

(2025) Setting higher wages for child care and home health care workers is long overdue. Available at: https://www.epi.org/publication/higher-wages-for-child-care-and-home-health-care-workers/.

Bartle

Lyu

El Shabazz-Thompson

, et al. (2022) “A second voice”: Investigating opportunities and challenges for interactive voice assistants to support home health aides. In: Proceedings of the 2022 CHI conference on human factors in computing systems, pp.1–17.

Bedaf

Gelderblom

De Witte

(2015) Overview and categorization of robots supporting independent living of elderly people: What activities do they support and how far have they developed. Assistive Technology 27(2): 88–100.

Blomberg

Suchman

Trigg

(1996) Reflections on a work-oriented design project. Human–Computer Interaction 11(3): 237–265.

Boyd

Crawford

(2012) Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information, Communication & Society 15(5): 662–679.

Brynjolfsson

Raymond

(2025) Generative AI at work. The Quarterly Journal of Economics 140(2): 889–942.

10.

Buhler-Wilkerson

(2003) No Place Like Home: A History of Nursing and Home Care in the United States. Baltimore and London: JHU Press.

11.

Cai

Winter

Steiner

, et al. (2019) “Hello AI”: Uncovering the onboarding needs of medical practitioners for human–AI collaborative decision-making. Proceedings of the ACM on Human–Computer Interaction 3(CSCW): 1–24.

12.

Cheng

Stapleton

Kawakami

, et al. (2022) How child welfare workers reduce racial disparities in algorithmic decisions. In: Proceedings of the 2022 CHI conference on human factors in computing systems, pp.1–22.

13.

Cho

Toffey

Silva

, et al. (2023) To care for them, we need to take care of ourselves: A qualitative study on the health of home health aides. Health Services Research 58(3): 697–704.

14.

Connacher

(2023) Direct care worker pay and benefits are low despite high demand for services. Available at: https://www.americanprogress.org/article/direct-care-worker-pay-and-benefits-are-low-despite-high-demand-for-services/.

15.

Coston

Kawakami

Zhu

, et al. (2023) A validity perspective on evaluating the justified use of data-driven decision-making algorithms. In: 2023 IEEE conference on secure and trustworthy machine learning (SaTML). IEEE, pp.690–704.

16.

Cotter

Hermsen

Ovadia

, et al. (2001) The glass ceiling effect. Social Forces 80(2): 655–681.

17.

Council on Social Work Education (2023) 2022–2023 Statistics on social work education in the United States.

18.

Delgado

Yang

Madaio

, et al. (2023) The participatory turn in AI design: Theoretical foundations and the current state of practice. In: Proceedings of the 3rd ACM conference on equity and access in algorithms, mechanisms, and optimization, pp.1–23.

19.

Dressel

Farid

(2018) The accuracy, fairness, and limits of predicting recidivism. Science Advances 4(1): eaao5580.

20.

Elwyn

Scholl

Tietbohl

, et al. (2013) “Many miles to go

\dots

”: A systematic review of the implementation of patient decision support interventions into routine clinical practice. BMC Medical Informatics and Decision Making 13(Suppl 2): S14.

21.

England

(2005) Emerging theories of care work. Annual Review of Sociology 31(1): 381–399.

22.

Enyedy

(2014) Personalized instruction: New interest, old rhetoric, limited results, and the need for a new direction for computer-mediated learning.

23.

Esposito

Fiorini

Limosani

, et al. (2016) Supporting active and healthy aging with advanced robotics integrated in smart environment. In: Optimizing assistive technologies for aging populations.

24.

Filippi

Bannò

Trento

(2023) Automation technologies and the risk of substitution of women: Can gender equality in the institutional context reduce the risk? Technological Forecasting and Social Change 191: 122528.

25.

Flores-Robles

Gantman

(2024) Notions of care labor are antithetical to profitable labor. Psychology of Women Quarterly 48(4): 475–490.

26.

Forsythe

(1993) The construction of work in artificial intelligence. Science, Technology, & Human Values 18(4): 460–479.

27.

Foucault

(2023) Discipline and punish. In: Social theory re-wired.

28.

Fox

Shorey

Kang

, et al. (2023) Patchwork: The hidden, human labor of AI integration within essential work. Proceedings of the ACM on Human–Computer Interaction 7(CSCW1): 1–20.

29.

Fox

Sobel

Rosner

(2019) Managerial visions: Stories of upgrading and maintaining the public restroom with IoT. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp.1–15.

30.

Gerchick

Jegede

Shah

, et al. (2023) The devil is in the details: Interrogating values embedded in the allegheny family screening tool. In: Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pp.1292–1310.

31.

Gilman

(2018) Automated immunity. Available at: https://brownpoliticalreview.org/2018/02/automated-immunity/.

32.

Haraway

(2013) Situated knowledges: The science question in feminism and the privilege of partial perspective 1. In: Women, science, and technology. Routledge, pp.455–472.

33.

Hendry

(2004) The Care of the Elderly in Japan. London: Routledge.

34.

Hittle

Agbonifo

Suarez

, et al. (2016) Complexity of occupational exposures for home health-care workers: Nurses vs. home health aides. Journal of Nursing Management 24(8): 1071–1079.

35.

Hoffmann

(2021) Terms of inclusion: Data, discourse, violence. New Media & Society 23(12): 3539–3556.

36.

Holmes

(2023) The unintended consequences of artificial intelligence and education.

37.

Holstein

McLaren

Aleven

(2017) Intelligent tutors as teachers’ aides: Exploring teacher needs for real-time analytics in blended classrooms. In: Proceedings of the seventh international learning analytics & knowledge conference, pp.257–266.

38.

Holstein

McLaren

Aleven

(2018) Student learning benefits of a mixed-reality teacher awareness tool in AI-enhanced classrooms. In: International conference on artificial intelligence in education. pp.154–168. ISBN 9783319938431.

39.

Holstein

Mclaren

Aleven

(2019) Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics 6: 27–52.

40.

Holten Møller

Shklovski

Hildebrandt

(2020) Shifting concepts of value: Designing algorithmic decision-support systems for public services. In: Proceedings of the 11th Nordic conference on human–computer interaction: Shaping experiences, shaping society, pp.1–12.

41.

Huising

Elmholdt

Mäkinen

(2025) Introduction: The changing constitution and ecology of expertise. Research in the Sociology of Organizations.

42.

Irani

Silberman

(2013) Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp.611–620.

43.

Jørgensen

Nissen

(2022) Making sense of decision support systems: Rationales, translations and potentials for critical reflections on the reality of child protection. Big Data & Society 9(2): 20539517221125163.

44.

June

(2022) Oregon is dropping an artificial intelligence tool used in child welfare system. Available at: https://www.npr.org/2022/06/02/1102661376/oregon-drops-artificial-intelligence-child-abuse-cases.

45.

Kawakami

Coston

Heidari

, et al. (2024a) Studying up public sector AI: How networks of power relations shape agency decisions around AI design and use. Proceedings of the ACM on Human–Computer Interaction 8(CSCW2): 1–24.

46.

Kawakami

Coston

Zhu

, et al. (2024b) The Situate AI Guidebook: Co-designing a toolkit to support multi-stakeholder early-stage deliberations around public sector AI proposals, pp.1–22.

47.

Kawakami

Sivaraman

Cheng

, et al. (2022a) Improving human–AI partnerships in child welfare: Understanding worker practices, challenges, and desires for algorithmic decision support. In: Proceedings of the 2022 CHI conference on human factors in computing systems, pp.1–18.

48.

Kawakami

Sivaraman

Stapleton

, et al. (2022b) “Why do I care what’s similar?” Probing challenges in AI-assisted child welfare decision-making through worker–AI interface design concepts. In: Proceedings of the 2022 ACM designing interactive systems conference, pp.454–470.

49.

Kocon

(2017) Is teaching undervalued because it’s “women’s work”? Available at: https://tntp.org/blog/is-teaching-undervalued-because-its-womens-work/.

50.

Kuo

EFC

Cho

Olaye

, et al. (2022) Understanding the technological landscape of home health aides: Scoping literature review and a landscape analysis of existing mHealth apps. Journal of Medical Internet Research 24(11): e39997.

51.

Lebovitz

Levina

Lifshitz-Assaf

(2021) Is AI ground truth really true? The dangers of training and evaluating AI tools based on experts’ know-what. MIS Quarterly 45(3): 1501–1526.

52.

Lebovitz

Lifshitz-Assaf

Levina

(2022) To engage or not to engage with AI for critical judgments: How professionals deal with opacity when using AI for medical diagnosis. Organization Science 33(1): 126–148.

53.

Mateescu

Elish

(2019) AI in context: The labor of integrating new technologies.

54.

Migda

(2015) The legacies of slavery and JIM crow live on with exclusion of home health care workers from fair labor laws.

55.

Ming

Kuo

, et al. (2023) “I go beyond and beyond”: Examining the invisible work of home health aides. Proceedings of the ACM on Human–Computer Interaction 7(CSCW1): 1–21.

56.

Modén

Tallvid

Lundin

, et al. (2021) Intelligent tutoring systems: Why teachers abandoned a technology aimed at automating teaching processes.

57.

Muller

Kuhn

(1993) Participatory design. CACM 36(6): 24–28.

58.

Orlikowski

(1992) Learning from notes: Organizational issues in groupware implementation. In: Proceedings of the 1992 ACM conference on computer-supported cooperative work, pp.362–369.

59.

Osterman

(2018) Improving long-term care by finally respecting home-care aides. Hastings Center Report 48: S67–S70.

60.

Osterman

(2019) Improving job quality for direct care workers. Economic Development Quarterly 33(2): 151–156.

61.

Persson

Redmalm

Iversen

(2022) Caregivers’ use of robots and their effect on work environment: A scoping review. Journal of Technology in Human Services 40(3): 251–277.

62.

Pruss

(2023) Ghosting the machine: Judicial resistance to a recidivism risk assessment instrument. In: Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pp.312–323.

63.

Ratner

Elmholdt

(2023) Algorithmic constructions of risk: Anticipating uncertain futures in child protection services. Big Data & Society 10(2): 20539517231186120.

64.

Roll

Wylie

(2016) Evolution and revolution in artificial intelligence in education. International Journal of Artificial Intelligence in Education 26(2): 582–599.

65.

Rury

(1991) Education and Women’s Work: Female Schooling and the Division of Labor in Urban America, 1870–1930. SUNY Series on Women and Work. ERIC.

66.

Samant

Horowitz

, et al. (2021) Family surveillance by algorithm: The rapidly spreading tools few have heard of.

67.

Saxena

Badillo-Urquiola

Wisniewski

, et al. (2021) A framework of high-stakes algorithmic decision-making for the public sector developed through a case study of child-welfare. Proceedings of the ACM on Human–Computer Interaction 5(CSCW2): 1–41.

68.

Schwandt

Gates

(2018) Case study methodology. The SAGE Handbook of Qualitative Research 5: 600–630.

69.

Selwyn

(2016) Education and Technology: Key Issues and Debates. London, New York and Dublin: Bloomsbury Publishing.

70.

Sendak

Ratliff

Sarro

, et al. (2020) Real-world integration of a sepsis deep learning technology into routine clinical care: Implementation study. JMIR Medical Informatics 8(7): e15182.

71.

Sharkey

Wood

(2014) The Paro seal robot: Demeaning or enabling. In: Proceedings of AISB, volume 36, p.2014.

72.

Sloane

Moss

Awomolo

, et al. (2022) Participation is not a design fix for machine learning. In: Proceedings of the 2nd ACM conference on equity and access in algorithms, mechanisms, and optimization, pp.1–6.

73.

Smith

(2022) Teaching in the light of women’s history. Available at: https://www.facinghistory.org/ideas-week/teaching-light-womens-history.

74.

soc (2022) School of social work study confirms human services workers are underpaid. Available at: https://socialwork.uw.edu/news/school-social-work-study-confirms-human-services-workers-are-underpaid/.

75.

Solano-Kamaiko

Tan

Ming

, et al. (2025) “Who is running it?” Towards equitable AI deployment in home care work. In: Proceedings of the 2025 CHI conference on human factors in computing systems, pp.1–19.

76.

Spektor

Fox

Awumey

, et al. (2023) Charting the automation of hospitality: An interdisciplinary literature review examining the evolution of frontline service work in the face of algorithmic management. Proceedings of the ACM on Human–Computer Interaction 7(CSCW1): 1–20.

77.

Spiel

Brulé

Frauenberger

, et al. (2020) In the details: The micro-ethics of negotiations and in-situ judgements in participatory design with marginalised children. CoDesign 16(1): 45–65.

78.

Spinuzzi

(2002) A Scandinavian challenge, a US response: Methodological assumptions in Scandinavian and US prototyping approaches. In: Proceedings of the 20th annual international conference on Computer documentation, pp.208–215.

79.

Star

(1999) The ethnography of infrastructure. American Behavioral Scientist 43(3): 377–391.

80.

Star

Strauss

(1999) Layers of silence, arenas of voice: The ecology of visible and invisible work. Computer Supported Cooperative Work (CSCW) 8(1): 9–30.

81.

Steiner

Woo

(2021) Job-related stress threatens the teacher supply. Rand Corporation 25: 08-1.

82.

Suchman

(1993) Do categories have politics? The language/action perspective reconsidered. Computer Supported Cooperative Work (CSCW) 2(3): 177–190.

83.

Suchman

(1995) Making work visible. Communications of the ACM 38(9): 56–64.

84.

Suchman

(2011) Anthropological relocations and the limits of design. Annual Review of Anthropology 40(1): 1–18.

85.

Suchman

(1987) Plans and Situated Actions: The Problem of Human–Machine Communication. Cambridge, New York and Melbourne: Cambridge University Press.

86.

Taie

Lewis

(2022) Characteristics of 2020–21 public and private K-12 school teachers in the United States: Results from the national teacher and principal survey. National Center for Education Statistics.

87.

tea (2018) Teacher education: From revolution to evolution. Available at: https://teachereducation.steinhardt.nyu.edu/teacher-training-evolution/.

88.

Tseng

Okeke

Sterling

, et al. (2020) “We can learn. Why not?” Designing technologies to engender equity for home health aides. In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp.1–14.

89.

Vaithianathan

Putnam-Hornstein

Jiang

, et al. (2017) Developing predictive models to support child maltreatment hotline screening decisions: Allegheny county methodology and implementation. Center for Social data Analytics.

90.

Wang

Kapoor

Barocas

, et al. (2024) Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. ACM Journal on Responsible Computing 1(1): 1–45.

91.

Watters

(2015) The history of the future of education.

92.

Wright

(2018) Tactile care, mechanical hugs: Japanese caregivers and robotic lifting devices. Asian Anthropology 17(1): 24–39.

93.

Wright

(2019) Robots vs migrants? Reconfiguring the future of Japanese institutional eldercare. Critical Asian Studies 51(3): 331–354.

94.

Wyatt

Altman

(1995) Commentary: Prognostic models: Clinically useful or quickly forgotten? BMJ (Clinical Research Ed.) 311(7019): 1539–1541.

95.

Yang

Steinfeld

Zimmerman

(2019) Unremarkable AI: Fitting intelligent decision support into critical, clinical decision-making processes. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp.1–11.

96.

Yang

Zimmerman

Steinfeld

, et al. (2016) Investigating the heart pump implant decision process: Opportunities for decision support tools to help. In: Proceedings of the 2016 CHI conference on human factors in computing systems, pp.4477–4488.

97.

Yildirim

Sayar

, et al. (2023) Creating design resources to scaffold the ideation of AI concepts. In: Proceedings of the 2023 ACM designing interactive systems conference, pp.2326–2346.

98.

York

CHN

(2022) There is no such thing as women’s work: The gender of labor and the devaluation of social workers. Available at: https://ny.covenanthouse.org/there-is-no-such-thing-as-womens-work-the-gender-of-labor-and-the-devaluation-of-social-workers/.

99.

Zhang

Rana

Boltz

, et al. (2024) Data probes as boundary objects for technology policy design: Demystifying technology for policymakers and aligning stakeholder objectives in rideshare gig work. In: Proceedings of the 2024 CHI conference on human factors in computing systems, pp.1–21.