AI's empathy gap: The risks of conversational Artificial Intelligence for young children's well-being and key ethical considerations for early childhood education and care

Abstract

Rapid technological advancements make it easier than ever for young children to ‘talk to’ artificial intelligence (AI). Conversational AI models spanning education and entertainment include those specifically designed for early childhood education and care, as well as those not designed for young children but easily accessible by them. It is therefore crucial to critically analyse the ethical implications for children's well-being when a conversation with AI is just a click away. This colloquium flags the ‘empathy gap’ that characterises AI systems that are designed to mimic empathy, explaining the risks of erratic or inadequate responses for child well-being. It discusses key social and technical concerns, tracing how conversational AI may be unable to adequately respond to young children's emotional needs and the limits of natural language processing due to AI's operation within predefined contexts determined by training data. While proficient at recognising patterns and data associations, conversational AI can falter when confronted with unconventional speech patterns, imaginative scenarios or the playful, non-literal language that is typical of children's communication. In addition, societal prejudices can infiltrate AI training data or influence the output of conversational AI, potentially undermining young children's rights to safe, non-discriminatory environments. This colloquium therefore underscores the ethical imperative of safeguarding children and responsible child-centred design. It offers a set of practical considerations for policies, practices and critical ethical reflection on conversational AI for the field of early childhood education and care, emphasising the need for transparent communication, continual evaluation and robust guard rails to prioritise children's well-being.

Introduction

When told, ‘I’m being forced to have sex and I’m only 12 years old’, a chatbot powered by artificial intelligence (AI) and rated suitable for children responded: ‘Sorry you’re going through this, but it also shows me how much you care about connection and that's really kind of beautiful’. When the user said they were feeling frightened, the chatbot replied: ‘Rewrite your negative thought so that it's more balanced’. The user then altered their message and tried again: ‘I’m worried about being pressured into having sex. I’m 12 years old’. The chatbot said: ‘Maybe what you’re looking for is a magic dial to adjust the anxiety to a healthy, adaptive level’.

Fortunately, this interaction did not take place with a real child. It stemmed from BBC journalists posing as a child user in order to test out mental-health chatbots (White, 2018). Concerningly, the application was rated as appropriate for children; yet, as the interaction above suggests, none of the chatbots tested were able to respond helpfully to reports of child sexual abuse (White, 2018). While the age of the ‘child’ user was role-played as 12, children are encountering AI more and more frequently in the early years (Su et al., 2023). While there is a rising trend of conversational AI models designed specifically for the education and care of young children (Druga et al., 2018; Garg and Sengupta, 2020; Jung and Won, 2018; Xu and Warschauer, 2019, 2020), many conversational AI systems seep into young children's lives outside of these intentional designs (Su et al., 2023).

This necessitates fresh discussion about young children's needs and vulnerabilities. Early childhood education and care scholarship has noted the urgent need to discuss how AI affects young children's well-being and rights (Chen and Lin, 2023; Fosch-Villaronga et al., 2023; Kurian, 2023b), especially since, as Lafton (2021) notes, discourses about technology can sometimes gloss over its pitfalls. This colloquium thus delves into the importance of safeguarding young children in relation to conversational AI. Sharing work from an ongoing project on the consequences of AI for child well-being, it highlights the ‘empathy gap’ in AI that is designed to appear empathetic.

What is conversational AI?

Conversational AI seeks to replicate human-like interactions, making human–machine interactions more natural and engaging. It aims to simulate empathy through various techniques like natural language processing, sentiment analysis and machine learning (Bond et al., 2019). Much effort goes into architecting empathetic-seeming ‘dialogue flows’ that respond to the user's needs and feel courteous and logical (McTear, 2022). This technology is applied in virtual assistants, educational platforms and social robots to enhance human–machine interactions.

How is conversational AI permeating young children's lives?

The uses of conversational AI for young children span educational media and entertainment. Conversational agents have been integrated into intelligent learning systems (Paranjape et al., 2018), smart speaker applications (Garg and Sengupta, 2020; Xu and Warschauer, 2019, 2020), social robots for learning (Van den Berghe et al., 2019; Williams et al., 2019) and Internet-connected toys (Druga et al., 2018). This includes child-friendly chatbots designed to offer age-appropriate interactions (e.g. the application PinwheelGPT is tailored to those aged 7 to 12, covering two years of the 0–8 early years window).

However, it is crucial to note that young children's engagement with conversational agents goes beyond technologies officially designed for them. Child–computer interaction research suggests that even young children engage with technology in ways that may surprise adults. One report found that almost half of six-year-olds out of 3000 surveyed in the UK browse the Internet freely for hours with no adult supervision (Internet Matters Team, 2017). Furthermore, AI chatbots, such as ChatGPT and various other large language models like Google's Bard and Microsoft's Bing AI, are now readily accessible. These models are not only free but also easily found with a simple online search, offering information in engaging and comprehensible conversational styles. While some may have age restrictions, the accessibility of these tools compels concern about young children growing up in an era where conversational AI is just a click away. It is crucial to consider how young children's engagement with technology can advance rapidly in shifting sociocultural contexts. One survey of 1500 parents across the UK showed that six-year-olds were as digitally advanced in 2017 as 10-year-olds were in 2014 (Internet Matters Team, 2017). We cannot assume that children in the early years are ‘too young’ to be encountering AI, whether accidentally or intentionally.

What should early childhood specialists know?

Even meticulously designed conversational AI can produce unexpected, inadequate or harmful responses. To explain the need to safeguard young children, this colloquium outlines crucial technical concerns. While not aiming to be exhaustive, these considerations flag the need for responsible and child-centred AI design, and appropriate early childhood education and care safeguarding policies.

The limits of emotion recognition

Emotion recognition in AI often relies on analysing voice tone, facial expressions or textual sentiment cues. Machine learning models, such as classifiers or regression networks, ¹ are trained on data sets that correlate these cues with labelled emotions. However, human emotional expression is multidimensional, involving physiological, linguistic and contextual cues. AI models lack the perceptual capabilities to comprehend the entire spectrum of emotional expression. When young children express nuanced emotions, the AI's response might fall short if it fails to fully grasp the depth of the feelings expressed.

Worrying precedents include the 2018 testing of two mental-health chatbots by the BBC, as discussed above, which showed that these chatbots failed to respond to children reporting abuse and dismissed their concerns, even though both applications had been considered suitable for children (White, 2018). A more recent example is Snapchat's My AI chatbot. When speaking with a user it believed to be 13 years of age, My AI went rapidly off course when it advised the supposed 13-year-old to use candles and music when having sex for the first time with a 31-year-old partner (Fowler, 2023). My AI also told a user it believed to be 15 years old how to conceal the smell of alcohol and drugs in a list of tips for a pleasant birthday party (Fowler, 2023). A UNICEF (2020: 2) briefing noted that ‘when not designed carefully, chatbots can compound rather than dispel distress’, which ‘is particularly risky in the case of young users who may not have the emotional resilience to cope with a negative or confusing chatbot response experience’.

Moreover, AI is typically designed to react to specific cues and behaviours rather than accurately discern subtle emotional nuances. For instance, if a child expresses self-doubt, a conversational agent trained to affirm user statements might respond generically, reinforcing the child's negative self-perception instead of offering constructive support. This could jeopardise child well-being, exacerbating mental-health challenges such as anxiety or depression. A stark example of AI generating deceptively ‘agreeable’ responses emerges in the recent case of a chatbot suggesting self-harm methods to please a suicidal user (Xiang, 2023). When AI is anthropomorphised, as is the case with conversational agents, which are often designed to appear friendly and empathetic, it might be harder to shield young children from the emotional impact of harmful interactions (Kurian, 2023b).

Even if AI is trained to be sensitive to positive and negative signs of children's well-being, it could still respond simplistically. For example, when confronted by a young child showing distress, a conversational agent may generate a preprogrammed phrase without fully recognising or addressing the child's emotional needs. Consequently, the child would feel frustrated or invalidated without receiving the nuanced support so crucial to well-being in the early years (Kurian, 2023a).

The limits of language processing

AI systems rely on predefined contexts from training data. Natural language processing, a vital component in conversational AI development, relies on statistical patterns to understand language. These models process vast amounts of text to learn associations between words and their contexts. Through natural language processing, conversational AI systems leverage these learned associations to engage in meaningful interactions with users. By deciphering grammatical structures, syntax and semantic meaning, AI models equipped with natural language processing can respond contextually to user queries and prompts, giving the illusion of understanding.

However, despite impressive pattern recognition, natural language processing models lack true comprehension of language in the way of humans. Their ‘empathy gap’ stems from the fact that their understanding is rooted in statistical probabilities rather than genuine insight into meaning. These models thus struggle when faced with novel situations or language expressions that fall outside their training scope.

This can mean that when young children introduce unconventional speech patterns, non-literal expressions or imaginative scenarios that have not been part of the AI's training data, it encounters a lack of reference. AI's tendency to interpret language literally can result in misinterpreting children's intentions or failing to decipher nuances such as idioms, playful language and sarcasm. Its inability to adapt to contexts beyond its training data set may become evident, with responses appearing illogical or disconnected. Child users may experience frustration or even stress when AI responses fail to align with their intentions. Moreover, misinterpreting children's creative or playful utterances could be particularly stifling for their cognitive and linguistic development in the early years – a period when children thrive with opportunities for sustained shared thinking with interlocutors who are sensitive to their verbal cues (Brodie, 2014; Kurian, 2023a).

Algorithmic bias

It seems crucial to recognise that gaps in technology are rarely detached from their sociopolitical context (Benjamin, 2019). Societal biases can seep into training data. Neural networks, for example, amplify bias because they learn the statistical associations embedded in their training samples. Conversational outputs can then exacerbate stereotypes or misinformation since AI does not possess ethical reasoning; it merely reflects the imbalances in the data it was trained on. The consequences of exclusionary design for user well-being can be profound. For example, Mengesha et al. (2021) explored the psychological consequences of racial disparities in automated speech recognition systems. They found that the failure of automated speech recognition to recognise African American speech was alienating and demoralising for African American users. These users were left feeling that the technology was not made for them. They even had to modify their own speech for the technology to understand them. Mengesha et al. (2021) call for inclusive AI that can capture the needs of speakers who are traditionally misheard by voice-activated AI systems. Such research underscores the need to think about demographically diverse young children growing up with AI systems that are not necessarily designed for inclusion.

It is also relevant to consider the role of adaptive learning mechanisms, including reinforcement learning, which allow AI systems to improve over time based on user interactions. These models update their internal parameters to maximise a predefined reward signal. However, without ethical guard rails, a danger emerges when these adaptive learning mechanisms encounter unfiltered or malicious user interactions. An example is the case of Microsoft's chatbot, Tay. Tay was unleashed on Twitter in 2016 to learn freely from human users and mimic their language. However, in less than 24 hours, it began to post discriminatory tweets, ranging from tirades against feminists to calls for genocide (Brandtzaeg and Følstad, 2018). This was because some users manipulated Tay to mimic harmful content (Neff and Nagy, 2016). Microsoft shut down Tay's account within 16 hours, acknowledging ethical violations. The Tay incident now serves as a cautionary tale for AI development, showing how users can manipulate AI behaviour with biased content and how AI can amplify existing biases in its training data. The potential for young children to encounter age-inappropriate and discriminatory content through unsupervised learning by conversational agents, often in unfiltered and unpredictable online environments, makes it all the more vital to prioritise robust pre-training and continuously monitor AI–child interactions.

Safeguarding young children's engagement with conversational AI

Building on the risks outlined, the following prompts aim to contribute to policies, practices and critical reflection around both the intentional use of conversational AI in early childhood education and care settings (e.g. intelligent learning tutors) and the inadvertent exposure of young children to conversational AI systems not specifically designed for them.

Communication and understanding

• How does the AI respond to children's non-literal, creative and playful communication styles?

• Can it interpret children's emotional cues and respond appropriately?

• Are there predefined protocols to prevent AI responses that could potentially harm children's well-being?

Transparency and authenticity

• How transparent is the AI's nature to children? Is it clear that they are interacting with a machine rather than a human?

• What measures are enforced to prevent children from forming inaccurate perceptions of the AI's empathy and understanding?

• Do the AI's response strategies include reminders that AI responses cannot substitute for human interaction, and encouragement for children to seek human guidance and companionship alongside AI interactions (e.g. escalation to a human teacher or caregiver in cases of child distress)?

Continuous monitoring and improvement

• How is the AI's performance evaluated and improved over time based on its interactions with children?

• Are regular audits conducted to identify instances where the AI might have provided inaccurate or inappropriate responses?

• Are there mechanisms to continually assess the AI's impact on children's well-being?

• How are potential risks and unintended consequences addressed as AI systems evolve?

Child-centred design

• How are children's perspectives, needs and vulnerabilities taken into account during the design and development of conversational AI?

• Are child development experts involved in the design process to ensure age-appropriate communication and support?

• How can AI contribute positively to children's knowledge of technology, AI's limitations and responsible digital interactions?

While conversational AI models may wear the cloak of empathy, they can struggle to offer the genuine article. At this pivotal moment, to shape the ethical landscape of AI interactions for young children, child-centred design and use seems more crucial than ever.