Robot’s adaptive emotional feedback sustains children’s social engagement and promotes their vocabulary learning: a long-term child–robot interaction study
Restricted accessResearch articleFirst published online August, 2019
Robot’s adaptive emotional feedback sustains children’s social engagement and promotes their vocabulary learning: a long-term child–robot interaction study
In this article, we present an emotion and memory model for a social robot. The model allowed the robot to create a memory account of a child’s emotional events over four individual sessions. The robot then adapted its behaviour based on the developed memory. The model was applied on the NAO robot to teach vocabulary to children while playing the popular game ‘Snakes and Ladders’. We conducted an exploratory evaluation of our model with 24 children at a primary school for 2 weeks to verify its impact on children’s long-term social engagement and overall vocabulary learning. Our preliminary results showed that the behaviour generated based on our model was able to sustain social engagement. In addition, it also helped children to improve their vocabulary. We also evaluated the impact of the positive, negative and neutral emotional feedback of the NAO robot on children’s vocabulary learning. Three groups of children (eight per group) interacted with the robot on four separate occasions over a period of 2 weeks. Our results showed that the condition where the robot displayed positive emotional feedback had a significantly positive effect on the child’s vocabulary learning performance as compared to the two other conditions: negative feedback and neutral feedback.
AhmadM.MubinO.EscuderoP. (2015). Using adaptive mobile agents in games based scenarios to facilitate foreign language word learning. In Proceedings of the 3rd International Conference on Human-Agent Interaction (pp. 255–257). New York, NY: ACM.
2.
AhmadM.MubinO.OrlandoJ. (2016a). Children views’ on social robot’s adaptations in education. In Proceedings of the 28th Australian Conference on Computer-human Interaction (pp. 145–149). New York, NY: ACM.
3.
AhmadM.MubinO.OrlandoJ. (2016b). Effect of different adaptations by a robot on children’s long-term engagement: An exploratory study. In Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology (p. 31). New York, NY: ACM.
4.
AhmadM.MubinO.OrlandoJ. (2016c). Understanding behaviours and roles for social and adaptive robots in education: teacher’s perspective. In Proceedings of the Fourth International Conference on Human Agent Interaction. (pp. 297–304). New York, NY: ACM.
5.
AhmadM.MubinO.OrlandoJ. (2017a). Adaptive social robot for sustaining social engagement during long-term children–robot interaction. International Journal of Human–computer Interaction, 34, 943–962.
6.
AhmadM.MubinO.OrlandoJ. (2017b). A systematic review of adaptivity in human-robot interaction. Multimodal Technologies and Interaction, 1, 14.
7.
ArgyleM.DeanJ. (1965). Eye-contact, distance and affiliation. Sociometry, 28, 289–304.
8.
BartneckC.KandaT.MubinO.Al MahmudA. (2007). The perception of animacy and intelligence based on a robot’s embodiment. In Proceedings of the 7th IEEE-RAS International Conference on Humanoid Robots (pp. 300–305). New York, NY: IEEE.
9.
BaxterP.AshurstE.ReadR.KennedyJ.BelpaemeT. (2017). Robot education peers in a situated primary school study: Personalisation promotes child learning. PLoS ONE, 12(5), e0178126.
10.
BeckerW. (1977). Teaching reading and language to the disadvantaged–What we have learned from field research. Harvard Educational Review, 47, 518–543.
11.
BelpaemeT.BaxterP. E.ReadR.WoodR.CuayáhuitlH.KieferB.. . . HumbertR. (2012). Multimodal child-robot interaction: Building social bonds. Journal of Human-robot Interaction, 1, 33–53.
12.
BenensonJ. F.PascoeJ.RadmoreN. (2007). Children’s altruistic behavior in the dictator game. Evolution and Human Behavior, 28, 168–175.
13.
BowerG. H. (1992). How might emotions affect learning. In ChristiansonS. A. (Ed.), The handbook of emotion and memory: Research and theory (Vol. 3, p. 31). London, England: Psychology Press.
14.
ButlerD. L.WinneP. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65, 245–281.
15.
CahillL.HaierR. J.FallonJ.AlkireM. T.TangC.KeatorD.McgaughJ. L. (1996). Amygdala activity at encoding correlated with long-term, free recall of emotional information. Proceedings of the National Academy of Sciences of the United States of America, 93, 8016–8021.
16.
CalvoR. A.D’MelloS. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1, 18–37.
17.
CanliT.ZhaoZ.BrewerJ.GabrieliJ. D.CahillL. (2000). Event-related activation in the human amygdala associates with later memory for individual emotional experience. Journal of Neuroscience, 20, RC99.
18.
CariniR. M.KuhG. D.KleinS. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47, 1–32.
19.
CastellanoG.PereiraA.LeiteI.PaivaA.McOwanP. W. (2009). Detecting user engagement with a robot companion using task and social interaction-based features. In Proceedings of the international conference on multimodal interfaces (pp. 119–126). New York, NY: ACM.
20.
ChristiansonS. A. (2014). The handbook of emotion and memory: Research and theory. London, England: Psychology Press.
21.
ComblainC.D’ArgembeauA.Van der LindenM. (2005). Phenomenal characteristics of autobiographical memories for emotional and neutral events in older and younger adults. Experimental Aging Research, 31, 173–189.
22.
ConinxA.BaxterP.OleariE.BelliniS.BiermanB.HenkemansO. B.. . . BelpaemeT. (2016). Towards long-term social child-robot interaction: Using multi-activity switching to engage young users. Journal of Human-robot Interaction, 5, 32–67.
23.
D’ArgembeauA.ComblainC.Van der LindenM. (2003). Phenomenal characteristics of autobiographical memories for positive, negative, and neutral events. Applied Cognitive Psychology, 17, 281–294.
24.
DautenhahnK. (1998). The art of designing socially intelligent agents: Science, fiction, and the human in the loop. Applied Artificial Intelligence, 12, 573–617.
25.
de HaasM.BaxterP.de JongC.KrahmerE.VogtP. (2017). Exploring different types of feedback in preschooler and robot interaction. In Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. (pp. 127–128). New York, NY: ACM.
26.
de HaasM.VogtP.KrahmerE. (2016). Enhancing child-robot tutoring interactions with appropriate feedback. In Proceedings of First Workshop on Long-term Child-robot Interaction. New York, NY: IEEE.
27.
Feldman BarrettL.RussellJ. A. (1998). Independence and bipolarity in the structure of current affect. Journal of Personality and Social Psychology, 74, 967–984.
28.
FriesenE.EkmanP. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.
29.
GaoA. Y.BarendregtW.CastellanoG. (2017). Personalised human-robot co-adaptation in instructional settings using reinforcement learning. In Persuasive Embodied Agents for Behavior Change (PEACH2017) Workshop at the International conference on Intelligent Virtual Agents (IVA2017). Uppsala, Sweden: Uppsala University.
30.
GoldfieldB. A.ReznickJ. S. (1990). Early lexical acquisition: Rate, content, and the vocabulary spurt. Journal of Child Language, 17, 171–183.
31.
Gonzalez-PachecoV.RameyA.Alonso-MartinF.Castro-GonzalezA.SalichsM. A. (2011). Maggie: A social robot as a gaming platform. International Journal of Social Robotics, 3, 371–381.
32.
GordonG.SpauldingS.WestlundJ. K.LeeJ. J.PlummerL.MartinezM.BreazealC. (2016). Affective personalization of a social robot tutor for children’s second language skills. In AAAI’16 Proceedings of the 30th AAAI conference on artificial intelligence (pp. 3951–3957). New York, NY: ACM.
33.
HastieH.LimM. Y.JanarthanamS.DeshmukhA.AylettR.FosterM. E.HallL. (2016). I remember you!: Interaction with memory for an empathic virtual robotic tutor. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems (pp. 931–939). Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.
34.
HattieJ.TimperleyH. (2007). The power of feedback. Review of Educational Research, 77, 81–112.
HoW. C.DautenhahnK.LimM. Y.VargasP. A.AylettR.EnzS. (2009). An initial memory model for virtual and robot companions supporting migration and long-term interaction. In RO–MAN 2009 – The 18th IEEE International Symposium on Robot and Human Interactive Communication (pp. 277–284). New York, NY: IEEE.
JanssenJ. B.van der WalC. C.NeerincxM. A.LooijeR. (2011). Motivating children to learn arithmetic with an adaptive robot game. In International Conference on Social Robotics (pp. 153–162). New York, NY: Springer.
39.
JeonM. (2017). Emotions and affect in human factors and human–computer interaction: Taxonomy, theories, approaches, and methods. Emotions and Affect in Human Factors and Human-Computer Interaction, 2017, 3–26.
40.
KandaT.HiranoT.EatonD.IshiguroH. (2004). Interactive robots as social partners and peer tutors for children: A field trial. Human-computer Interaction, 19, 61–84.
41.
KasapZ.Magnenat-ThalmannN. (2010). Towards episodic memory-based long-term affective interaction with a human-like robot. In 19th International Symposium in Robot and Human Interactive Communication (pp. 452–457). New York, NY: IEEE.
42.
KennedyJ.BaxterP.SenftE.BelpaemeT. (2016). Social robot tutoring for child second language learning. In Human-robot Interaction (HRI), 11th ACM/IEEE International Conference on IEEE (pp. 231–238). New York, NY: IEEE.
43.
KomatsubaraT.ShiomiM.KandaT.IshiguroH.HagitaN. (2014). Can a social robot help children’s understanding of science in classrooms? In Proceedings of the Second International Conference on Human-agent Interaction (pp. 83–90). New York, NY: IEEE.
44.
KoryJ.BreazealC. (2014). Storytelling with robots: Learning companions for preschool children’s language development. In Robot and Human Interactive Communication, 2014 RO-MAN: The 23rd IEEE International Symposium on IEEE (pp. 643–648). New York, NY: IEEE.
LeDouxJ. E. (1993). Emotional memory systems in the brain. Behavioural Brain Research, 58, 69–79.
48.
LeeM. K.ForlizziJ.KieslerS.RybskiP.AntanitisJ.SavetsilaS. (2012). Personalization in HRI: A longitudinal field experiment. In Human-robot Interaction (HRI), 2012 7th ACM/IEEE International Conference on IEEE (pp. 319–326). New York, NY: IEEE.
49.
LeeS.NohH.LeeJ.LeeK.LeeG. G.SagongS.KimM. (2011). On the effectiveness of robot-assisted language learning. Recall, 23, 25–58.
50.
LeiteI. (2013). Long-term interactions with empathic social robots (PhD Thesis). Instituto Superior Técnico, Universidade de Lisboa, Lisboa.
51.
LeiteI.CastellanoG.PereiraA.MartinhoC.PaivaA. (2014). Empathic robots for long-term interaction. International Journal of Social Robotics, 6, 329–341.
52.
LeiteI.MartinhoC.PaivaA. (2013). Social robots for long-term interaction: A survey. International Journal of Social Robotics, 5, 291–308.
53.
LeiteI.PereiraA.LehmanJ. F. (2017). Persistent memory in repeated child-robot conversations. In Proceedings of the 2017 Conference on Interaction Design and Children (pp. 238–247). New York, NY: IEEE.
54.
LeverJ. (1976). Sex differences in the games children play. Social Problems, 23, 478–487.
55.
LevineL. J.BurgessS. L. (1997). Beyond general arousal: Effects of specific emotions on memory. Social Cognition, 15, 157–181.
56.
LevineL. J.PizarroD. A. (2004). Emotion and memory research: A grumpy overview. Social Cognition, 22, 530–554.
57.
LeyzbergD.SpauldingS.ScassellatiB. (2014). Personalizing robot tutors to individuals’ learning differences. In Proceedings of the 2014 ACM/IEEE International Conference on Human-robot Interaction (pp. 423–430). New York, NY: IEEE.
58.
McClintockC. G.NuttinJ. M.Jr. (1969). Development of competitive game behavior in children across two cultures. Journal of Experimental Social Psychology, 5, 203–218.
59.
McCulloughJ. A.CunninghamL. A.WilsonR. H. (1992). Auditory/visual word-identification test materials: Computer application with children. Journal of the American Academy of Audiology, 3, 208–214.
60.
MillerG. A.GildeaP. M. (1987). How children learn words. Scientific American, 257, 94–99.
61.
MoskovskyC.AlrabaiF. (2009). Intrinsic motivation in Saudi learners of English as a foreign language. The Open Applied Linguistics Journal, 2, 1–10.
62.
MubinO. (2011). ROILA: Robot interaction language (PhD Thesis). Technische Universiteit Eindhoven, Eindhoven, The Netherlands.
63.
MubinO.KhanA.ObaidM. (2016). #naorobot: Exploring NAO discourse on Twitter. In Proceedings of the 28Th Australian Conference on Computer-human Interaction (pp. 155–159). New York, NY: ACM.
64.
MubinO.StevensC. J.ShahidS.Al MahmudA.DongJ. J. (2013). A review of the applicability of robots in education. Journal of Technology in Education and Learning. Retrieved from http://roila.org/wp-content/uploads/2013/07/209-0015.pdf
PekrunR.Linnenbrink-GarciaL. (2012). Academic emotions and student engagement. In SandraC. L.ReschlyA. L. (Eds.), Handbook of research on student engagement (pp. 259–282). New York, NY: Springer.
67.
PrenskyM.BerryB. D. (2001). Do they really think differently. On the Horizon, 9, 1–9.
68.
RieberL. P. (1996). Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research and Development, 44, 43–58.
69.
RinaldiD.IspitaA. (2012). Teaching vocabulary by using capsule vocabulary strategy at fourth grade of elementary school. E-jurnal Mahasiswa Prodi Pend Bahasa Inggris, 1, 2012.
70.
Rosenthal-von der PüttenA. M.StraßmannC.KrämerN. C. (2016). Robots or agents–neither helps you more or less during second language acquisition. In International Conference on Intelligent Virtual Agents (pp. 256–268). New York, NY: Springer.
71.
SharanS.TanI. G. C. (2008). Student engagement in learning. In TanI. G. C.SharanS. (Eds.), Organizing schools for productive learning (pp. 41–45). New York, NY: Springer.
72.
SidnerC. L.LeeC.KiddC. D.LeshN.RichC. (2005). Explorations in engagement for humans and robots. Artificial Intelligence, 166, 140–164.
73.
SungJ.ChristensenH. I.GrinterR. E. (2009). Robots in the wild: Understanding long-term use. In Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction (pp. 45–52). New York, NY: ACM.
74.
SzafirD.MutluB. (2012). Pay attention! Designing adaptive agents that monitor and improve user engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 11–20). New York, NY: ACM.
TraftonG.HiattL.HarrisonA.TamborelloF.KhemlaniS.SchultzA. (2013). Act-r/e: An embodied cognitive architecture for human-robot interaction. Journal of Human-robot Interaction, 2(1), 30–55.
77.
TulvingE. (1972). Episodic and semantic memory. In TulvingE.DonaldsonW. (Eds.), Organization of memory (Vol. 1, pp. 381–403). New York, NY: Academic Press.
78.
UluerP.AkalinN.KöseH. (2015). A new robotic platform for sign language tutoring. International Journal of Social Robotics, 7, 571–585.
79.
Van LangeP. A.De BruinE.OttenW.JoiremanJ. A. (1997). Development of prosocial, individualistic, and competitive orientations: theory and preliminary evidence. Journal of Personality and Social Psychology, 73, 733–746.
80.
VygotskyL. S. (1980). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard university press.
81.
WentzelK. R. (2002). Are effective teachers like good parents? Teaching styles and student adjustment in early adolescence. Child Development, 73, 287–301.
82.
WestlundJ. M. K.DickensL.JeongS.HarrisP. L.De StenoD.BreazealC. L. (2017). Children use non-verbal cues to learn new words from robots as well as people. International Journal of Child-computer Interaction, 13, 1–9.
83.
WilliamsA. B. (2003). The qualitative impact of using Lego Mindstorms robots to teach computer engineering. IEEE Transactions on Education, 46, 206.
