This study aimed to assess the extent to which human participants co-represent the lexico-semantic processing of a humanoid robot partner. Specifically, we investigated whether participants would engage their speech production system to predict the robot’s upcoming words, and how they would progressively adapt to the robot’s verbal behaviour. In the experiment, a human participant and a robot alternated in naming pictures of objects from 15 semantic categories, while the participant’s electrophysiological activity was recorded. We manipulated word frequency as a measure of lexical access, with half of the pictures associated with high-frequency names and the other half with low-frequency names. In addition, the robot was programmed to provide semantic category labels (e.g., “tool” for the picture of a hammer) instead of the more typical basic-level names (e.g., “hammer”) for items in five categories. Analysis of the stimulus-locked activity revealed a comparable event-related potential (ERP) associated with word frequency both when it was the participant’s and the robot’s turn to speak. Analysis of the response-locked activity showed a different pattern for the category and basic-level responses in the first but not in the second part of the experiment, suggesting that participants adapted to the robot’s lexico-semantic patterns over time. These findings provide empirical evidence for two key points: (1) participants engage their speech production system to predict the robot’s upcoming words and (2) partner-adaptive behaviour facilitates comprehension of the robot’s speech.
AlmeidaJ.KnobelM.FinkbeinerM.CaramazzaA. (2007). The locus of the frequency effect in picture naming: When recognizing is not enough. Psychonomic Bulletin & Review, 14(6), 1177–1182. https://doi.org/10.3758/BF03193109
2.
Al MoubayedS.BeskowJ.SkantzeG.GranströmB. (2012). Furhat: A back-projected human-like robot head for multiparty human-machine interaction. In EspositoA.EspositoA. M.VinciarelliA.HoffmannR.MüllerV. C. (Eds.), Cognitive behavioural systems. Lecture notes in computer science (Vol. 7403, pp. 114–130). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-34584-5_9
AtmacaS.SebanzN.KnoblichG. (2011). The joint flanker effect: Sharing tasks with real and imagined co-actors. Experimental Brain Research. Experimentelle Hirnforschung. Experimentation Cerebrale, 211(3–4), 371–385. https://doi.org/10.1007/s00221-011-2709-9
5.
BainbridgeW. A.HartJ. W.KimE. S.ScassellatiB. (2011). The benefits of interactions with physically present robots over video-displayed agents. International Journal of Social Robotics, 3(1), 41–52. https://doi.org/10.1007/s12369-010-0082-7
6.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. https://doi.org/10.18637/jss.v067.i01
7.
BausC.SebanzN.FuenteV.de la BranziF. M.MartinC.CostaA. (2014). On predicting others’ words: Electrophysiological evidence of prediction in speech production. Cognition, 133(2), 395–407. https://doi.org/10.1016/j.cognition.2014.07.006
8.
BelhasseinK.Fernández-CastroV.MayimaA.ClodicA.PacherieE.GuidettiM.AlamiR.CochetH. (2022). Addressing joint action challenges in HRI: Insights from psychology and philosophy. Acta Psychologica, 222, 103476. https://doi.org/10.1016/j.actpsy.2021.103476
9.
BellL.GustafsonJ. (2003). Prosodic adaptation in human—Computer interaction.
10.
BlombergM.SkantzeG.MoubayedS. A.GustafsonJ.BeskowJ.GranströmB. (2012). Children and adults in dialogue with the robot head Furhat—Corpus collection and initial analysis.
BraniganH. P.PickeringM. J.PearsonJ.McLeanJ. F. (2010). Linguistic alignment between people and computers. Journal of Pragmatics, 42(9), 2355–2368. https://doi.org/10.1016/j.pragma.2009.12.012
15.
BraniganH. P.PickeringM. J.PearsonJ.McLeanJ. F.BrownA. (2011). The role of beliefs in lexical alignment: Evidence from dialogs with humans and computers. Cognition, 121(1), 41–57. https://doi.org/10.1016/j.cognition.2011.05.011
16.
BrehmL.TaschenbergerL.MeyerA. (2019). Mental representations of partner task cause interference in picture naming. Acta Psychologica, 199, 102888. https://doi.org/10.1016/j.actpsy.2019.102888
17.
BrennanS. E. (1991). Conversation with and through computers. User Modeling and User-Adapted Interaction, 1(1), 67–86. https://doi.org/10.1007/BF00158952
18.
BrennanS. E.ClarkH. (1996). Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology. Learning, Memory, and Cognition, 22(6), 1482–1493. https://doi.org/10.1037/0278-7393.22.6.1482
19.
BrennanS. E.GalatiA.KuhlenA. K. (2010). Chapter 8—Two minds, one dialog: Coordinating speaking and understanding. In RossB. H. (Ed.), Psychology of learning and motivation (Vol. 53, pp. 301–344). Academic Press. https://doi.org/10.1016/S0079-7421(10)53008-1
20.
CherakaraN.VargheseF.ShabanaS.NelsonN.KarukayilA.KulothunganR.FarhanM. A.NessetB.MoujahidM.DinkarT.RieserV.LemonO. (2023). FurChat: An embodied conversational agent using LLMs, combining open and closed-domain dialogue with facial expressions (arXiv:2308.15214). arXiv. https://doi.org/10.48550/arXiv.2308.15214
21.
CirilloG.RunnqvistE.StrijkersK.NguyenN.BausC. (2022). Conceptual alignment in a joint picture-naming task performed with a social robot. Cognition, 227, 105213. https://doi.org/10.1016/j.cognition.2022.105213
22.
CirilloG.StrijkersK.RunnqvistE.BausC. (2024). Effects of shared attention on joint language production across processing stages. Language, Cognition and Neuroscience, 39(2), 203–214. https://doi.org/10.1080/23273798.2023.2260021
23.
CooperS.GowS.FensomeS. F. P.DragoneM.KourtisD. (2020). An EEG investigation on planning human-robot handover tasks. 2020 IEEE International Conference on Human-Machine Systems (ICHMS), Rome, Italy, pp. 1–6. https://doi.org/10.1109/ICHMS49158.2020.9209543
24.
CorpsR.GambiC.PickeringM. (2017). Coordinating utterances during turn-taking: The role of prediction, response preparation, and articulation. Discourse Processes, 55, 1–11. https://doi.org/10.1080/0163853X.2017.1330031
25.
CrossE. S.HortensiusR.WykowskaA. (2019). From social brains to social robots: Applying neurocognitive insights to human–robot interaction. Philosophical Transactions of the Royal Society B: Biological Sciences, 374(1771), 20180024. https://doi.org/10.1098/rstb.2018.0024
26.
DellG. S.ChangF. (2014). The P-chain: Relating sentence production and its disorders to comprehension and acquisition. Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1634), 20120394. https://doi.org/10.1098/rstb.2012.0394
27.
DikkerS.PylkkanenL. (2011). Before the N400: Effects of lexical–semantic violations in visual cortex. Brain and Language, 118(1), 23–28. https://doi.org/10.1016/j.bandl.2011.02.006
28.
DuñabeitiaJ. A.CrepaldiD.MeyerA. S.NewB.PliatsikasC.SmolkaE.BrysbaertM. (2018). MultiPic: A standardized set of 750 drawings with norms for six European languages. Quarterly Journal of Experimental Psychology, 71(4), 808–816. https://doi.org/10.1080/17470218.2017.1310261
29.
FedermeierK. D.KutasM. (1999). A rose by any other name: Long-term memory structure and sentence processing. Journal of Memory and Language, 41(4), 469–495. https://doi.org/10.1006/jmla.1999.2660
30.
FischerK. (2021). Tracking anthropomorphizing behavior in human-robot interaction. Journal of Human-Robot Interaction, 11(1), 41–4:28. https://doi.org/10.1145/3442677
FreunbergerD.RoehmD. (2016). Semantic prediction in language comprehension: Evidence from brain potentials. Language, Cognition and Neuroscience, 31(9), 1193–1205. https://doi.org/10.1080/23273798.2016.1205202
35.
FusaroliR.TylénK.GarlyK.SteensigJ.ChristiansenM. H.DingemanseM. (2017). Measures and mechanisms of common ground: Backchannels, conversational repair, and interactive alignment in free and task-oriented social interactions. 2055–2060. https://pure.mpg.de/pubman/faces/ViewItemOverviewPage.jsp?itemId=item_2449737
36.
GambiC.PickeringM. (2023). Joint language production and the representation of other speakers’ utterances (pp. 304–321). https://doi.org/10.4324/9781003145790-13
37.
GambiC.Van deCavey J.PickeringM. J. (2015). Interference in joint picture naming. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(1), 1–21. https://doi.org/10.1037/a0037438
38.
GambinoA.FoxJ.RatanR. (2020). Building a stronger CASA: Extending the computers are social actors paradigm. Human-Machine Communication, 1, 71–86. https://doi.org/10.30658/hmc.1.5
39.
GarrodS.AndersonA. (1987). Saying what you mean in dialogue: A study in conceptual and semantic co-ordination. Cognition, 27(2), 181–218. https://doi.org/10.1016/0010-0277(87)90018-7
40.
GarrodS.PickeringM. J. (2015). The use of content and timing to predict turn transitions. Frontiers in Psychology, 6, Article 751. https://doi.org/10.3389/fpsyg.2015.00751
41.
HodappA.RabovskyM. (2021). The N400 ERP component reflects an error-based implicit learning signal during language comprehension. European Journal of Neuroscience, 54(9), 7125–7140. https://doi.org/10.1111/ejn.15462
42.
HoedemakerR. S.ErnstJ.MeyerA. S.BelkeE. (2017). Language production in a shared task: Cumulative semantic interference from self- and other-produced context words. Acta Psychologica, 172, 55–63. https://doi.org/10.1016/j.actpsy.2016.11.007
43.
HoedemakerR. S.MeyerA. S. (2019). Planning and coordination of utterances in a joint naming task. Journal of Experimental Psychology. Learning, Memory, and Cognition, 45(4), 732–752. https://doi.org/10.1037/xlm0000603
44.
IioT.ShiomiM.ShinozawaK.ShimoharaK.MikiM.HagitaN. (2015). Lexical entrainment in human robot interaction. International Journal of Social Robotics, 7(2), 253–263. https://doi.org/10.1007/s12369-014-0255-x
KaanE.HarrisA.GibsonE.HolcombP. (2000). The P600 as an index of syntactic integration difficulty. Language and Cognitive Processes, 15(2), 159–201. https://doi.org/10.1080/016909600386084
47.
KilnerJ. M.PaulignanY.BlakemoreS. J. (2003). An interference effect of observed biological movement on action. Current Biology: CB, 13(6), 522–525. https://doi.org/10.1016/s0960-9822(03)00165-9
48.
KirtayM.WudarczykO. A.PischeddaD.KuhlenA. K.RahmanR. A.HaynesJ.-D.HafnerV. V. (2020). Modeling robot co-representation: State-of-the-art, open issues, and predictive learning as a possible framework. 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), Valparaiso, Chile, pp. 1–8. https://doi.org/10.1109/ICDL-EpiRob48136.2020.9278031
49.
KuhlenA. K.Abdel RahmanR. (2017). Having a task partner affects lexical retrieval: Spoken word production in shared task settings. Cognition, 166, 94–106. https://doi.org/10.1016/j.cognition.2017.05.024
50.
KuhlenA. K.Abdel RahmanR. (2021). Joint language production: An electrophysiological investigation of simulated lexical access on behalf of a task partner. Journal of Experimental Psychology. Learning, Memory, and Cognition, 47(8), 1317–1337. https://doi.org/10.1037/xlm0001025
51.
KuhlenA. K.Abdel RahmanR. (2023). Beyond speaking: Neurocognitive perspectives on language production in social interaction. Philosophical Transactions of the Royal Society B: Biological Sciences, 378(1875), 20210483. https://doi.org/10.1098/rstb.2021.0483
52.
KutasM.FedermeierK. D. (2011). Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP). Annual Review of Psychology, 62, 621. https://doi.org/10.1146/annurev.psych.093008.131123
53.
KutasM.HillyardS. A. (1980). Reading between the lines: Event-related brain potentials during natural sentence processing. Brain and Language, 11(2), 354–373. https://doi.org/10.1016/0093-934X(80)90133-9
MaessB.MamashliF.ObleserJ.HelleL.FriedericiA. D. (2016). Prediction signatures in the brain: Semantic pre-activation during language comprehension. Frontiers in Human Neuroscience, 10, Article 591. https://www.frontiersin.org/articles/10.3389/fnhum.2016.00591
56.
MargeM.Espy-WilsonC.WardN. G.AlwanA.ArtziY.BansalM.BlankenshipG.ChaiJ.DauméH.DeyD.HarperM.HowardT.KenningtonC.Kruijff-KorbayováI.ManochaD.MatuszekC.MeadR.MooneyR.MooreR. K.. . .YuZ. (2022). Spoken language interaction with robots: Recommendations for future research. Computer Speech & Language, 71, 101255. https://doi.org/10.1016/j.csl.2021.101255
57.
MartinC. D.BranziF. M.BarM. (2018). Prediction is production: The missing link between language production and comprehension. Scientific Reports, 8(1), 1079. https://doi.org/10.1038/s41598-018-19499-4
58.
MartinC. D.MolnarM.CarreirasM. (2016). The proactive bilingual brain: Using interlocutor identity to generate predictions for language processing. Scientific Reports, 6(1), Article 1. https://doi.org/10.1038/srep26171
59.
MathôtS.SchreijD.TheeuwesJ. (2012). OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods, 44(2), 314–324. https://doi.org/10.3758/s13428-011-0168-7
60.
MissF. M.AdriaenseJ. E. C.BurkartJ. M. (2022). Towards integrating joint action research: Developmental and evolutionary perspectives on co-representation. Neuroscience and Biobehavioral Reviews, 143, 104924. https://doi.org/10.1016/j.neubiorev.2022.104924
61.
MolnarM.Ibáñez-MolinaA.CarreirasM. (2015). Interlocutor identity affects language activation in bilinguals. Journal of Memory and Language, 81, 91–104. https://doi.org/10.1016/j.jml.2015.01.002
62.
MoubayedS. A.BeskowJ.SkantzeG. (2013). The Furhat social companion talking head.
63.
MüllerB. C. N.BrassM.KühnS.TsaiC.-C.NieuwboerW.DijksterhuisA.van BaarenR. B. (2011). When Pinocchio acts like a human, a wooden hand becomes embodied. Action co-representation for non-biological agents. Neuropsychologia, 49(5), 1373–1377. https://doi.org/10.1016/j.neuropsychologia.2011.01.022
64.
NassC.MoonY. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81–103. https://doi.org/10.1111/0022-4537.00153
65.
NavarreteE.BasagniB.AlarioF.-X.CostaA. (2006). Does word frequency affect lexical selection in speech production?Quarterly Journal of Experimental Psychology, 59(10), 1681–1690. https://doi.org/10.1080/17470210600750558
66.
NewB.PallierC.FerrandL.MatosR. (2001). Une base de données lexicales du français contemporain sur internet: LEXIQUETM//A lexical database for contemporary french : LEXIQUETM. Annee Psychologique—ANNEE PSYCHOL, 101, 447–462. https://doi.org/10.3406/psy.2001.1341
67.
NieuwlandM. S. (2019). Do “early” brain responses reveal word form prediction during language comprehension? A critical review. Neuroscience and Biobehavioral Reviews, 96, 367–400. https://doi.org/10.1016/j.neubiorev.2018.11.019
68.
NieuwlandM. S.Van BerkumJ. J. A. (2006). When peanuts fall in love: N400 evidence for the power of discourse. Journal of Cognitive Neuroscience, 18(7), 1098–1111. https://doi.org/10.1162/jocn.2006.18.7.1098
69.
OhC. S.BailensonJ. N.WelchG. F. (2018). A systematic review of social presence: Definition, antecedents, and implications. Frontiers in Robotics and AI, 5, Article 114. https://doi.org/10.3389/frobt.2018.00114
70.
PaetzelM.CastellanoG. (2019). Let me get to know you better: Can interactions help to overcome uncanny feelings?Proceedings of the 7th International Conference on Human-Agent Interaction, Kyoto, Japan, pp. 59–67. https://doi.org/10.1145/3349537.3351894
71.
PickeringM. J.GarrodS. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27(2), 169–190. https://doi.org/10.1017/S0140525X04000056
72.
PickeringM. J.GarrodS. (2006). Alignment as the basis for successful communication. Research on Language and Computation, 4(2), 203–228. https://doi.org/10.1007/s11168-006-9004-0
73.
PickeringM. J.GarrodS. (2007). Do people use language production to make predictions during comprehension?Trends in Cognitive Sciences, 11(3), 105–110. https://doi.org/10.1016/j.tics.2006.12.002
74.
PickeringM. J.GarrodS. (2013a). An integrated theory of language production and comprehension. Behavioral and Brain Sciences, 36(04), Article 04. https://doi.org/10.1017/S0140525X12001495
75.
PickeringM. J.GarrodS. (2013b). Forward models and their implications for production, comprehension, and dialogue. Behavioral and Brain Sciences, 36(4), 377–392. https://doi.org/10.1017/S0140525X12003238
76.
RamscarM.DyeM.McCauleyS. (2013). Error and expectation in language learning: The curious absence of “mouses” in adult speech. Language, 89, 760–793. https://doi.org/10.1353/lan.2013.0068
77.
RauchbauerB.NazarianB.BourhisM.OchsM.PrévotL.ChaminadeT. (2019). Brain activity during reciprocal social interaction investigated using conversational robots as control condition. Philosophical Transactions of the Royal Society B: Biological Sciences, 374(1771), 20180033. https://doi.org/10.1098/rstb.2018.0033
78.
SahaïA.DesantisA.GrynszpanO.PacherieE.BerberianB. (2019). Action co-representation and the sense of agency during a joint Simon task: Comparing human and machine co-agents. Consciousness and Cognition, 67, 44–55. https://doi.org/10.1016/j.concog.2018.11.008
79.
SalemM.EysselF.RohlfingK.KoppS.JoublinF. (2013). To err is human(-like): Effects of robot gesture on perceived anthropomorphism and likability. International Journal of Social Robotics, 5(3), 313–323. https://doi.org/10.1007/s12369-013-0196-9
80.
SebanzN.BekkeringH.KnoblichG. (2006). Joint action: Bodies and minds moving together. Trends in Cognitive Sciences, 10(2), 70–76. https://doi.org/10.1016/j.tics.2005.12.009
SebanzN.KnoblichG.PrinzW. (2005). How two share a task: Corepresenting stimulus-response mappings. Journal of Experimental Psychology. Human Perception and Performance, 31(6), 1234–1246. https://doi.org/10.1037/0096-1523.31.6.1234
84.
SignoretC.AndersenL. M.DahlströmÖ.BlombergR.LundqvistD.RudnerM.RönnbergJ. (2020). The influence of form- and meaning-based predictions on cortical speech processing under challenging listening conditions: A MEG study. Frontiers in Neuroscience, 14, Article 573254. https://www.frontiersin.org/articles/10.3389/fnins.2020.573254
85.
SkantzeG. (2016). Real-time coordination in human-robot interaction using face and voice. AI Magazine, 37(4), Article 4. https://doi.org/10.1609/aimag.v37i4.2686
86.
StenzelA.ChinellatoE.BouM. A. T.del PobilÁ. P.LappeM.LiepeltR. (2012). When humanoid robots become human-like interaction partners: Corepresentation of robotic actions. Journal of Experimental Psychology: Human Perception and Performance, 38(5), 1073–1077. https://doi.org/10.1037/a0029493
87.
StrijkersK.CostaA. (2011). Riding the lexical speedway: A critical review on the time course of lexical selection in speech production. Frontiers in Psychology,2, Article 356. https://doi.org/10.3389/fpsyg.2011.00356
88.
StrijkersK.CostaA. (2016). The cortical dynamics of speaking: Present shortcomings and future avenues. Language, Cognition and Neuroscience, 31(4), 484–503. https://doi.org/10.1080/23273798.2015.1120878
89.
StrijkersK.CostaA.ThierryG. (2010). Tracking lexical access in speech production: Electrophysiological correlates of word frequency and cognate effects. Cerebral Cortex, 20(4), 912–928. https://doi.org/10.1093/cercor/bhp153
ThunbergS.ArnelidM.ZiemkeT. (2022). Older adults’ perception of the Furhat robot. Proceedings of the 10th International Conference on Human-Agent Interaction, Christchurch, New Zealand, pp. 4–12. https://doi.org/10.1145/3527188.3561924
92.
Van PettenC.LukaB. J. (2012). Prediction during language comprehension: Benefits, costs, and ERP components. International Journal of Psychophysiology, 83(2), 176–190. https://doi.org/10.1016/j.ijpsycho.2011.09.015
93.
VesperC.AbramovaE.BütepageJ.CiardoF.CrosseyB.EffenbergA.HristovaD.KarlinskyA.McEllinL.NijssenS. R. R.SchmitzL.WahnB. (2017). Joint action: Mental representations, shared information and general mechanisms for coordinating with others. Frontiers in Psychology, 7, Article 2039. https://www.frontiersin.org/articles/10.3389/fpsyg.2016.02039
94.
WudarczykO.KirtayM.KuhlenA. K.Abdel RahmanR.HaynesJ.-D.HafnerV. V.PischeddaD. (2021). Bringing together robotics, neuroscience, and psychology: Lessons learned from an interdisciplinary project. Frontiers in Human Neuroscience, 15, Article 630789. https://www.frontiersin.org/article/10.3389/fnhum.2021.630789
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
