What did you hear and what did you see? Understanding the transparency of facial recognition and speech recognition systems during human–robot interaction
Restricted accessResearch articleFirst published online October, 2025
What did you hear and what did you see? Understanding the transparency of facial recognition and speech recognition systems during human–robot interaction
As social robots begin to assume various social roles in society, the demand for understanding how social robots work and communicate grows rapidly. While literature on explainable artificial intelligence suggests that transparency about a social robot’s working mechanism can evoke users’ positive attitudes, transparency may also have negative outcomes. This study investigates the paradoxical effects of the transparency of facial recognition technology and speech recognition technology in human–robot interactions. Based on a lab experiment and combined analyses of users’ quantitative and qualitative responses, this study suggests that the transparency of facial recognition technology in human–robot interaction increases users’ social presence, reduces privacy concerns, and enhances users’ acceptance of robots. However, exposure to both facial and speech recognition technologies revives users’ privacy worries. This study further parses users’ open-ended evaluation of the prospective application of social robots’ tracking technologies and discusses the theoretical, practical, and ethical value of the findings.
AdadiABerradaM (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access6: 52138–52160.
2.
AnannyMCrawfordK (2018) Seeing without knowing: limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society20(3): 973–989.
3.
BasilMD (1996) The use of student samples in communication research. Journal of Broadcasting & Electronic Media40: 431–531.
4.
BioccaFHarmsCBurgoonJK (2003) Toward a more robust theory and measure of social presence: review and suggested criteria. Presence: Teleoperators & Virtual Environments12(5): 456–480.
5.
BrackenCCLombardM (2004) Social presence and children: praise, intrinsic motivation, and learning with computers. Journal of Communication54(1): 22–37.
6.
ChamikaraMAPBertokPKhalilI, et al. (2020) Privacy preserving face recognition utilizing differential privacy. Computers & Security97: 101951.
ChuahSHWAwECXYeeD (2021) Unveiling the complexity of consumers’ intention to use service robots: an fsQCA approach. Computers in Human Behavior123: 106870.
9.
CookTD (1985) Post-positivist critical multiplism. In: ShadishWRReichardtCS (eds) Reproduced in Evaluation Studies Review Annual. Thousand Oaks, CA: Sage, pp. 21–62.
10.
CummingsJJWertzEE (2023) Capturing social presence: concept explication through an empirical analysis of social presence measures. Journal of Computer-Mediated Communication28(1): zmac027.
11.
De GraafMMDraganAMalleBF, et al. (2021) Introduction to the special issue on explainable robotic systems. ACM Transactions on Human-Robot Interaction10(3): 1–4.
12.
DenzinNK (1978) The Research Act: A Theoretical Introduction to Sociological Methods. New York: McGraw Hill.
13.
EdwardsCEdwardsASpencePR, et al. (2016) Initial interaction expectations with robots: testing the human-to-human interaction script. Communication Studies67(2): 227–238.
14.
EmbgenSLuberMBecker-AsanoC, et al. (2012) Robot-specific social cues in emotional body language. In: 2012 IEEE RO-MAN: The 21st IEEE international symposium on robot and human interactive communication, Paris, 9–13 September, pp. 1019–1025. New York: IEEE.
15.
FioreSMWiltshireTJLobatoEJ, et al. (2013) Toward understanding social cues and signals in human–robot interaction: effects of robot gaze and proxemic behavior. Frontiers in Psychology4: 859.
16.
FischerKFothKRohlfingKJ, et al. (2011) Mindful tutors: linguistic choice and action demonstration in speech to infants and a simulated robot. Interaction Studies12(1): 134–161.
17.
FischerKWeigelinHMBodenhagenL (2018) Increasing trust in human–robot medical interactions: effects of transparency and adaptability. Paladyn, Journal of Behavioral Robotics9(1): 95–109.
18.
GongLNassC (2007) When a talking-face computer agent is half-human and half-humanoid: human identity and consistency preference. Human Communication Research33(2): 163–193.
19.
Hesse-BiberSLeavyP (2010) The Practice of Qualitative Research. Thousand Oaks, CA: Sage.
20.
JohnsonDGardnerJWilesJ (2004) Experience as a moderator of the media equation: the impact of flattery and praise. International Journal of Human-Computer Studies61(3): 237–258.
21.
KandaTMiyashitaTOsadaT, et al. (2008) Analysis of humanoid appearances in human–robot interaction. IEEE Transactions on Robotics24(3): 725–735.
22.
KhuranaAAlamzadehPChilanaPK (2021) ChatrEx: designing explainable chatbot interfaces for enhancing usefulness, transparency, and trust. In: 2021 IEEE symposium on visual languages and human-centric computing, St Louis, MO, 10–13 October, pp. 1–11. New York: IEEE.
23.
KimSSWatkinsEARussakovskyO, et al. (2023) Help me help the AI: understanding how explainability can support human-AI interaction. In: Proceedings of the 2023 CHI conference on human factors in computing systems, Hamburg, 23–28 April, pp. 1–17. New York: ACM.
24.
KrämerNCRosenthal-Von der PüttenAMHoffmannL (2015) Social effects of virtual and robot companions. In: SundarSS (ed.) The Handbook of the Psychology of Communication Technology. Hoboken, NJ: John Wiley & Sons, pp. 137–159.
25.
KruppMMRuebenMGrimmCM, et al. (2017) Privacy and telepresence robotics: what do non-scientists think? In: Proceedings of the Companion of the 2017 ACM/IEEE international conference on human-robot interaction, Vienna, 6–9 March, pp. 175–176. New York: ACM.
26.
KuchenbrandtDEysselFBobingerS, et al. (2013) When a robot’s group membership matters: anthropomorphization of robots as a function of social categorization. International Journal of Social Robotics5: 409–417.
27.
LawELCSoleimaniSWatkinsD, et al. (2021) Automatic voice emotion recognition of child-parent conversations in natural settings. Behaviour & Information Technology40(11): 1072–1089.
28.
LeeEJ (2008) Flattery may get computers somewhere, sometimes: the moderating role of output modality, computer gender, and user gender. International Journal of Human-Computer Studies66(11): 789–800.
29.
LeeEJ (2023) Minding the source: toward an integrative theory of human–machine communication. Human Communication Research50: hqad034.
30.
LeeKM (2004) Presence, explicated. Communication Theory14(1): 27–50.
31.
LeeKMNassC (2005) Social-psychological origins of feelings of presence: creating social presence with machine-generated voices. Media Psychology7(1): 31–45.
32.
LeeKMParkNSongH (2005) Can a robot be perceived as a developing creature? Effects of a robot’s long-term cognitive developments on its social presence and people’s social responses toward it. Human Communication Research31(4): 538–563.
33.
LeeKMPengWJinSA, et al. (2006) Can robots manifest personality? An empirical test of personality recognition, social responses, and social presence in human–robot interaction. Journal of Communication56(4): 754–772.
34.
LiCLiH (2023) Disentangling facial recognition payment service usage behavior: a trust perspective. Telematics and Informatics77: 101939.
35.
LiJJJuWReevesB (2017) Touching a mechanical body: tactile contact with body parts of a humanoid robot is physiologically arousing. Journal of Human-Robot Interaction6(3): 118–130.
36.
LinPCYanksonBChauhanV, et al. (2022) Building a speech recognition system with privacy identification information based on Google voice for social robots. The Journal of Supercomputing78(13): 15060–15088.
37.
LiuYLYanWHuB (2021) Resistance to facial recognition payment in China: the influence of privacy-related factors. Telecommunications Policy45(5): 102155.
38.
LombardMDittonT (1997) At the heart of it all: the concept of presence. Journal of Computer-Mediated Communication3(2): JCMC321.
39.
LombardMXuK (2021) Social responses to media technologies in the 21st century: the media are social actors paradigm. Human-Machine Communication2: 29–55.
40.
LutzCSchöttlerMHoffmannCP (2019) The privacy implications of social robots: scoping review and expert interviews. Mobile Media & Communication7(3): 412–434.
41.
MalleBF (2006) How the Mind Explains Behavior: Folk Explanations, Meaning, and Social Interaction. Cambridge, MA: MIT Press.
42.
Martínez-PlumedFGómezEHernández-OralloJ (2021) Futures of artificial intelligence through technology readiness levels. Telematics and Informatics58: 101525.
43.
MarwickAEboydd (2014) Networked privacy: how teenagers negotiate context in social media. New Media & Society16: 1051–1067.
44.
MillerT (2019) Explanation in artificial intelligence: insights from the social sciences. Artificial Intelligence267: 1–38.
45.
NassC (2004) Etiquette equality: exhibitions and expectations of computer politeness. Communications of the ACM47(4): 35–37.
46.
NassCFoggBJMoonY (1996) Can computers be teammates?International Journal of Human-Computer Studies45: 669–678.
47.
NassCMoonY (2000) Machines and mindlessness: Social responses to computers. Journal of Social Issues56(1): 81–103.
48.
NassCMoonYGreenN (1997) Are computers gender-neutral? Gender stereotypic responses to computers. Journal of Applied Social Psychology27: 864–876.
49.
NassCSteuerJTauberER (1994) Computers are social actors. In: Proceedings of SIGCHI ’94 human factors in computing systems, Boston, MA, 24–28 April, pp. 72–78. New York: ACM.
50.
NessetBRobbDALopesJ, et al. (2021) Transparency in HRI: trust and decision making in the face of robot errors. In: Companion of the 2021 ACM/IEEE international conference on human-robot interaction, Boulder, CO, 8–11 March, pp. 313–317. New York: ACM.
51.
PantanoE (2020) Non-verbal evaluation of retail service encounters through consumers’ facial expressions. Computers in Human Behavior111: 106448.
52.
PengY (2022) The role of ideological dimensions in shaping acceptance of facial recognition technology and reactions to algorithm bias. Public Understanding of Science32: 190–207.
53.
PetrescuRV (2019) Face recognition as a biometric application. Journal of Mechatronics and Robotics3: 237–257.
PradhanALazarAFindlaterL (2020) Use of intelligent voice assistants by older adults with low technology use. ACM Transactions on Computer-Human Interaction27(4): 1–27.
56.
RaiA (2020) Explainable AI: from black box to glass box. Journal of the Academy of Marketing Science48: 137–141.
57.
ReevesBNassC (1996) The Media Equation: How People Treat Computers, Television, and New Media Like Real People. Cambridge: Cambridge University Press.
58.
ReevesBHancockJLiuX (2020) Social robots are like real people: first impressions, attributes, and stereotyping of social robots. Technology, Mind, and Behavior1(1): 1–37.
59.
ReevesBYeykelisLCummingsJJ (2016) The use of media in media psychology. Media Psychology19(1): 49–71.
60.
RichardsRJSpencePREdwardsCC (2022) Human-machine communication scholarship trends: an examination of research from 2011 to 2021 in communication journals. Human-Machine Communication4: 45–62.
61.
RobinsonSMendelsonAL (2012) A qualitative experiment: research on mediated meaning construction using a hybrid approach. Journal of Mixed Methods Research6(4): 332–347.
62.
Rosenthal-Von der PüttenAMKrämerNCHerrmannJ (2018) The effects of humanlike and robot-specific affective nonverbal behavior on perception, emotion, and behavior. International Journal of Social Robotics10: 569–582.
63.
SchadenbergBRReidsmaDHeylenDK, et al. (2021) “I see what you did there” understanding people’s social perception of a robot and its predictability. ACM Transactions on Human-Robot Interaction10(3): 1–28.
64.
ShresthaMCOnishiTKobayashiA, et al. (2018) Communicating directional intent in robot navigation using project indicators. Proceedings of the 27th IEEE international symposium on robot and human interactive communication, Nanjing, China, 27–31 August, pp. 27–38. New York: IEEE.
65.
SongZ (2020) English speech recognition based on deep learning with multiple features. Computing102(3): 663–682.
66.
SpencePRWestermanDEdwardsC, et al. (2014) Welcoming our robot overlords: initial expectations about interaction with a robot. Communication Research Reports31(3): 272–280.
67.
SuchmanLA (2007) Human-Machine Reconfigurations: Plans and Situated Actions. Cambridge: Cambridge University Press.
68.
SuenHYHungKE (2023) Building trust in automatic video interviews using various AI interfaces: tangibility, immediacy, and transparency. Computers in Human Behavior143: 107713.
69.
VitaleJTonkinMHerseS, et al. (2018) Be more transparent and users will like you: a robot privacy and user experience design experiment. In: Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction, Chicago, IL, 5–8 March, pp. 379–387. New York: ACM.
70.
WeitzKSchillerDSchlagowskiR, et al. (2019) “Do you trust me?” Increasing user-trust by integrating virtual agents in explainable AI interaction design. Proceedings of the 19th ACM international conference on intelligent virtual agents, Paris, 2–5 July, pp. 7–9. New York: ACM.
71.
WheelessLRGrotzJ (1977) The measurement of trust and its relationship to self-disclosure. Human Communication Research3(3): 250–257.
72.
WilkinsonDAlkan Liao ÖQVMattettiM, et al. (2021) Why or why not? The effect of justification styles on chatbot recommendations. ACM Transactions on Information Systems39(4): 1–21.
73.
WolfCTRinglandKE (2020) Designing accessible, explainable AI (XAI) experiences. ACM SIGACCESS Accessibility and Computing125: 6.
74.
XieYZhuKZhouP, et al. (2023) How does anthropomorphism improve human-AI interaction satisfaction: a dual-path model. Computers in Human Behavior148: 107878.
75.
XuK (2019) First encounter with robot Alpha: how individual differences interact with vocal and kinetic cues in users’ social responses. New Media & Society21(11–12): 2522–2547.
76.
XuKLiaoT (2020) Explicating cues: a typology for understanding emerging media technologies. Journal of Computer-Mediated Communication25(1): 32–43.
77.
ZhangYLiaoQVBellamyRK (2020) Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In: Proceedings of the 2020 conference on fairness, accountability, and transparency, Barcelona, 27–30 January, pp. 295–305. New York: ACM.
78.
ZhaoS (2006) Humanoid social robots as a medium of communication. New Media & Society8(3): 401–419.
