Mixed-initiative collaboration between a humanoid robot and a virtual human through a common platform for a real-world common task: Evaluation and benchmarking

Abstract

This article presents mixed-initiative collaboration between two artificial agents of heterogeneous realities (a humanoid robot and a virtual human) for a real-world social task communicated through a common platform. A detailed framework for the collaboration is developed. Based on the framework, a representative real-world task to be performed through collaboration between the artificial agents for the wellbeing of humans is decided, which is to search for a missing (hidden) object in a homely environment. The agents are enriched with various similar intelligence and autonomy, functionalities and interaction modalities, and are integrated through a common communication platform based on a collaboration scheme so that they can collaborate (assist each other) to perform the real-world task. As a part of the collaboration scheme, a robot-virtual human bilateral trust model is derived and a real-time trust measurement method is developed so that the role of taking initiative in the collaboration can be switched between the agents triggered by the bilateral trust, which results in a mixed-initiative collaboration. An evaluation scheme is developed to evaluate the performance of the agents for the collaborative task. To benchmark their performance in the collaboration, human’s collaborations with the artificial agents and with some other allied agents for the same task are studied. The evaluation and benchmarking results show that both the robot and the virtual human perform satisfactorily in their collaboration, which proves the effectiveness of the real-world collaboration between the artificial agents of heterogeneous realities as well as justifies the effectiveness of the common platform and of the bilateral trust-based mixed-initiatives between the artificial agents. The results can be used to develop intelligent agents of heterogeneous realities to assist humans in various real-world tasks or help humans get real-world tasks performed in cooperation between artificial agents of heterogeneous realities.

Keywords

Human-robot interaction human-virtual human interaction robot-virtual human collaboration interaction evaluation benchmark common platform mixed-initiative trust smart homes assisted living telepresence

Get full access to this article

View all access options for this article.

References

Alkhalifah

et al., Using NAO humanoid robot in kindergarten: A proposed system, in: Proc. of IEEE 15th Int. Conf. on Advanced Learning Technologies, 2015, pp. 166–167.

Campbell,

Hays,

Core,

Birch and

Bosack, Interpersonal and leadership skills: Using virtual humans to teach new officers, in: Interservice/Industry Training, Simulation, and Education Conf., 2011, Paper No. 11358.

Castellano and

Peters, Socially perceptive robots: Challenges and concerns, Interaction Studies 11(2) (2010), 201–207. doi:10.1075/is.11.2.04cas.

Causo,

Vo,

Toh,

Chen,

Yeo and

Tzuo, Developing and benchmarking show & tell robotic puppet for preschool education, in: Proc. of IEEE Int. Conf. on Robotics and Automation, 2015, pp. 6114–6119.

Chang,

Tsai,

Guo and

Chen, Remote cloud data center backup using HBase and Cassandra with user-friendly GUI, in: Proc. of IEEE International Conference on Consumer Electronics, Taipei, 2015, pp. 420–421.

Chipalkatty,

Droge and

Egerstedt, Less is more: Mixed-initiative model-predictive control with human inputs, IEEE Trans. on Robotics 29(3) (2013), 695–703. doi:10.1109/TRO.2013.2248551.

Courgeon

et al., Joint attention simulation using eye-tracking and virtual humans, IEEE Trans. on Affective Computing 5(3) (2014), 238–250. doi:10.1109/TAFFC.2014.2335740.

Dragone,

Duffy and

O’Hare, Social interaction between robots, avatars & humans, in: IEEE Int. Workshop Robot Human Interactive Communication, 2005, pp. 24–29.

Ferrer,

Perdomo,

H.R.

Ali,

Fies and

Quarles, Virtual humans for temperature visualization in a tangible augmented reality educational game, in: Proc. of IEEE Virtual Reality Workshop on K-12 Embodied Learning Through Virtual & Augmented Reality, CA, 2017, pp. 1–6.

10.

Flinn,

Geary and

Ward, Ecological dominance, social competition, and coalitionary arms races: Why humans evolved extraordinary intelligence, Evolution and Human Behavior 26(1) (2005), 10–46. doi:10.1016/j.evolhumbehav.2004.08.005.

11.

Forland and

Russa, Virtual humans vs. anthropomorphic robots for education: How can they work together? in: Proc. of ASEE/IEEE Frontiers in Education Conf., 2005, p. S3G.

12.

Gratch,

Rickel,

Andre,

Badler,

Cassell and

Petajan, Creating interactive virtual humans: Some assembly required, in: IEEE Intelligent Systems, 2002, pp. 2–11.

13.

Guo,

Lv,

Zhou,

Peng and

Wang, Mixing control of animating virtual human for maintenance simulation, in: Proc. of 12th World Congress on Intelligent Control and Automation, Guilin, 2016, pp. 1091–1098.

14.

Hays,

Campbell,

Trimmer,

Poore,

Webb,

Stark and

King, Can role-play with virtual humans teach interpersonal skills? in: Proc. of Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), 2012, Paper No. 12318.

15.

Heitmeyer and

Leonard, Obtaining trust in autonomous systems: Tools for formal model synthesis and validation, in: Proc. of IEEE/ACM FME Workshop on Formal Methods in Software Engineering (FormaliSE), 2015, pp. 54–60.

16.

Hoff and

Bashir, Trust in automation: Integrating empirical evidence on factors that influence trust, Human Factors: The Journal of the Human Factors and Ergonomics Society 57(3) (2015), 407–434. doi:10.1177/0018720814547570.

17.

Hoffman, Evaluating fluency in human-robot collaboration, in: Proc. of Robotics: Science and Systems Workshop on Human-Robot Collaboration, 2013.

18.

Kang,

Subagdja,

Tan,

Ong and

Miao, Virtual characters in agent-augmented co-space, in: Proc. of the 11th Int. Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), Valencia, Spain, 2012, pp. 4–8.

19.

Kapadia,

Shoulson,

Boatright and

Huang, What’s next? The new era of autonomous virtual humans, LNCS 7660 (2012), 170–181.

20.

Kasap and

Thalmann, Intelligent virtual humans with autonomy and personality: State-of-the-art, Intelligent Decision Technologies 1 (2007), 3–15. doi:10.3233/IDT-2007-11-202.

21.

Kipp,

Ndiaye and

Patrick, Evaluating the tangible interface and virtual characters in the interactive COHIBIT exhibit, in: Proc. of the Int. Conf. on Intelligent Virtual Agents (IVA’06), Springer, 2006, pp. 434–444. doi:10.1007/11821830_35.

22.

Kondaxakis,

Gulzar and

Kyrki, Temporal arm tracking and probabilistic pointed object selection for robot to robot interaction using deictic gestures, in: Proc. of IEEE-RAS 16th Int. Conf. on Humanoid Robots, 2016, pp. 186–193.

23.

Kumar,

Kumar and

Giri, Design and implementation of three phase commit protocol (3PC) directory structure through remote procedure call (RPC) application, in: Proc. Int. Conf. Info. Communication and Embedded Sys., 2014, pp. 1–5.

24.

Lawford,

Narracott,

McCormack,

Bisbal,

Martin,

Brook,

Zachariou,

Kohl,

Fletcher and

Diaz-Zucczrini, Virtual physiological human: Training challenges, Phil. Trans. R. Soc. A 368(1921) (2010), 2841–2851. doi:10.1098/rsta.2010.0082.

25.

Lee and

Moray, Trust, self-confidence, and operators’ adaptation to automation, Int. Journal of Human-Computer Studies 40 (1994), 153–184. doi:10.1006/ijhc.1994.1007.

26.

Leite,

Martinho and

Paiva, Social robots for long-term interaction: A survey, International Journal of Social Robotics 5(2) (2013), 291–308. doi:10.1007/s12369-013-0178-y.

27.

Li,

Li and

Mu, Design of modular humanoid rehabilitation robot for apoplectic hemiplegia, in: Proc. of 29th Chinese Control and Decision Conf., Chongqing, 2017, pp. 7526–7532.

28.

Li,

Tee,

Chan,

Yan,

Chua and

Limbu, Continuous role adaptation for human–robot shared control, IEEE Trans. on Robotics 31(3) (2015), 672–681. doi:10.1109/TRO.2015.2419873.

29.

Pan,

Okada,

Uchiyama and

Suzuki, Direct and indirect social robot interactions in a hotel public space, in: Proc. of IEEE Int. Conf. on Robotics and Biomimetics, 2013, pp. 1881–1886.

30.

Panzirsch,

Weber,

Rubio,

Coloma,

Ferre and

Artigas, Tele-healthcare with humanoid robots: A user study on the evaluation of force feedback effects, in: Proc. of IEEE World Haptics Conference (WHC), 2017, pp. 245–250.

31.

Peng,

Harris and

Sawa, Detecting phishing attacks using natural language processing and machine learning, in: Proc. of 2018 IEEE 12th International Conference on Semantic Computing (ICSC), Laguna Hills, CA, 2018, pp. 300–301. doi:10.1109/ICSC.2018.00056.

32.

Qing-mei and

Ai-lian, Artificial emotional interaction model based on EFSM, in: Proc. of Int. Conf. on Electronic and Mechanical Engineering and Information Technology (EMEIT), Vol. 4, 2011, pp. 1852–1855.

33.

S.M.M.

Rahman, Generating human-like social motion in a human-looking humanoid robot: The biomimetic approach, in: Proc. of IEEE Int. Conf. on Robotics and Biomimetics, 2013, pp. 1377–1383.

34.

S.M.M.

Rahman, Evaluating and benchmarking the interactions between a humanoid robot and a virtual human for a real-world social task, in: Proc. of the 6th Int. Conference on Advances in Information Technology (IAIT 2013), Bangkok, Thailand, Dec 12–13, 2013, Communications in Computer and Information Science, Vol. 409, Springer, 2013, pp. 184–197.

35.

S.M.M.

Rahman, People-centric adaptive social ecology between humanoid robot and virtual human for social cooperation, in: Proc. of the 2nd International Workshop on Adaptive Robotic Ecologies (ARE 2013) at the 4th International Joint Conf. on Ambient Intelligence (AmI 2013), Dublin, Ireland, Dec 3–5, 2013, Communications in Computer and Information Science, Vol. 413, Springer, 2013, pp. 120–135.

36.

S.M.M.

Rahman, Collaboration between a physical robot and a virtual human through a unified platform for personal assistance to humans, in: Personal Assistants: Emerging Computational Technologies,

Costa,

Julian and

Novais, eds, Intel. Syst. Ref. Library Series, Vol. 132, Springer, 2017, pp. 149–177. ISBN 978-3-319-62529-4. doi:10.1007/978-3-319-62530-0_9.

37.

S.M.M.

Rahman, Cyber-physical-social system between a humanoid robot and a virtual human through a shared platform for adaptive agent ecology, IEEE/CAA Journal of Automatica Sinica 5(1) (2018), 190–203. doi:10.1109/JAS.2017.7510760.

38.

S.M.M.

Rahman, Improving performance and cybersecurity in an IoT-based system of systems integrating robots and virtual human, in: Proc. of the 2019 14th IEEE Annual Conference on System of Systems Engineering (SoSE), Anchorage, AK, USA, 2019, pp. 218–223. doi:10.1109/SYSOSE.2019.8753854.

39.

S.M.M.

Rahman,

S.M.

Chacko and

Kapila, Building trust in robots in robotics-focused STEM education under TPACK framework in middle schools, in Proc. of 2017 ASEE Annual Conf. & Exposition, USA, pp. 1–25, Paper ID #18477.

40.

S.M.M.

Rahman,

Sadr and

Wang, Trust-based optimal subtask allocation and model predictive control for human-robot collaborative assembly in manufacturing, in: Proc. of ASME Dynamic Systems and Controls Conference, 2015, Paper No. DSCC2015-9850.

41.

S.M.M.

Rahman and

Wang, Dynamic affection-based motion control of a humanoid robot to collaborate with human in flexible assembly in manufacturing, in: Proc. of ASME Dynamic Systems and Controls Conference, 2015, Paper No. DSCC2015-9841.

42.

Reina

et al., Augmented reality for robots: Virtual sensing technology applied to a swarm of e-pucks, in: Proc. of NASA/ESA Conference on Adaptive Hardware and Systems (AHS), Montreal, QC, Canada, 2015, pp. 1–6.

43.

Rezvani,

Driggs-Campbell,

Sadigh,

S.S.

Sastry,

S.A.

Seshia and

Bajcsy, Towards trustworthy automation: User interfaces that convey internal and external awareness, in: Proc. of IEEE 19th Int. Conf. on Intelligent Transportation Systems (ITSC), Rio de Janeiro, 2016, pp. 682–688.

44.

Rodriguez

et al., Humanizing NAO robot teleoperation using ROS, in: Proc. of 14th IEEE-RAS Int. Conference on Humanoid Robots (Humanoids), 2014, pp. 179–186.

45.

Roncone,

Mangin and

Scassellati, Transparent role assignment and task allocation in human robot collaboration, in: Proc. of 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 2017, pp. 1014–1021.

46.

Saleh, The value of virtual patients in medical education, Annals of Behavioral Science and Medical Education 16(2) (2010), 29–31. doi:10.1007/BF03355129.

47.

Salvador,

Silver and

M.H.

Mahoor, An emotion recognition comparative study of autistic and typically-developing children using the zeno robot, in: Proc. of IEEE Int. Conf. on Robotics and Automation, 2015, pp. 6128–6133.

48.

Sato,

Kamei,

Nishio and

Hagita, The ubiquitous network robot platform: Common platform for continuous daily robotic services, in: Proc. of 2011 IEEE/SICE Int. Symposium on System Integration, Kyoto, 2011, pp. 318–323.

49.

Schwiegelshohn

et al., A holistic approach for advancing robots in ambient assisted living environments, in: Proc. of IEEE 13th International Conference on Embedded and Ubiquitous Computing, Porto, 2015, pp. 140–147.

50.

SikLanyi,

Geiszt,

Karolyi and

Magyar, Virtual reality in special needs early education, The International Journal of Virtual Reality 5(4) (2006), 55–68.

51.

Swartout,

Gratch,

R.W.

Hill,

Hovy,

Marsella,

Rickel and

Traum, Toward virtual humans, AI Magazine 27(2) (2006), 96–108.

52.

Tham,

Duin,

Gee,

Ernst,

Abdelqader and

McGrath, Understanding virtual reality: Presence, embodiment, and professional practice, IEEE Transactions on Professional Communication 61(2) (2018), 178–195. doi:10.1109/TPC.2018.2804238.

53.

Wagoner

et al., Humanoid robots rescuing humans and extinguishing fires for cooperative fire security system using HARMS, in: Proc. of 6th International Conference on Automation, Robotics and Applications, 2015, pp. 411–415.

54.

Wu,

Yan and

Fan, Data mining algorithms and statistical analysis for sales data forecast, in: Proc. of Int. Conf. on Computational Sciences and Optimization, 2012, pp. 577–581.

55.

Yoo and

Kim, Gaze control-based navigation architecture with a situation-specific preference approach for humanoid robots, IEEE/ASME Trans. on Mechatronics 20(5) (2015), 2425–2436. doi:10.1109/TMECH.2014.2382633.

56.

Yoshida

et al., RoboJockey: Designing an entertainment experience with robots, IEEE Computer Graphics and Applications 36(1) (2016), 62–69. doi:10.1109/MCG.2015.1.

57.

Yunzhong and

Wei, Leader follower multi-agents network with discrete-time rendezvous via swarm social system strategies, in: Proc. of 31st Chinese Control Conference (CCC), 2012, pp. 6081–6086.

58.

Zanatto,

Patacchiola,

Goslin and

Cangelosi, Priming anthropomorphism: Can the credibility of humanlike robots be transferred to non-humanlike robots? in: Proc. of 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Christchurch, 2016, pp. 543–544.