The paper presents an overview of the current state of research in intelligent avatars, with a particular focus on the detection and expression of emotions in virtual environments. Our interest lies particularly in the potential use of such tools for the development of medical (mental and physical health) training virtual environments, and virtual simulations and serious games in particular.
MiheljMNovakDBegušS. Virtual reality technology and applications. Dordrecht, Netherlands: Springer Science & Business Media, 2014.
2.
RienerRHardersM. Virtual Reality in Medicine. London, UK: Springer Science & Business Media, 2012.
3.
HallSTakahashiR. Augmented and virtual reality: The promise and peril of immersive technologies, McKinsey, New York, October 2017.
4.
HuffGSaxbergB. Full immersion – How will 10-year-olds learn? Educ. Next.2009; 9: 79-82.
5.
HuangHMLiawSSLaiCM. Exploring learner acceptance of the use of virtual reality in medical education: a case study of desktop and projection-based display systems. Interact. Learn. Environ.2016 Jan 2; 24(1): 3-19.
6.
GleggSMLevacDE. Barriers, facilitators and interventions to support virtual reality implementation in rehabilitation: a scoping review. PM&R.2018; 1;10(11): 1237-51.
7.
SchwartzRSteptoeW. The Immersive VR Self: Performance, Embodiment and Presence in Immersive Virtual Reality Environments. In A Networked Self and Human Augmentics, Artificial Intelligence, Sentience 2018 Jul 11, pp. 124-132.
8.
ShinD. Empathy and embodied experience in virtual environment: To what extent can virtual reality stimulate empathy and embodied experience? Comput. Human Behav.2018; 78: 64-73.
9.
LeeSCarlsonGJonesSJohnsonALeighJRenambotL. Designing an expressive avatar of a real person. In: AllbeckJBadlerNBickmoreTPelachaudCSafonovaA, editors. Intelligent virtual agents. Berlin, Heidelberg: Springer Lecture Notes in Computer Science, 2010; vol. 6356.
10.
FoxJAhnSJJanssenJHYeykelisLSegoviaKYBailensonJN. Avatars versus agents: a meta-analysis quantifying the effect of agency on social influence. Hum Comput Interact.2015 Sep 3; 30(5): 401-32.
11.
JenningsMDeenyP. ‘Acts of caring’: applied drama, puppetry, medical simulation and ‘sympathetic presence in person-centred nursing. In The Broken Puppet: A Symposium on Puppetry, Disability, and Health 2017 Jun 2.
12.
TanAJLeeCCLinPYCooperSLauLSChuaWLLiawSY. Designing and evaluating the effectiveness of a serious game for safe administration of blood transfusion: A randomized controlled trial. Nurse Educ. Today.2017 Aug 1; 55: 38-44.
13.
WiegmannDBrüggemannHRauschA. Design and Evaluation of a Human-Like Puppet as an Input Device for Ergonomic Simulation. In International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management 2017 Jul 9, pp. 393-403.
14.
Reid-SearlKO’NeillBDwyerTCrowleyK. Using a procedural puppet to teach pediatric nursing procedures. Clin Simul Nurs2017 Jan 1; 13(1): 15-23.
15.
TrowbridgeRLSnydmanLKSkolfieldJHaflerJBing-YouRG. A systematic review of the use and effectiveness of the Objective Structured Teaching Encounter. Med. Teach.2011 Nov 1; 33(11): 893-903.
16.
GriffithTDwyerTKinardCFlynnJRKirazianV. Research on the use of puppeteering to improve realism in army simulations and training games. In International Conference on Virtual, Augmented and Mixed Reality 2016 Jul 17, pp. 386-396.
17.
HametPTremblayJ. Artificial intelligence in medicine. Metabolism.2017 Apr 1; 69: S36-40.
18.
Bruce Damer and Randy Hinrichs, “The Virtuality and reality of Avatar Cyberspace” The Oxford Handbook of Virtuality, ed. Mark Grimshaw, 2014 Oxford: Oxford U Press pp. 17-41.
19.
ElliottAMAlexanderSCMescherCAMohanDBarnatoAE. Differences in physicians’ verbal and nonverbal communication with black and white patients at the end of life. J. Pain Symptom Manage.2016 Jan 1; 51(1): 1-8.
20.
De BylP. A conceptual affective design framework for the use of emotions in computer game design. Cyberpsychology.2015; Oct 1; 9(3).
21.
BailliePLukoseD. An affective decision making agent architecture using emotion appraisals. In Pacific Rim International Conference on Artificial Intelligence 2002; Aug 18, pp. 581-590.
22.
RobisonJMcQuigganSLesterJ. Evaluating the consequences of affective feedback in intelligent tutoring systems. In 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops 2009 Sep 10, pp. 1-6.
23.
SweetserPWyethP. GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain.2005 Jul 1; 3(3): 3.
24.
JohnsonDWilesJ. Effective affective user interface design in games. Ergonomics.2003; 46(13-14): 1332-1345.
25.
HudlickaE. Affective computing for game design. In Proceedings of the 4th Intl. North American Conference on Intelligent Games and Simulation 2008 Aug (pp. 5-12). McGill University Montreal, Canada.
26.
RuschDCKönigN. Barthes revisited: Perspectives on emotion strategies in computer games. Computer Philology Yearbook, 2007.
27.
HubbardJMannellRC. Testing competing models of the leisure constraint negotiation process in a corporate employee recreation setting. Leisure Sciences.2001 Jul 1; 23(3): 145-63.
28.
LeftonLA. Psychology. 5th ed. Boston: Allyn and Bacon, 1994.
29.
Marín-MoralesJHiguera-TrujilloJLGrecoAGuixeresJLlinaresCScilingoEPAlcañizMValenzaG. Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci Rep.2018 Sep 12; 8(1): 13657.
30.
Magnenat-ThalmannNKasapZ. Virtual humans in serious games. In 2009 International Conference on CyberWorlds 2009 Sep 7, pp. 71-79.
31.
ArellanoDVaronaJPeralesFJ. Generation and visualization of emotional states in virtual characters. Comput. Animat. Virt.2008; 19(3-4): 259-70.
32.
EkmanP. Expression and the nature of emotion. Approaches to Emotion.1984 Jan; 3: 19-344.
33.
EkmanP. An argument for basic emotions. Cognition & Emotion.1992; 6(3-4): 69-200.
34.
EkmanP. Facial expressions of emotion: an old controversy and new findings. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences.1992 Jan 29; 335(1273): 63-9.
35.
EkmanP. Basic emotions. Handbook of Cognition and Emotion. 1999; 45-60.
36.
JohnsonEHervásRMondéjarTBravoJOchoaSF. Improving social communication disorders through human-avatar interaction. In Ambient Intelligence for Health 2015 Dec 1, pp. 237-243.
37.
JohnsonEHervásRde la FrancaCGLMondéjarTOchoaSFFavelaJ. Assessing empathy and managing emotions through interactions with an affective avatar. Health Informatics J.2018; 24(2): 182-193.
38.
RinconJACostaANovaisPJulianVCarrascosaC. Using Emotions in Intelligent Virtual Environments: The EJaCalIVE Framework. Wireless Communications and Mobile Computing.2017; 2017.
39.
RoundJConradiEPoultonT. Improving assessment with virtual patients. Med Teach.2009 Jan 1; 31(8): 759-63.
40.
StrekalovaYAKriegerJLKleinhekselAJKotranzaA. Empathic Communication in virtual education for nursing students: I’m sorry to hear that. Nurse Educ.2017; 42(1): 18-22.
41.
GoriniAGaggioliARivaG. A second life for eHealth: prospects for the use of 3-D virtual worlds in clinical psychology. J Med Internet Res.2008; 10(3): e21.
42.
RehmICFoenanderEWallaceKAbbottJAKyriosMThomasN. What role can avatars play in e-mental health interventions? Exploring New models of client – therapist interaction. Front Psychiatry.2016 Nov 18; 7: 186.
43.
KozlowskiDHutchinsonMHurleyJRowleyJSutherlandJ. The role of emotion in clinical decision making: an integrative literature review. BMC Med Educ.2017; 17(1): 255. Published 2017 Dec 15. doi: 10.1186/s12909-017-1089-7.
44.
Blanch-HartiganD. An effective training to increase accurate recognition of patient emotion cues. Patient Educ. Couns.2012; 89(2): 274-280.
45.
WilcocksKKapralosBKalpinAChandrossDAbbassA. A dialogue intervention simulation framework to facilitate psychotherapy training. In Proceedings of the 2019 IEEE Italy Section Consumer Electronics Society International Symposium on Consumer Technologies, June 19-21 2019, Ancona, Italy.
46.
KayRHLoverockS. Assessing emotions related to learning new software: The computer emotion scale. Comput. Human Behav.2008 Jul 1; 24(4): 1605-23.
47.
LugrinJLZilchDRothDBenteGLatoschikME. Facebo: Real-time face and body tracking for faithful avatar synthesis. In 2016 IEEE Virtual Reality (VR) 2016 Mar 19, pp. 225-226.
48.
LeeJCheonMMoonSELeeJS. Gaze analysis of avatar-based navigation with different perspectives in 3D virtual space. In Proceedings of the 3rd International Conference on Human-Agent Interaction 2015 Oct 21, pp. 223-226.
49.
GreeneSThapliyalHCaban-HoltA. A survey of affective computing for stress detection: Evaluating technologies in stress detection for better health. IEEE Consumer Electronics Magazine.2016 Oct; 5(4): 44-56.
50.
SoleymaniMAsghari-EsfedenSFuYPanticM. Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affective Comput.2015 May 22; 7(1): 17-28.
51.
RayAChakrabartiA. Biophysical signal based emotion detection for technology enabled affective learning. In 2015 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT) 2015 Mar 5, pp. 1-6.
52.
MatlovicTGasparPMoroRSimkoJBielikovaM. Emotions detection using facial expressions recognition and EEG. In 2016 11th international workshop on semantic and social media adaptation and personalization (SMAP) 2016 Oct 20, pp. 18-23.
53.
ZhaoMAdibFKatabiD. Emotion recognition using wireless signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking 2016 Oct 3, pp. 95-108.
54.
HametPTremblayJ. Artificial intelligence in medicine. Metabolism.2017 Apr 1; 69: S36-40.
55.
LiuMFanDZhangXGongX. Human emotion recognition based on galvanic skin response signal feature selection and SVM. In 2016 International Conference on Smart City and Systems Engineering (ICSCSE) 2016 Nov 25, pp. 157-160.
56.
KanjoEAl-HusainLChamberlainA. Emotions in context: examining pervasive affective sensing systems, applications, and analyses. Pers. Ubiquit. Comput.2015 Oct 1; 19(7): 1197-212.
57.
BlackMHChenNTIyerKKLippOVBölteSFalkmerMTanTGirdlerS. Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography. Neurosci. Biobehav. Rev.2017 Sep 1; 80: 488-515.
58.
LuYZhengWLLiBLuBL. Combining eye movements and EEG to enhance emotion recognition. In Twenty-Fourth International Joint Conference on Artificial Intelligence. 2015 Jun 23.
59.
AlghowinemSAlShehriMGoeckeRWagnerM. Exploring eye activity as an indication of emotional states using an eye-tracking sensor. In Intelligent systems for science and information, 2014; pp. 261-276.
60.
ChiesaPALiuzzaMTAcciarinoAAgliotiSM. Subliminal perception of others’ physical pain and pleasure. Exp. Rrain Res.2015 Aug 1; 233(8): 2373-82.
61.
LiaoHIYoneyaMKidaniSKashinoMFurukawaS. Human pupillary dilation response to deviant auditory stimuli: Effects of stimulus properties and voluntary attention. Front. Neurosci.2016 Feb 17; 10: 43.
62.
OguroHSuyamaNKarinoKYamaguchiS. Pupillary response to auditory stimuli in depressive state. J. Neurol. Res.2016 Mar 9; 6(1): 8-11.
63.
SnowdenRJO’FarrellKRBurleyDErichsenJTNewtonNVGrayNS. The pupil’s response to affective pictures: Role of image duration, habituation, and viewing mode. J. Psychophysiol.2016 Aug; 53(8): 1217-23.
64.
RajanSChenniappanPDevarajSMadianN. Facial expression recognition techniques: a comprehensive survey. IET Image Processing.2019 Mar 22.
65.
MaoQRPanXYZhanYZShenXJ. Using Kinect for real-time emotion recognition via facial expressions. Frontiers of Information Technology & Electronic Engineering.2015 Apr 1; 16(4): 272-82.
66.
LeHTVeaLA. A customer emotion recognition through facial expression using Kinect sensors v1 and v2: A comparative analysis. In Proceedings of the 10th International Conference on Ubiquitous Information Management and Communication 2016 Jan 4, p. 80.
67.
RizzoAALucasGMGratchJStratouGMorencyLPChavezKShillingRSchererS. Automatic Behavior Analysis During a Clinical Interview with a Virtual Human. In Medicine Meets Virtual Reality. 2016 Apr 19, pp. 316-322.
68.
SwartoutWArtsteinRForbellEFoutzSLaneHCLangeBMorieJFRizzoASTraumD. Virtual humans for learning. AI magazine.2013 Dec 15; 34(4): 13-30.
69.
NorooziFKaminskaDCorneanuCSapinskiTEscaleraSAnbarjafariG. Survey on emotional body gesture recognition. IEEE Trans. Affective Comput.2018 Oct 16.
70.
KishorePVPrasadMVKumarDASastryAS. Optical flow hand tracking and active contour hand shape features for continuous sign language recognition with artificial neural networks. In 2016 IEEE 6th international conference on advanced computing (IACC) 2016 Feb 27, pp. 346-351.
71.
HuangJZhouWLiHLiW. Sign language recognition using real-sense. In 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP) 2015 Jul 12, pp. 166-170.
72.
AbhishekKSQubeleyLCFHoD. Glove-based hand gesture recognition sign language translator using capacitive touch sensor. In 2016 IEEE International Conference on Electron Devices and Solid-State Circuits (EDSSC), 2016, August, pp. 334-337.
73.
RangelFRossoP. On the impact of emotions on author profiling. Inf. Process. Manag.2016 Jan 1; 52(1): 73-92.
74.
BalahurAHermidaJMMontoyoA. Detecting implicit expressions of emotion in text: A comparative analysis. Decis. Support Syst.2012 Nov 1; 53(4): 742-53.
75.
SuttlesJIdeN. Distant supervision for emotion classification with discrete binary values, In Proceedings of the 14th International Conference on Computational Linguistics and Intelligent Text Processing 2013 March 24-30, 2013, pp. 121-136.
76.
CanalesLMartínez-BarcoP. Emotion detection from text: A survey. In Proceedings of the Workshop on Natural Language Processing in the 5th Information Systems Research Working Days (JISIC) 2014, pp. 37-43.
77.
KratzwaldBIlićSKrausMFeuerriegelSPrendingerH. Deep learning for affective computing: Text-based emotion recognition in decision support. Decis. Support Sys.2018 Nov 1; 115: 24-35.
78.
HuangJZhouWLiHLiW. Sign language recognition using real-sense. In 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP) 2015 Jul 12, pp. 166-170.
79.
BerteroDFungP. A first look into a convolutional neural network for speech emotion detection. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), 2017, March, pp. 5115-5119. IEEE.
SchererKRShumanVFontaineJSoriano SalinasC. The GRID meets the Wheel: Assessing emotional feeling via self-report, 2013.
82.
BänzigerTTranVSchererKR. The Geneva Emotion Wheel: A tool for the verbal report of emotional reactions. In Proceeding so the ISRE.149; (2005): 271-294.
83.
PennebakerJWRoberts,TA. Toward a his and hers theory of emotion: Gender differences in visceral perception. J Soc Clin Psychol.1992; 11(3): 199-212.
