Sage Journals: Discover world-class research

Abstract

Background

The addition of biofeedback and artificial intelligence (AI) in simulation training and serious games has shown promising results in improving the effectiveness of training and can lead to increased engagement, motivation, and retention of information. This systematic literature review explores the integration of biofeedback and artificial intelligence into eXtended reality (XR) training scenarios and is the first review to provide a consolidated overview of applied biofeedback and AI technologies in this area.

Method

This review was conducted using keywords related to biofeedback, AI, XR, and training and included papers that: contained the use of biofeedback and AI in XR training scenarios; reported on at least one outcome related to training effectiveness; were published in English; were peer-reviewed; date from 1 January 2016 – 7 February 2022.

Results

The results indicate that many studies collect two or more biosignals using a single biosensing device. This is particularly relevant in applied settings, where ease of use and minimal interference in training/education activities is desired. Also, that light, portable devices such as wrist bands, wireless straps, or headbands are preferred. Additionally, eye tracking, electrodermal activity (EDA), and photoplethysmograms (PPG) present as particularly useful biomarkers of stress and/or cognitive load in XR training contexts. A wide variety of machine learning (ML) approaches were used to support biofeedback systems in XR environments. However, a limited number of studies employed real-time analysis of biosignals (just 1% of studies) which indicates current challenges in implementing such systems.

Conclusion

The majority of papers meeting the selection criteria were from the fields of education and healthcare. Further research in other domains, such as defense and general industry, is needed to gain a comprehensive understanding of the potential for biofeedback and AI integration in XR training scenarios used in these domains.

Keywords

artificial intelligence biofeedback review extended reality simulation-based training

Introduction

The use of artificial intelligence (AI) and biofeedback or physiological measurement in eXtended Reality (XR) training applications is seeing increasing use (Marín-Morales et al., 2021; Suhaimi et al., 2021). Studies have found that the integration of biofeedback and AI in simulation training or games can lead to increased engagement (Houzangbe et al., 2020), motivation (Ciolacu et al., 2020), and retention of information (Leiker et al., 2016). It can also help to reduce stress and anxiety during training, leading to better performance (Sharma et al., 2022). Further, research has been conducted in various fields such as education, healthcare, and defense, showing the potential benefits of using biofeedback and AI in simulation training (Ciolacu & Svasta, 2021).

Use of biofeedback in these contexts often aims to promote emotion-regulation of trainees (Jerčić & Sundstedt, 2019) with biosignals aiding the understanding of training (Habibnezhad et al., 2021), and/or providing measures of training performance (Jerčić & Sundstedt, 2019). In emotion-regulation contexts, trainees can be presented with a real-time visual indicator of their physiological state during tasks, enabling trainees to learn and/or practice self-regulation techniques (Parnandi & Gutierrez-Osuna, 2017).

The capture of biosignals can also provide insight into training affect, or the degree to which a trainee is affected by the virtual training task and environment (Kim, Kim, & Ahn, 2021). This insight is useful for broader considerations of training fidelity in stressful and/or emotionally charged contexts. Also, the change in measured physiological signals over the training continuum may indicate training effectiveness, where trainees, through practice, are able to undertake training tasks with increasing resilience (Gamble, et al. 2018). In these areas, the use of AI is typically associated with processing of biosignals (signal processing, pattern recognition) and may potentially be applied in the actuation of biofeedback in virtual environments (see Figure 1).

Figure 1.

Biofeedback following a general closed loop model. Within virtual environments using XR technologies, the use of AI may focus on processing of biosignals (signal processing and pattern recognition) and/or the actuation of dynamic feedback. Decisions regarding biosensors and choice of biofeedback actuation approaches will also be impacted by technology constraints (Adapted from (van den Broek & Westerink, 2012)).

However, for XR practitioners seeking to add biosignal collection and/or biofeedback into their training environments, a number of challenges present. In addition to determining what might be a useful measure, for example cognitive load, heartrate, or temperature, there is a plethora of different technologies available to collect this data. This is further complicated with some technologies supporting single biosignal collection and others supporting the collection of multiple different biosignals. Also, there are physical constraints on how different technologies are applied that may limit their compatibility, for example, the use of multiple head-worn devices to collect electroencephalogram (EEG), eye movement, and forehead temperature may not be practical depending on the XR hardware used. In training and learning contexts, the use of biosensors that capture real-time physiological and/or emotional states are an important step toward achieving adaptive synthetic training environments (Seyderhelm et al., 2019).

In addition to the physical challenges, each biosignal device will collect data at different granularities and have different levels of noise in the data that will need to be removed (Stangl et al., 2023). For real-time systems, this will need to be done on-on-fly and, additionally for post-session data analysis, the volume of data collected can be significant. Thus, the use of AI techniques become attractive, to clean noisy data (Saganowski, 2022), and to support any downstream data use, for example for feature classification (Delvigne et al., 2020; Sakib et al., 2020). Again, for the XR practitioner seeking to add AI support into biofeedback enabled XR experiences, determining the appropriate technologies and approaches can be a significant challenge.

This paper aims to support XR practitioners by summarizing the features of recent use cases from the research literature involving the integration of biofeedback and AI into XR training systems. This paper presents results across several core aspects including biosignal types, measurement categories, biofeedback and AI technology types and frequency of use. Thus, the research overview presented here provides a resource for XR practitioners looking for examples of current practice and analogous exemplars to support their own custom biofeedback and AI integration needs.

Method

A systematic review of literature seeks to take a “snapshot” of the current state of the art within the academic literature in a specific area using a repeatable search and review approach. To establish appropriate search terms to guide the identification of literature, a research question for this study was defined as:

What biofeedback technologies and approaches are being used within AI enabled XR systems for training and/or educational applications?

PRISMA guidelines were used as the basis of this systematic literature review (Moher et al., 2015). PRISMA guidelines provide specific methodology details relating to eligibility criteria, information sources, search strategy, and study records (including data management, selection process, and collection process). This section provides specific details on how these guidelines were applied for the review.

An initial scoping review, searching the existing research, revealed many theoretical and untested-on-human concepts and approaches. From this scoping review, the key inclusion criteria for this literature review were defined. One main requirement was an inclusion criterion that the included biofeedback enabled systems in the research articles were validated on real human participants and therefore, exemplars of real use cases, with actual experimental results.

To locate the most relevant literature, a set of parameters were defined before the examination of the literature began (Table 1). The literature was sourced from Scopus (https://www.scopus.com/home.uri) noting its recognition as the bibliometric database with the largest coverage of research paper relevant to this review (Meho & Rogers, 2008) and was restricted to common peer-reviewed document types such as conference papers, journal articles, and book chapters.

Table 1.

Parameters and search sources.

Bibliographic Database	Scopus
Article Types	Conference Paper, Journal Article, Book Chapter
Search On	Title, Abstract, Keywords
Sort on Return	Relevance
Language	English
Period of Publication	2016-2022
Date Conducted	7^th February 2022 (7/2/2022)

Within Scopus, the search results were limited to documents whose Title, Abstract, or Keywords, met the search criteria (see Table 2). The search was conducted in early February 2022 and returned articles in English for the period of 1 January 2016 – 7 February 2022. Given the advances in technologies for capturing biosignals and creating XR experiences, and consequential proliferation of literature relating to this, only literature from this five-year window was considered.

Table 2.

Inclusion and exclusion criteria for the search string.

Criteria 1: Inclusion	“VR” or “virtual reality” or “AR” or “augmented reality” or “MR” or “mixed reality” or XR or “extended reality” AND
Criteria 2: Inclusion	“training” or “simulation” or “situational awareness” or “adaptive scenarios” or “dynamic scenarios” or "education" or "learning" AND
Criteria 3: Inclusion	“biometrics" OR "biosensors” or "sensors OR physiological OR wearables” AND
Criteria 4: Inclusion	“AI" or "artificial intelligence" or "ML" or "machine learning” or "deep learning"
Criteria 5: Excluded Terms	None

The inclusion and exclusion criteria for the search string were developed to maximize the number of relevant studies being returned (see Table 2). Similar terms were grouped into specific criterion, for example, Criteria 1 searches for virtual reality related terms such as VR or virtual reality or XR or extended reality. Criteria 2 searches specifically for training, simulation, or scenarios keywords in studies where end-users engage with various educational domains. Criteria 3 returns studies that have used biosensors, sensors, or physiological measurement tools as part of their investigation. Criteria 4 relates to publications that discuss AI and machine learning aspects. No terms were specifically excluded from the search string.

These criteria were combined into the following Boolean search string which was run in Scopus to gather the most relevant literature:

TITLE-ABS-KEY( ( "VR" OR "virtual reality" OR "AR" OR "augmented reality" OR "MR" OR "mixed reality" OR "XR" OR "extended reality" ) AND ( "training" OR "simulation" OR "situational awareness" OR "adaptive scenarios" OR "dynamic scenarios" OR "education" OR "learning" ) AND ( "biometrics" OR "biosensors" OR "sensors" OR "physiological" OR "wearables" ) AND ( "AI" OR "artificial intelligence" OR ml OR "machine learning" OR "deep learning" ) ) AND ( LIMIT-TO ( PUBYEAR , 2022 ) OR LIMIT-TO ( PUBYEAR , 2021 ) OR LIMIT-TO ( PUBYEAR , 2020 ) OR LIMIT-TO ( PUBYEAR , 2019 ) OR LIMIT-TO ( PUBYEAR , 2018 ) OR LIMIT-TO ( PUBYEAR , 2017 ) OR LIMIT-TO ( PUBYEAR , 2016 ) ) AND ( LIMIT-TO ( DOCTYPE , "cp" ) OR LIMIT-TO ( DOCTYPE , "ar" ) OR LIMIT-TO ( DOCTYPE , "ch" ) ) AND ( LIMIT-TO ( LANGUAGE , "English" ) )

This string returned a total of 803 documents, including 324 Articles, 23 Book Chapters, and 455 Conference papers. From this set of documents 3 duplicates were removed, which left 800 studies to be screened (see Table 3).

Table 3.

Parameters and search sources.

Document Type	Count
Article	324
Book Chapter	23
Conference Paper	455
Duplicates Removed	3
Total Number of studies screened	800

A double-blind review process was then carried out using the Covidence content management system (https://www.covidence.org/). For this process, each study was independently screened by two people using the inclusion and exclusion criteria outlined in Table 4.

Table 4.

Inclusion and exclusion criteria for the studies.

Inclusion	Exclusion
Does not meet exclusion criteria Details the use of VR/AR/XR technology Includes physiological measurement /biofeedback/sensing equipment Implements or proposes use of AI/machine learning/deep learning Must include some implementation details for an education/training use case Must include an evaluation with a minimum of 5 participants	Does not meet inclusion criteria Full text not available Not in English Insufficient detail for assessment of utility Review only No evaluation, or less the 5 participants in evaluation

To be included in the results and analysis, a study must have included details of an XR technology. The study must also have included the capture of some form of physiological measurement in addition to using AI or machine learning methods in the data processing and/or analysis. Systems must have been implemented, and some details of a use case in an education/training context must have been provided. These studies must also have included a minimum of 5 participants in an evaluation and must not meet any of the exclusion criteria in Table 4.

Additionally, for a study to be included in the results, the full text of the document must be fully available online and presented in English. If the paper had insufficient details of the assessment of its utility, the study was excluded. Reviews and evaluations were also excluded as they would not meet the inclusion criteria of examining a use case with at least 5 participants. Once the documents were screened using this double-blind peer review process (see Figure 2), data from the remaining articles (n= 48) were extracted for analysis.

Figure 2.

PRISMA flow for the selection of relevant literature.

Biosignal Results

The following sections summarize the biosignal collection approaches and consider these in the context of each study's primary measurement category. Primary measurement categories refer to the key user physiological and/or emotional response that the researchers sought to identify. Table 5 defines the acronyms used in this section.

Table 5.

Acronym summary.

Acronym	Term
AR	Augmented Reality
ECG	Electrocardiogram
EDA	Electrodermal activity
EEG	Electroencephalogram
EMG	Electromyography
ET	Eye Tracking
fNIRS	Functional near-infrared spectroscopy
HR	Heart rate
HRV	Heart Rate Variability
MR	Mixed Reality
PPG	Photoplethysmography
RSP	Respiration
SST	Skin Surface Temperature
VR	Virtual Reality
XR	Extended Reality

Biosignal Collection Summary and Primary Measurement Categories

Within each of the final 48 studies, between one (1) and seven (7) biosignals were obtained, with the collection of a single biosignal the most commonly occurring (n=22). However, most studies (n=26) collected 2 or more biosignals (see Figure 3). The maximum number of individual sensors used was three (3), with most studies using only one (1) (n=31, 65%) or two (2) (n=16, 33%) sensing devices.

Figure 3.

Frequency of the number of biosignals captured in studies.

The use of biofeedback in the research papers evaluated in this review addressed a range of primary measures (see Figure 4). Emotion recognition in the users/participants was the most frequently primary measure, followed by measures of stress and attention. Three of the studies categorized as “stress” here considered anxiety as the primary measure, however, given the biosignal was captured while the stimulus (stressor) was present, these are effectively also capturing elements of stress and thus were considered within the single “stress” category.

Figure 4.

Primary measures by number of studies.

Ten different biosignals were captured across the different studies. When considering the use of biosignals by primary measure, stress made use of the most individual biosignal types (see Table 6 and Figure 5).

Table 6.

Frequency of biosignals captured in studies by primary measure.

Primary Measure	ECG	EDA	EEG	EMG	ET	fNIRS	HR	PPG	RSP	SST	Total
Attention		1	3		5						9
Authentication					2						2
Cognitive Load	2	4		1	5		2	4	2	3	23
Cybersickness	2	1	2					1			6
Effort/Motivation	1	2	2				2	1		1	9
Emotions	5	4	5	1			1				18
Other	2	2		2	1						7
Stress	3	4	2		3	2	5	3	1	1	24
Total	15	18	14	4	16	2	10	9	5	5	98

Figure 5.

Primary measures by biosignals captures across the number of studies.

Electrodermal activity (EDA), also referred to as galvanic skin response (GSR), was the most widely used biosignal, with most application in the measurement of stress, cognitive load, and emotions. Eye tracking was also frequently used in the measurement of stress, cognitive load, and attention. The relevance for these usage patterns is provided in the summary of biosignals provided in Table 6.

Technologies and Equipment Used for Biosignal Collection

A total of 42 different biosensor devices were identified in the studies. These sensors covered a range of different implementation approaches, from purpose-built integrations (Luong et al., 2020), experimental lab setups (Hsiao et al., 2020; Zhang et al., 2017) to implementations using commercial off-the-shelf equipment (Migovich et al., 2021; Sakib et al., 2020). A full summary of the biosensing technologies used in the studies reviewed, together with the associated biosignals captured by these technologies, is provided in Table 7.

Table 7.

Biosensing technologies and associated biosignals.

Biosensor	ECG	EDA	EEG	EMG	ET	fNIR	HR	PPG	RSP	SST	Total
32 electrode actiCHamp recorder			1								1
64-channel AntNeuro dry electrode cap			1								1
ActiCHamp			1								1
Actiheart 5	2						1				3
Arduino		1		1			2				4
B-Alert x10	1										1
Biopac acquisition set	2	3		1			1	2	2	3	14
Bitalina	1	1									2
Brainvision 64 Channel ActiCHamp EEG			1								1
Cognionics AIM Gen2	1										1
Easycap GmbH)			1								1
ECG100C hardware module	1										1
EDA100C hardware module		1									1
EMOTIV EPOC+ 14 Channel Mobile EEG			1								1
Emotive Insight			1								1
Empatica E4	1	4		1			3	2	1	2	14
g.USBamp				1							1
Garmin Vivosmart HR							1				1
Hololens					1						1
HTC Vive Pro Eye					11						11
IBMT’sBiofeedbackSystem(BFS)		1					1				2
InterAxon Muse			1								1
Maxim MAX30102								1			1
Muse			2								2
Muse 2016			4								4
NeurOne Tesla				1							1
NeXus10 MKII	1										1
NIRSport2						2					2
Not identified							1				1
Polar Electro							1				1
Procomp5 Infiniti System	1	1									2
Samsung Gear Fit							1				1
Shimmer3 GSR+	1	3						2		1	7
Sony Smartband 2							1				1
Tobii					4						4
Trigno EKG	1										1
Wearable designed by FuBIntLab	1	1									2
Wireless EMG sensors				1							1
Wireless HR Monitor (brand not specified)							1				1
World Famous Electronics PPG Sensor							1	1			2
Zephyr BioJHarness							1		1		2
Total	14	16	14	6	16	2	16	8	4	6	102

Each of the biosignals in the studies enables biofeedback mechanisms and can form the basis of AI/machine learning approaches to remove data noise, process signals into usable data, and/or identify patterns. The biosignals themselves provide insights into physiological and/or emotional processes in users/participants. Descriptions of each of these biosignals is provided in Table 8, together with the primary measures captured, studies incorporating them, and an image demonstrating the implementation of the biosignal from a sample study.

Table 8.

Biosensing technologies and associated biosignals captured.

Biosignal / Primary Measure(s)	Description	Studies Incorporating	Technologies/ Implementations
Eye Tracking (ET)Stress, Cognitive Load, Attention, Other	Eye behaviour which can provide valuable information on higher cognitive functions and state of affect (via pupil dilations) and/or attention (via gaze). Measures include Gaze, Fixations, Saccades, Pupil Dilation, Blink rate.	(Abdurrahman et al., 2021), (Bhalla et al., 2021),(De Bruyne et al., 2021),(Delvigne et al., 2022), (Delvigne et al., 2020), (Georgiou & Demiris, 2017), (Jones et al., 2020), (Liebers et al., 2021), (Leiker et al., 2016),(Migovich et al., 2021),(Noghabaei et al., 2021),(Porssut et al., 2022), (Shi et al., 2020), (Tsai et al., 2021), (Wilson et al., 2021), (Zhang et al., 2017)	From (Porssut et al., 2022)
Electrodermal activity (EDA)Stress, Cognitive Load, Emotions, Effort /Motivation, Attention, Other	Measuring the electrical properties of the skin that are most influenced by sweating. Commonly used to detect changes in sympathetic activity caused by emotional and cognitive processing, thus often applied to gauge stress and emotional reactions. Measures include Skin Conductance Level (SCL), Skin Potential Response (SPR), Mean, Standard Deviation, Peaks.	(Arquissandas et al., 2023), (Bauer et al., 2019), (Bui et al., 2019), (Cho et al., 2017), (Dell’Agnola et al., 2020), (Granato et al., 2020), (Ihmig et al., 2020), (Islam et al., 2019), (Kim et al., 2019), (Luong et al., 2020), (Martin et al., 2020), (Migovich et al., 2021), (Sakib et al., 2020), (Sayis et al., 2022), (Tabbaa et al., 2022), (Tsai et al., 2021), (Wilson et al., 2021), (Zhang et al., 2017)	From (Martin et al., 2020)
Electroencephalogram (EEG)Emotions, Attention, Stress, Cybersickness, Effort / Motivation	Measuring the electrical activity of the brain on the scalp. Measures include Frequency Bands (Alpha, Beta, Gamma, Delta, and Theta), Mean Bandpower, Event Related Potential, Mean Amplitude.	(Bauer et al., 2019), (Bin Suhaimi et al., 2020), (Delvigne et al., 2022), (Delvigne et al., 2020), (Jones et al., 2020), (Kaur et al., 2019), (Kim et al., 2020), (McDermott et al., 2022), (Noghabaei et al., 2021), (Recenti et al., 2021), (Suhaimi et al., 2020), (Suhaimi et al., 2021), (Suhaimi et al., 2022), (Suhaimi et al., 2017)	From (Suhaimi et al., 2021)
Electromyography (EMG)Other, Cognitive Load, Emotions	Measuring muscle contraction strength from electrical signals. Primarily used for measuring human movement, although facial EMG (fEMG) has been found useful for measuring affect. Measures include Mean, Standard Deviation, Minimum, Maximum, Peaks.	(Dwivedi et al., 2020), (Granato et al., 2020), (Hsiao et al., 2020), (Recenti et al., 2021), (Zhang et al., 2017)	From (Hsiao et al., 2020)
Electrocardiogram (ECG) and Heart Rate (HR)Cognitive Load, Cybersickness, Stress, Effort / Motivation, Emotions, Attention, Other	Measurement of cardiovascular activity through electrical (ECG) or blood pressure (BP) sensors. Common contextual measures include Heart Rate (HR), Heart Rate Variability (HRV), Low-Frequency power, High-Frequency power.	(ECG)(Dell’Agnola et al., 2020), (Zhang et al., 2017), (Oh & Kim, 2021), (Kim et al., 2020), (Rodrigues et al., 2021), (Sayis et al., 2022), (Arquissandas et al., 2023), (Bauer et al., 2019), (Granato et al., 2020), (Kaur et al., 2019), (Marín-Morales et al., 2021), (Sakib et al., 2020), (Sakib et al., 2022), (Tabbaa et al., 2022), (Tsai et al., 2021)(HR)(Abdurrahman et al., 2021), (Aljabri et al., 2020), (Bauer et al., 2019), (Chauhan et al., 2018), (Cho et al., 2017), (Ihmig et al., 2020), (Jones et al., 2020), (Migovich et al., 2021), (Stranick & Lopez, 2022), (Zhang et al., 2017)	From (Sakib et al., 2022) From (Sakib et al., 2022)
Photoplethysmogram (PPG)	Measurement of cardiovascular activity through optical sensors. Like ECG/HR, common contextual measures include Heart Rate (HR), Heart Rate Variability (HRV), Low-Frequency power, High-Frequency power.	(Chauhan et al., 2018), (Dell’Agnola et al., 2020), (Kim et al., 2019), (Luong et al., 2020), (Martin et al., 2020), (Migovich et al., 2021), (Sakib et al., 2020), (Wilson et al., 2021), (Zhang et al., 2017)	From (Chauhan et al., 2018)
Respiration (RSP)Cognitive Load, Emotions, Stress	Measurement of breathing activity, with variations associated physiological responses to cognitive and emotional state (Park et el., 2021). Measures include Respiration Rate, Tidal Volume, O₂ consumption, Mean, Standard Deviation, Peaks.	(Bui et al., 2019), (Dell’Agnola et al., 2020), (Granato et al., 2020), (Jones et al., 2020), (Zhang et al., 2017)	From (Bui et al., 2019)
Skin Surface Temperature (SST)Cognitive Load, Effort / Motivation, Stress	Also referred to as Skin Temperature (SKT), measures the temperature of the certain areas of the skin. Measures include Mean, Maximum, and Minimum.	(Cho et al., 2017), (Dell’Agnola et al., 2020), (Kim et al., 2019), (Wilson et al., 2021), (Zhang et al., 2017)	From (Wilson et al., 2021)
Functional near-infrared spectroscopy (fNIR)Stress	A non-invasive optical imaging technique that measures changes in haemoglobin (Hb) concentrations in the brain. FNIR can be used to distinguish cognitive load from stress (Parent et al., 2019). Measures include oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) concentrations. HbO and HbR yields the change in total hemoglobin (HbT) for a specific region of the brain.	(Abujelala et al., 2021), (Shi et al., 2020)	From (Abujelala et al., 2021)

Details on the specific software environments for the capture and analysis of the biosignal data is not discussed in this paper. Approaches vary from custom built software solutions, use of existing software tools and libraries, to complete off-the-shelf proprietary software available with some biosensing technologies. A summary of tools for the recording, synchronization, and processing of physiological data can be found in (Halbig & Latoschik, 2021).

AI/ML Results

Each of the included papers (n=48) were evaluated to determine the type of machine learning approach implemented, whether single or multiple approaches were used, and if these were applied in real-time or not. Following this, the specific uses for these implementations were considered, together with a listing of the definitions of all approaches used with the references to the specific research studies from the review that they were used in.

AI/ML Approach

Within each of the final 48 studies, between one (1) and eleven (11) machine learning (ML) approaches were used, with the use of a single ML approach the most commonly occurring (n=25) (Figure 6).

Figure 6.

Number of ML approaches used per paper.

On investigation, only four (1%) of articles implemented real-time AI in their solutions (Figure 7 and Table 9). In an additional 17 studies (15%), it was unclear whether the implementation resulted in real-time application of AI.

Figure 7.

Number of papers that used real-time approaches.

Table 9.

Studies with real-time application AI technologies.

Reference	# of AI/ML Approaches	Primary Use	AI/ML 1	AI/ML 2	AI/ML 3
Porssut et al., 2022	1	Noise reduction for real-time control	Reinforcement Learning
Aljabri et al., 2020	1	Feature Extraction	Multi-Layer Perceptron classifier (which in the name itself connects to a Neural Network)
Islam et al., 2019	1	Feature Extraction	Neural Network Classifier
Martin et al., 2020	3	Detect VR Sickness Levels	Gradient Boosting (GB)	Random Forest (RF)	Support Vector Machine (SVM)

AI/ML Uses and Definitions of Approaches

The most common primary use involved classification tasks, with classification of user/trainee emotion a leading application (Table 10). Feature extraction (from biosignals) was also commonly used, resulting in the application of ML in the data processing phase of the biofeedback loop (see Figure 1). However, some broader applications focusing on decoding user interactions in virtual environments (Foxman, 2019) and estimating “game” intensity levels (Rodrigues et al., 2021). A brief description of each of the ML/AI technologies considered is provided in Table 11

Table 10.

Primary use of AI technologies.

Primary Use	Count
Classify Attention	1
Classify Data (baseline vs. scenario)	1
Classify Data (pre-VR and post-VR stimulation)	1
Classify Data, not defined further	2
Classify Emotion	8
Classify multiple tasks	1
Classify participants	1
Correlate bio-signals and subjective reporting for motion sickness	1
Data Classification	1
Data encoding and retrieval	1
Decodes user interactions in a virtual environment	1
Detect Emotion (stress)	1
Detect VR Sickness Levels	1
Electrodermal and heart rate used to build a model	1
Emotion tracking (stress)	1
Estimate game intensity levels	1
Feature Extraction	7
Identify data patterns	1
Identify participants and gestures	1
Measure/Evaluate Cognitive Workload	2
Model Training	1
No Details Provided	1
Noise reduction for real-time control	1
Patient Treatment (Phobia)	1
Pattern recognition	1
Performance Tracking	1
Predict Cyber Sickness	1
Predict Emotions	1
Predicting activity level	2
Read HRV and RSP to build a Cyber sickness model	1
Training Data	1
Total	48

Table 11.

Definitions of machine learning and AI technologies used in the studies reviewed.

AI/ML Tech	Description	Studies Incorporating
AdaBoost	A “boosting” algorithm, where boosting “is an approach to machine learning based on the idea of creating a highly accurate prediction rule by combining many relatively weak and inaccurate rules” ⁺⁺⁺⁺	(Abujelala et al., 2021), (Chauhan et al., 2018), (Oh & Kim, 2021), (Recenti et al., 2021)
Artificial neural network (ANN)	Artificial Neural Network (ANN) is a computational model in machine learning, which is inspired by the biological structures and functions of the mammalian brain. Such a model consists of multiple units called artificial neurons which build connections between each other to pass information. The advantage of such a model is that it progressively “learns” the tasks from the given data without specific programming for a single task. ⁺⁺	(Abdurrahman et al., 2021), (Islam et al., 2019), (Zhang et al., 2017)
Convolutional Neural Network (CNN)	A convolutional neural network is a specific type of artificial neural network that uses perceptrons, a machine learning unit algorithm, for supervised learning, to analyze data. CNNs apply to image processing, natural language processing, and other kinds of cognitive tasks. A convolutional neural network is also known as a ConvNet. A CNN consists of an input and output layer as well as multiple hidden layers which are formed as mathematical operations. The hidden layers include convolutional layer, pooling layer, normalization, and fully connected layers. Since the success of AlexNet applied the ImageNet competition in 2013, there has been a rapid evolution of CNNs. VGGNet, GoogLeNet, ResNet, and DenseNet are some successful examples. ⁺⁺	(Delvigne et al., 2020), (Delvigne et al., 2022)
Decision Tree (DT)	A decision tree uses tree-like graph or model as a structure to perform decision analysis. It uses each node to represent a test on an attribute, each branch to represent the outcome of the test, and each leaf node to represent a class label. ⁺⁺	(Abdurrahman et al., 2021), (Abujelala et al., 2021), (Kim et al., 2019), (Recenti et al., 2021), (Rodrigues et al., 2021), (Sayis et al., 2022), (Tsai et al., 2021), (Zhang et al., 2017)
Deep Learning	Deep Learning is a subfield of machine learning concerned with algorithms that are inspired by the human brain that works in a hierarchical way. Deep Learning models, which are mostly based on the (artificial) neural networks, have been applied to different fields, such as speech recognition, computer vision, and natural language processing. ⁺⁺	(Jones et al., 2020), (Kim et al., 2020), (Suhaimi et al., 2020)
Discriminant Analysis	Discriminant analysis builds a predictive model for group membership. The model is composed of a discriminant function (or, for more than two groups, a set of discriminant functions) based on linear combinations of the predictor variables that provide the best discrimination between the groups. The functions are generated from a sample of cases for which group membership is known; the functions can then be applied to new cases that have measurements for the predictor variables but have unknown group membership. ⁺⁺⁺	(Abdurrahman et al., 2021), (Zhang et al., 2017)
Feedforward Artificial Neural Network (FANN)	A feedforward neural network is an artificial neural network in which the connections between units do not form a cycle. The feedforward neural network has an input layer, hidden layers, and an output layer. Information always travels in one direction—from the input layer to the output layer—and never goes backward. ⁺⁺	(Sakib et al., 2022), (Suhaimi et al., 2021)
Gradient Boosting Trees (GBT)	A gradient boost machine is a type of machine learning technique that uses an ensemble of weak prediction models to perform regression and classification tasks. ⁺⁺	(Abujelala et al., 2021), (Bin Suhaimi et al., 2020), (Granato et al., 2020), (Martin et al., 2020), (Recenti et al., 2021), (Suhaimi, et al., 2022)
K-Nearest Neighbour (KNN)	The k-Nearest Neighbour (KNN) family of classification algorithms and regression algorithms is often referred to as memory-based learning or instance-based learning. Sometimes, it is also called lazy learning. These terms correspond to the main concept of KNN. The concept is to replace model creation by memorizing the training data set and then use this data to make predictions. ⁺⁺⁺	(Abdurrahman et al., 2021), (Abujelala et al., 2021), (Bhalla et al., 2021), (Kaur et al., 2019), (Liebers et al., 2021), (Migovich et al., 2021), (Noghabaei, et al. 2021), (Oh & Kim, 2021), (Recenti et al., 2021), (Rodrigues et al., 2021), (Suhaimi et al., 2017), (Suhaimi et al., 2020), (Suhaimi, et al., 2022), (Zhang et al., 2017)
Logistic Regression (LR)	Logistic regression is a kind of statistical analysis that is used to predict the outcome of a dependent variable based on prior observations. For example, an algorithm could determine the winner of a presidential election based on past election results and economic data. Logistic regression algorithms are popular in machine learning. ⁺⁺	(Abujelala et al., 2021), (Stranick & Lopez, 2022), (Tsai et al., 2021)
Naive Bayes (NB)	A family of simple probabilistic classifiers based on applying Bayes’ theorem with strong independence assumptions between the features. ⁺	(Abujelala et al., 2021), (Bin Suhaimi et al., 2020), (Stranick & Lopez, 2022), (Suhaimi, et al., 2022)
Random Forest (RF)	Random Forests or Random Decision Forests are ensembling learning methods for data classification and regression. They construct a multitude of decision trees during the training and output the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. ⁺⁺	(Abujelala et al., 2021), (Bhalla et al., 2021), (Dwivedi et al., 2020), (Granato et al., 2020),(Luong et al., 2020), (Martin et al., 2020), (Oh & Kim, 2021), (Recenti et al., 2021), (Suhaimi, et al., 2022), (Wilson et al., 2021)
Support Vector Machine (SVM)	Support Vector Machine, or in short SVM, is a supervised machine learning model for data classification and regression analysis. One of the most used classifiers in machine learning. It optimizes the width of the gap between the points of separate categories in feature space. ⁺⁺	(Abdurrahman et al., 2021), (Abujelala et al., 2021), (Bhalla et al., 2021), (Bui et al., 2019), (De Bruyne et al., 2021), (Granato et al., 2020), (Kim et al., 2019), (Martin et al., 2020), (Marín-Morales et al., 2021), (Noghabaei et al., 2021), (Oh & Kim, 2021), (Recenti et al., 2021), (Rodrigues et al., 2021), (Stranick & Lopez, 2022),(Suhaimi et al., 2017),(Suhaimi et al., 2020),(Suhaimi, et al., 2022), (Tabbaa et al., 2022), (Zhang et al., 2017)

+ From “The Appen Artificial Intelligence Glossary”, https://appen.com/ai-glossary/ [last access 19/12/2023].

++ From “AI: A Glossary of Terms”, Jiapan Guo, Violet Farhang-Razi and Paul Algra (editor), https://link.springer.com/content/pdf/bbm:978-3-319-94878-2/1.pdf [last access 19/12/2023].

+++ From https://www.ibm.com/docs/ [last access 19/12/2023].

++++ From Schapire, R. E. (2013). Explaining AdaBoost. In Empirical Inference (pp. 37-52). Springer, Berlin.

Discussion

The results of the systematic literature review have highlighted several trends in the integration of biofeedback and artificial intelligence into XR training scenarios. The identification of these trends provides a useful starting point for the types of decisions that XR practitioners will need to make when considering using these technologies. In the following sections we highlight the implications of the results found, firstly in the context of biosignals/biofeedback and secondly when considering the use of AI technologies.

Many studies reviewed collected two or more biosignals using a single biosensing device. This is particularly relevant in applied settings, where ease of use and minimal interference in training/education activities is desired (Sakib et al., 2020). For example, Luong et al. (2020) notes, in terms of cumbersomeness, that there are advantages to integrating multiple physiological sensors into a VR HMD to monitor a user’s psychological state, rather than having to fit and maintain multiple sensors. Granato et al. (2020) observe that almost all the electrodes used in their experiments were located near the hands and face of the users. Therefore, there are opportunities to integrate multiple sensor technologies directly into the hardware being used, and in this case, a gamepad and a VR headset. With increasing use of off-shelf technologies, the provision for multiple biosignal capture in such technologies will be vital when moving from controlled laboratory/hospital environments to support more widespread usage (Tabba et al., 2022).

Light, portable devices such as wrist bands, wireless straps, or headbands are preferred (Migovich et al., 2021) for the collection of biosignals. Suhaimi et al. (2021) used a wearable EEG headset device that was made to be “portable and easy to set up without the need of any adhesive material to attach the electrodes onto the skin of any participant, the device also removes the need for a trained medical professional to operate the device”. Also, Wilson et al. (2021) observed that many applications can get “good-enough” results using only the sensors from the Empatica E4 wrist-band. Therefore, light and unobtrusive, while still providing the required biosignal support, is an important consideration when selecting biofeedback technology.

In terms of biosignals, eye tracking, electrodermal activity (EDA), and photoplethysmograms (PPG) present as particularly useful biomarkers of stress and/or cognitive load in XR training contexts. All can be captured with minimal interference to end users and have robust research supporting their use (Cho et al., 2017; Chauhan et al, 2018; Dell’Agnola et al., 2020; Kim et al, 2019; Luong et al., 2020). In addition to being used in primary data analysis, these biosignals are useful as a secondary source to support machine learning approaches (Cho et al., 2017; Jones et al., 2020; Martin et al., 2020). The generation and distribution of the relevant datasets will be important for growth in the use of these approaches.

Of those using studies using off-the-shelf biosensing equipment, the Empatica E4 emerged as the most common. Of note is the capacity for the device to capture multiple biosignals, although this is a noted feature of similar devices in the market. Also, a “plug and play” approach (Aljabri et al., 2020) to sensing technology is recommended, allowing sensing technologies to be updated/upgraded overtime while the fundamental benefits of the biofeedback implementation are retained.

As such, details regarding integration of biosensing technologies with synthetic environment development tools (i.e., game engines) should be a focus of development approaches. The Unity game engine was used in several studies (including De Bruyne et al., 2021; Delvigne et al., 2022; Dwivedi et al., 2020; Marín-Morales et al., 2021; Porssut et al., 2022; Rodrigues et al., 2021; Stranick & Lopez, 2022; Tsai et al., 2021) and the Unreal Engine 4 was used in only one study from the 48 studies included (McDermott et al., 2022). Hardware and software integration is important for system development and Unity is currently dominating the “lock-in”, where platforms are made dependent on and interoperable with each other to assure market viability (Foxman, 2019). It remains to be seen whether this current domination will extend from gaming into future XR application development.

A wide variety of machine learning (ML) approaches were applied to the task of implementing biofeedback systems in XR environments. Although there is significant promise, the limited number of studies in this review employing real-time analysis of biosignals (1% of studies) point to some of the challenges in implementing such systems. Biosignal data is commonly “noisy”, containing a variety of data artefacts from sources such as human movement (Zhang et al., 2017) and sensor connection issues. As such, biosignal data typically requires data cleaning, processing and/or filtering prior to analysis (Sakib et al., 2020). This is further compounded by individual variations in human biosignals (e.g., heart-rate variations (Marín-Morales et al., 2021)), which can introduce an additional transformation or baselining step to the pre-analysis process. Undertaking this preparatory work in real-time without introducing time lags that impact on provision of the feedback component of the biofeedback loop, or on the immersive experience, may not be feasible. Working with reduced datasets or sources, applied to a pre-trained ML system, is thus preferable (Sakib et al., 2020).

An additional challenge in identifying ML implementations of use in XR training systems incorporating biofeedback lies in the level of explanatory detail provided in some studies. Of the studies reviewed, 17 (15%) did not include sufficient detail to determine whether the implementation was in real-time or applied to pre-recorded data. The reasons for this lack of detail are not clear, although the cross-disciplinary nature of the work might offer some explanation. For example, teams involved in these studies may include members from the application domain (e.g., Healthcare professionals) together with members from the technical and/or computer science areas. Where articles are published in the application domain, fewer technical details may be included to improve the relevance of the work to the intended audience. Irrespective of this, care should be taken in inferring the appropriateness of approaches with limited technical detail on any implementation.

Despite these issues, the results of the review identified several ML approaches relevant to biofeedback enabled XR training/education solutions. Classification approaches dominate the results, followed by feature extraction. From the studies, 13 different ML algorithms (or algorithmic families) were identified, with K-Nearest Neighbour (KNN), decision trees, and artificial neural networks (ANN) featuring heavily. However, the implementation of ML approaches should be guided by current best practices, and we have produced three basic recommendations:

• Application of data reduction techniques to select features (variables) with the highest prediction/classification fit should be performed as part of the machine learning pipeline.

• Implementations should benchmark machine learning approaches and algorithms to identify optimum fit to the specific data and application/use in terms of accuracy, precision, and speed.

• Consideration of the representativeness of data used for training machine learning algorithms is important. Overfitting and lack of generalizability are well-known issues that can be alleviated through the inclusion of sufficiently diverse training data representative of the full data space.

There are two main limitations with the review presented in this article. Firstly, the literature was sourced from a single repository, namely Scopus. However, Meho and Rogers (2008) completed a comparison of Scopus and Web of Science and found that Scopus provided a more compressive listing of research, specifically due to the inclusion of conference publications, namely in the ACM and IEEE peer-reviewed conference proceedings and the Springer Lecture Notes in Computer Science series. For computing research, conference publication forms a significant volume of current research and given the computing and human-computer interaction base of the biosignal/XR domains, Scopus is an appropriate choice. With the addition of using the formalism of the PRISMA guidelines as the basis of this systematic literature review, we are confident our snapshot of the literature is comprehensive.

Conversely, the snapshot nature of the literature review is the second limitation. Our review was conducted in February 2022 and covered the period of 1 January 2016 – 7 February 2022. As with any technology-based review, new publications can outdate the reviews results. Nevertheless, our review snapshot does provide insight across a 5-year period and has identified several interesting trends in both the use and integration of the technologies considered. Given the pace of journal article publication, our review is strengthened by the focus on the Scopus repository and the inclusion of peer-reviewed conference articles, which are often published in a timelier manner. Thus, this review presents as a benchmark across working technologies rather than a state-of-the-art review. As is often the case, tried and tested exemplars are the most useful for practitioners and this is the target audience for this review.

Conclusion

This review has highlighted the recent research literature implementing XR technologies in combination with biofeedback and AI approaches, with a focus on the specific biofeedback sensors used in the context of simulation training or education contexts.

The results indicated a number of trends including the efficiencies of multiple biosignal data collection with single biosensing devices, the need for practical considerations, such as the use of light and portable devices, and that some biosignals, such as eye tracking, electrodermal activity (EDA), and photoplethysmograms (PPG) present as useful biomarkers of stress and/or cognitive load which are likely to be of particular interest in XR training contexts. Although a wide variety of machine learning (ML) approaches were identified, only a limited number employed real-time analysis of biosignals (1% of studies) which indicates current challenges in implementing such systems and the need for more work in this area.

Papers that met the selection criteria were predominately from the fields of education and healthcare, indicating the increasing use of biofeedback and AI technologies in these domains. However, other domains, such as defence and general industry, are expanding the use of XR technologies for training so there is significant scope for the application of the lessons learnt from the studies documented in this review. Thus, this review serves as a valuable resource for researchers, practitioners, and policymakers interested in the use of XR, biofeedback, and AI in training and skill development.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by a Real Response Pty Ltd Research Grant.

ORCID iD

Shamus P. Smith

Author Biographies

Karen L. Blackmore received a B. IT (Spatial Analysis) (with Distinction) and Ph.D (Computer Science) from Charles Sturt University, Australia, in 1999 and 2008. She is a Professor in Computing and Information Technology at the University of Newcastle (Australia) and has research interests in extended reality technologies, human-computer interaction, and the use of immersive technologies and serious games for simulation-based training. She has published over 70 research articles and 13 industry reports. Professor Blackmore is a Senior Member of the IEEE.

Shamus P. Smith received B.Sc., B.Sc. (Honours) and Ph.D degrees in computer science from Massey University, New Zealand, in 1992, 1993 and 1999. He is a Senior Lecturer in Immersive Technologies at Griffith University (Australia) and has research interests in virtual reality, human-computer interaction and technology enhanced learning. He has published over 100 research articles. Dr Smith is a Senior Member of the IEEE.

Jacqueline D. Bailey received a B. IT and Ph.D IT from University of Newcastle, Australia, in 2015 and 2021. She is an Associate Lecturer in Computing and Information Technology at the University of Newcastle (Australia) and has research interests in human-centred computing, human-avatar interaction, human-computer interaction and immersive technologies such as virtual and augmented reality. She has published 9 research articles and 1 industry report.

Benjamin Krynski received B.Sc in 2010 and his BEmergHealth(Pmed) in 2016 from Monash University. He is the Co-Founder and Director of Future Projects at Real Response Pty Ltd an Australian simulation training provider and innovator of future training technologies. He is also a Registered Paramedic with 15+ years of domestic and international pre-hospital experience. Ben has a strong passion for human factors and building innovative and impactful systems for critical skill training.

References

Abdurrahman

U. A.

Yeh

S-C.

Wong

Wei

(2021). Effects of neuro-cognitive load on learning transfer using a virtual reality-based driving system. Big Data and Cognitive Computing, 5(4), https://doi.org/10.3390/bdcc5040054

Abujelala

Karthikeyan

Tyagi

Mehta

R. K.

(2021). Brain activity-based metrics for assessing learning states in VR under stress among firefighters: An explorative machine learning approach in neuroergonomics. Brain Sciences, 11(7). https://doi.org/10.3390/brainsci11070885

Aljabri

Rashwan

Qasem

Fakeeh

Albeladi

Sassi

(2020). Overcoming speech anxiety using virtual reality with voice and heart rate analysis. In Proceedings of the 13th International Conference on Developments in eSystems Engineering (DeSE), Liverpool, United Kingdom, 2020, page 311-316. https://doi.org/10.1109/DeSE51703.2020.9450783

Arquissandas

Lamas

D.R.

Oliveira

(2023). Moving from VR into AR using bio-cybernetic loops and physiological sensory devices for intervention on anxiety disorders. Virtual Reality, 27, 233–243. https://doi.org/10.1007/s10055-021-00549-8

Bauer

Bräuer

Schuldt

Niemann

& M.

Krömker

(2019). Application of wearable technology for the acquisition of learning motivation in an adaptive e-learning platform. In Ahram

(eds) Advances in Human Factors in Wearable Technologies and Game Design (AHFE 2018). Advances in Intelligent Systems and Computing, 795. Springer, Cham. https://doi.org/10.1007/978-3-319-94619-1_4

Bhalla

Sluganovic

Krawiecka

Martinovic

(2021). MoveAR: Continuous biometric authentication for augmented reality headsets. In Proceedings of the 7th ACM on Cyber-Physical System Security Workshop (CPSS '21), pages 41–52. Association for Computing Machinery, New York, NY, USA, 2021, https://doi.org/10.1145/3457339.3457983

Bin Suhaimi

N. S.

Mountstephens

Teo

(2020). Emotional state classification with distributed random forest, gradient boosting machine and naïve bayes in virtual reality using wearable electroencephalography and inertial sensing, In Proceedings of 2020 IEEE 10th Symposium on Computer Applications & Industrial Electronics (ISCAIE), Malaysia, 2020, pages 12-17. https://doi.org/10.1109/ISCAIE47305.2020.9108821

Bui

Veitch

Power

(2019). Autonomic nervous system approach to measure physiological arousal and scenario difficulty in simulation-based training environment. In Ayaz

Mazur

(eds) Advances in Neuroergonomics and Cognitive Engineering (AHFE 2018). Advances in Intelligent Systems and Computing, 775. Springer, Cham. https://doi.org/10.1007/978-3-319-94866-9_13

Chauhan

Reithinger

Mackey

J. R.

(2018). Real-time stress assessment through PPG sensor for VR biofeedback. In In the Proceedings of the 20th International Conference on Multimodal Interaction: Adjunct (ICMI '18), Association for Computing Machinery, New York, NY, USA, 2018, article 5, pages 1–5. https://doi.org/10.1145/3281151.3281156

10.

Cho

Ham

Park

Kim

Lee

N.-K

&Lee

(2017). Detection of stress levels from biosignals measured in virtual reality environments using a kernel-based extreme learning machine. Sensors, 17(10), 2435. https://doi.org/10.3390/s17102435

11.

Ciolacu

M. I.

Svasta

(2021). Education 4.0: AI empowers smart blended learning process with biofeedback, In Proceedings of the 2021 IEEE Global Engineering Education Conference (EDUCON), Vienna, Austria, 2021, pages 1443-1448, https://doi.org/10.1109/EDUCON46332.2021.9453959

12.

Ciolacu

M. I.

Svasta

Hart

Görzen

(2020). Education 4.0: Smart blended learning assisted by artificial intelligence, biofeedback and sensors. In Proceedings of the 2020 International Symposium on Electronics and Telecommunications (ISETC), Timisoara, Romania, 2020, pages 1-4, https://doi.org/10.1109/ISETC50328.2020.9301034.

13.

De Bruyne

Joundi

Morton

Van Kets

Van Wallendael

Talsma

Saldien

De Marez

Durnez

Bombeke

(2021). Smooth operator: A virtual environment to prototype and analyse operator support in CCTV surveillance rooms. In Proceedings of the 23rd HCI International Conference (HCII 2021), Part II, 1420, 2021, pages 233–240. https://doi.org/10.1007/978-3-030-78642-7_31.

14.

Dell’Agnola

Momeni

Arza

Atienza

(2020). Cognitive workload monitoring in virtual reality based rescue missions with drones. In Chen

J.Y.C.

Fragomeni

(eds) Virtual, Augmented and Mixed Reality. Design and Interaction. HCII 2020. Lecture Notes in Computer Science, 12190. Springer, Cham. 2021, https://doi.org/10.1007/978-3-030-49695-1_26

15.

Delvigne

Wannous

Dutoit

Ris

Vandeborre

J. -P.

(2022). PhyDAA: Physiological dataset assessing attention, IEEE Transactions on Circuits and Systems for Video Technology, 32(5), 2612-2623. https://doi.org/10.1109/TCSVT.2021.3061719

16.

Delvigne

Wannous

Vandeborre

J. -P.

Ris

Dutoit

(2020) Attention estimation in virtual reality with EEG based image regression, In Proceedings of the 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), Utrecht, Netherlands, 2020, pages 10-16. https://doi.org/10.1109/AIVR50618.2020.00012

17.

Dwivedi

Kwon

Liarokapis

(2020). EMG-based decoding of manipulation motions in virtual reality: Towards immersive interfaces. In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada, 2020, pages 3296-3303. https://doi.org/10.1109/SMC42975.2020.9283270

18.

Foxman

(2019). United We Stand: Platforms, Tools and Innovation With the Unity Game Engine. Social Media + Society, 5(4). https://doi.org/10.1177/2056305119880177

19.

Gamble

K. R.

Vettel

J. M.

Patton

D. J.

Eddy

M. D.

Davis

F. C.

Garcia

J. O.

Brooks

J. R.

(2018). Different profiles of decision making and physiology under varying levels of stress in trained military personnel. International Journal of Psychophysiology, 131, 73-80. https://doi.org/10.1016/j.ijpsycho.2018.03.017

20.

Georgiou

Demiris

(2017). Adaptive user modelling in car racing games using behavioural and physiological data. User Modeling and User-Adapted Interaction, 27, 267–311. https://doi.org/10.1007/s11257-017-9192-3

21.

Granato

Gadia

Maggiorini

Ripamonti

L. A.

(2020). An empirical study of players’ emotions in VR racing games based on a dataset of physiological data. Multimedia Tools and Applications, 79, 33657–33686. https://doi.org/10.1007/s11042-019-08585-y

22.

Habibnezhad

Shayesteh

Jebelli

Puckett

Stentz

(2021). Comparison of ironworker's fall risk assessment systems using an immersive biofeedback simulator. Automation in Construction, 122, 103471. https://doi.org/10.1016/j.autcon.2020.103471

23.

Halbig

Latoschik

M. E.

(2021). A systematic review of physiological measurements, factors, methods, and applications in virtual reality. Frontiers in Virtual Reality, 2. https://doi.org/10.3389/frvir.2021.694567

24.

Houzangbe

Christmann

Gorisse

Richir

(2020). Effects of voluntary heart rate control on user engagement and agency in a virtual reality game. Virtual Reality, 24, 665-681. https://doi.org/10.1007/s10055-020-00429-7

25.

Hsiao

C. -C.

P. -C.

Lee

R. -G.

Jiang

(2020). Deep-learning LSTM mechanism and wearable devices based virtual fitness-coach information system for barbell bench press. In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada, 2020, pages 3880-3885. https://doi.org/10.1109/SMC42975.2020.9283422

26.

Ihmig

F. R.

Gogeascoechea

Neurohr-Parakenings

H., F.

Schäfer

S. K.

Lass-Hennemann

Michael

(2020). On-line anxiety level detection from biosignals: Machine learning based on a randomized controlled trial with spider-fearful individuals, PLoS ONE, 15(6), e0231517. https://doi.org/10.1371/journal.pone.0231517

27.

Islam

Gedeon

Hossain

M. Z.

Liu

Y. -H.

(2019). Measuring user responses to driving simulators: A galvanic skin response based study. In Proceedings of the 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), San Diego, CA, USA, 2019, pp. 33-337. https://doi.org/10.1109/AIVR46125.2019.00015.

28.

Jerčić

Sundstedt

(2019). Practicing emotion-regulation through biofeedback on the decision-making performance in the context of serious games: A systematic review. Entertainment Computing, 29, 75-86. https://doi.org/10.1016/j.entcom.2019.01.001

29.

Jones

Smith-MacDonald

Miguel-Cruz

Pike

van Gelderen

Lentz

Shiu

M. Y.

Tang

Sawalha

Greenshaw

Rhind

S. G.

Fang

Norbash

Jetly

Vermetten

Brémault-Phillips

(2020) Virtual reality-based treatment for military members and veterans with combat-related posttraumatic stress disorder: Protocol for a multimodular motion-assisted memory desensitization and reconsolidation randomized controlled trial, JMIR Research Protocols, 9(10), e20620. https://doi.org/10.2196/20620

30.

Kaur

Sun

Ziegelman

Sowers

Hernandez

M. E.

(2019). Using virtual reality to examine the neural and physiological anxiety-related responses to balance-demanding target-reaching leaning tasks. In Proceedings of the 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), Toronto, ON, Canada, 2019, pages 1-7. https://doi.org/10.1109/Humanoids43949.2019.9035020.

31.

Kim

Ahn

C. R.

(2021). Predicting workers’ inattentiveness to struck-by hazards by monitoring biosignals during a construction task: A virtual reality experiment. Advanced Engineering Informatics, 49, 101359. https://doi.org/10.1016/j.aei.2021.101359

32.

Kim

J. -Y.

Nam

Lee

S. -A.

(2019). Classification of rehabilitation participation in elderly in-patients with mild cognitive impairments utilizing physiological responses. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 2019, pages 5176-5179. https://doi.org/10.1109/EMBC.2019.8856370.

33.

Kim

Lee

Y. M.

(2020). Estimating VR sickness caused by camera shake in VR videography. In Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, 2020, pages 3433-3437. https://doi.org/10.1109/ICIP40778.2020.9190721.

34.

Liebers

Horn

Burschik Gruenefeld

Schneegass

(2021). Using gaze behavior and head orientation for implicit identification in virtual reality”. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (VRST '21), Association for Computing Machinery, New York, NY, USA, 2021, article 22, pages 1–9. https://doi.org/10.1145/3489849.3489880

35.

Leiker

Miller

Brewer

Nelson

Siow

Lohse

(2016). The relationship between engagement and neurophysiological measures of attention in motion-controlled video games: A randomized controlled trial,” JMIR Serious Games, 4(1), https://doi.org/10.2196/games.5460

36.

Luong

Martin

Raison

Argelaguet

Diverrez

J. -M.

Lécuyer

(2020). Towards real-time recognition of users mental workload using integrated physiological sensors into a VR HMD. In Proceedings of the 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Porto de Galinhas, Brazil, 2020, pages 425-437. https://doi.org/10.1109/ISMAR50242.2020.00068

37.

Marín-Morales

Higuera-Trujillo

J. L.

Guixeres

Llinares

Alcañiz

Valenza

(2021). Heart rate variability analysis for the assessment of immersive emotional arousal using virtual reality: Comparing real and virtual scenarios, PLoS ONE, 16(7), e0254098. https://doi.org/10.1371/journal.pone.0254098

38.

Martin

Mathieu

Pallamin

Ragot

Diverrez

J. -M.

(2020). Virtual reality sickness detection: an approach based on physiological signals and machine learning. In Proceedings of the 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Porto de Galinhas, Brazil, 2020, pages 387-399. https://doi.org/10.1109/ISMAR50242.2020.00065

39.

McDermott

E. J.

Raggam

Kirsch

Belardinelli

Ziemann

Zrenner

(2022). Artifacts in EEG-based BCI therapies: Friend or foe? Sensors, 22(1), 96. https://doi.org/10.3390/s22010096

40.

Meho

L. I.

Rogers

(2008). Citation counting, citation ranking, and h-index of human-computer interaction researchers: A comparison of Scopus and Web of Science. Journal of the American Society for Information Science and Technology, 59(11). https://doi.org/10.1002/asi.20874

41.

Migovich

Korman

Wade

Sarkar

(2021). Design and validation of a stress detection model for use with a VR based interview simulator for autistic young adults. In Proceedings of Universal Access in Human-Computer Interaction. Design Methods and User Experience (HCII 2021). Lecture Notes in Computer Science, volume 12768. Springer, Cham, 2019, https://doi.org/10.1007/978-3-030-78092-0_40

42.

Moher

Shamseer

Clarke

Ghersi

Liberati

Petticrew

Shekelle

Stewart

L. A.

(2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1), 3. https://doi.org/10.1186/2046-4053-4-1

43.

Noghabaei

Han

Albert

(2021). Feasibility study to identify brain activity and eye-tracking features for assessing hazard recognition using consumer-grade wearables in an immersive virtual environment. Journal of Construction Engineering and Management, 147(9), 35. https://doi.org/10.1061/(ASCE)CO.1943-7862.000213

44.

Kim

D. K.

(2021). Machine-deep-ensemble learning model for classifying cybersickness caused by virtual reality immersion, Cyberpsychology, Behavior, and Social Networking, 24(11), 729-736, 2021. https://doi.org/10.1089/cyber.2020.0613

45.

Parent

Peysakhovich

Mandrick

Tremblay

Causse

(2019). The diagnosticity of psychophysiological signatures: Can we disentangle mental workload from acute stress with ECG and fNIRS? International Journal of Psychophysiology, 146, 139-147. https://doi.org/10.1016/j.ijpsycho.2019.09.005

46.

Park

Seok

H. S.

Kim

S. S.

Shin

and H.

(2021). Photoplethysmogram analysis and applications: An integrative review. Frontiers in Physiology, 11(1), 72. https://doi.org/10.3389/fphys.2021.808451

47.

Parnandi

Gutierrez-Osuna

(2017). Visual biofeedback and game adaptation in relaxation skill transfer. IEEE Transactions on Affective Computing, 10 (2), 276-289 https://doi.org/10.1109/TAFFC.2017.2705088

48.

Porssut

Hou

Blanke

Herbelin

Boulic

(2022). Adapting virtual embodiment through reinforcement learning. IEEE Transactions on Visualization and Computer Graphics, 28(9), 3193-3205. https://doi.org/10.1109/TVCG.2021.3057797

49.

Recenti

Ricciardi

Aubonnet

Picone

Jacob

Svansson

H. Á. R.

Agnarsdóttir

Karlsson

G. H.

Baeringsdóttir

Petersen

Gargiulo

(2021). Toward predicting motion sickness using virtual reality and a moving platform assessing brain, muscles, and heart signals. Frontiers in Bioengineering and Biotechnology, 9(2), 27. https://doi.org/10.3389/fbioe.2021.635661

50.

Rodrigues

M. J.

Postolache

Cercas

(2021). Autonomic nervous system assessment based on HRV analysis during virtual reality serious games. In Proceedings of Computational Collective Intelligence (ICCCI 2021), Lecture Notes in Computer Science, volume 12876, Springer, Cham, 2021. https://doi.org/10.1007/978-3-030-88081-1_57

51.

Saganowski

(2022). Bringing emotion recognition out of the lab into real life: Recent advances in sensors and machine learning. Electronics, 11(3), 28. https://doi.org/10.3390/electronics11030496

52.

Sakib

M. N.

Chaspari

Behzadan

A. H.

(2020). Physiological data models to understand the effectiveness of drone operation training in immersive virtual reality, Journal of Computing in Civil Engineering, 35(1), 25. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000941

53.

Sakib

M. N.

Chaspari

Behzadan

A. H.

(2022). A feedforward neural network for drone accident prediction from physiological signals. Smart and Sustainable Built Environment, 11(4), 1017-1041. https://doi.org/10.1108/SASBE-12-2020-0181

54.

Sayis

Ramirez

Pares

(2022). Mixed reality or LEGO game play? Fostering social interaction in children with autism, Virtual Reality, 26, 771–787. https://doi.org/10.1007/s10055-021-00580-9

55.

Seyderhelm

J. A.

Blackmore

K. L.

Nesbitt

(2019). Towards cognitive adaptive serious games: A conceptual framework. In Proceedings of Entertainment Computing and Serious Games (ICEC-JCSG 2019). Lecture Notes in Computer Science, volume 11863. Springer, Cham, 2019. https://doi.org/10.1007/978-3-030-34644-7_27

56.

Sharma

Lee-Cultura

Giannakos

(2022). Keep calm and do not carry-forward: Toward sensor-data driven AI agent to enhance human learning,” Frontiers in Artificial Intelligence, 4. https://doi.org/10.3389/frai.2021.713176

57.

Shi

Zhu

Mehta

R. K.

(2020). A neurophysiological approach to assess training outcome under stress: A virtual reality experiment of industrial shutdown maintenance using functional near-infrared spectroscopy (fNIRS). Advanced Engineering Informatics, 46, 101153. https://doi.org/10.1016/j.aei.2020.101153

58.

Stangl

Maoz

S. L.

Suthana

(2023). Mobile cognition: imaging the human brain in the ‘real world’, Nature Reviews Neuroscience, 24, 347–362. https://doi.org/10.1038/s41583-023-00692-y

59.

Stranick

Lopez

(2022). Adaptive virtual reality exergame: Promoting physical activity among workers. Journal of Computing and Information Science in Engineering, 22(3), 031002. https://doi.org/10.1115/1.4053002

60.

Suhaimi

N. S.

Mountstephens

Teo

(2020). Parameter tuning for enhancing inter-subject emotion classification in four classes for VR-EEG predictive analytics. International Journal of Advanced Science and Technology, 29(6s), 1483–1491. https://sersc.org/journals/index.php/IJAST/article/view/9288

61.

Suhaimi

N. S.

Mountstephens

Teo

(2021). Class-based analysis of Russell's four-quadrant emotion prediction in virtual reality using multi-layer feedforward ANNs. In Proceedings of the 10th International Conference on Software and Computer Applications (ICSCA '21), Association for Computing Machinery, New York, NY, USA, 2021, pages 155–161. https://doi.org/10.1145/3457784.3457809

62.

Suhaimi

N. S.

Mountstephens

Teo

(2022). A dataset for emotion recognition using virtual reality and EEG (DER-VREEG): Emotional state classification using low-cost wearable VR-EEG headsets. Big Data and Cognitive Computing, 6(1). https://doi.org/10.3390/bdcc6010016

63.

Suhaimi

N. S.

Teo

Mountstephens

(2017). Emotional state classification in virtual reality using wearable electroencephalography. In Proceedings of the International Conference on Applied Electronic and Engineering 2017 (ICAEE2017), Kuching, Sarawak, Malaysia, IOP Conference Series: Materials Science and Engineering, 2017, volume 341. https://doi.org/10.1088/1757-899X/341/1/012016

64.

Tabbaa

Searle

Bafti

S. M.

Hossain

M. M.

Intarasisrisawat

Glancy

Ang

C. S.

(2022). VREED: Virtual reality emotion recognition dataset using eye tracking & physiological measures. In Proceedings of ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, volume 5, number 4, article 178. https://doi.org/10.1145/3495002

65.

Tsai

M-C.

Chung

C-R.

Chen

C-C.

Chen

J-

Yeh

S-C.

Lin

C-H.

Chen

Y-J.

Tsai

M-C.

Wang

Y-L

Lin

C-J.

E. H-K.

(2021). An intelligent virtual-reality system with multi-model sensing for cue-elicited craving in patients with methamphetamine use disorder, IEEE Transactions on Biomedical Engineering, 68(7), 2270-2280. https://doi.org/10.1109/TBME.2021.3058805

66.

van den Broek

E. L.

Westerink

J. H. D. M.

(2012). Biofeedback for stress reduction: Towards a bright future for a revitalized field. In Proceedings of HEALTHINF 2012: International Conference on Health Informatics, Vilamoura, Algarve, Portugal, 2012, pages 499-504. https://resolver.tudelft.nl/uuid:28270e3f-0a88-4bf2-8102-c198e4af63dc

67.

Wilson

J. C.

Nair

Scielzo

Larson

E. C.

(2021). Objective measures of cognitive load using deep multi-modal learning: A use-case in aviation, In Proceedings of ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2021, volume 5, number 1, article 40. https://doi.org/10.1145/3448111

68.

Zhang

Wade

Bian

Fan

Swanson

Weitlauf

Warren

Sarkar

(2017). Cognitive load measurement in a virtual reality-based driving system for autism intervention IEEE Transactions on Affective Computing, 8(2), 176-189. https://doi.org/10.1109/TAFFC.2016.2582490

Integrating Biofeedback and Artificial Intelligence into eXtended Reality Training Scenarios: A Systematic Literature Review

Abstract

Background

Method

Results

Conclusion

Keywords

Introduction

Method

Biosignal Results

Biosignal Collection Summary and Primary Measurement Categories

Technologies and Equipment Used for Biosignal Collection

AI/ML Results

AI/ML Approach

AI/ML Uses and Definitions of Approaches

Discussion

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

Author Biographies

References