Sage Journals: Discover world-class research

Abstract

This work develops interpretable models to predict near-real-time cognitive workload (CWL) in teams operating in complex environments. Existing approaches using neurological sensors are impractical for field use. Our approach integrates multimodal data from non-invasive behavioral and physiological sensors to robustly detect CWL changes. We apply multidimensional recurrence quantification analysis (MdRQA) with a novel pattern analysis extension to identify recurring multimodal signatures indicative of different CWL states. A multiparty dataset with fNIRS, behavioral, and physiological measures from teams performing a gamified search and rescue mission and individual resting state tasks were used. The findings indicate that the multimodal patterns derived from non-invasive measures were significantly associated with a neurological measure of CWL within 10s time slices. Moreover, the multimodal patterns were predictive of individual and team performance over-and-above the neurological measure of CWL. This can enable timely interventions by intelligent systems to optimally manage team CWL and enhance human-machine teaming in demanding environments.

Keywords

cognitive workload interpretable modeling recurrence quantification analysis sensors

Introduction

Cognitive workload (CWL) is widely acknowledged as a key factor that affects human performance, where human error is especially likely when CWL is relatively high (overload) or, conversely, low (underload). CWL change may occur due to influences like time constraints, surroundings, expertise level, and task load intensity itself, with the overarching result being that more demanding tasks induce greater workload (Charles & Nixon, 2019). Task overload can result in rushed decisions and heightened stress (Wickens, 1979), while task underload can cause mind-wandering and complacency (Zhang & Kumada, 2017).

Because it is crucial to manage CWL in important situations, from everyday driving to air traffic control (Flin et al., 2016), research has examined how unobtrusive measures, such as sensor technologies, can be used to detect real-time changes in workload. Moreover, as technologies to augment cognition (Fiore & Wiltshire, 2016) become more common in educational settings (e.g., intelligent tutoring), or control rooms (e.g., information displays), there is continued interest in monitoring CWL in real time. Sensor-based approaches have been explored for real-time detection of CWL. For example, functional near-infrared spectroscopy (fNIRS) has emerged as a promising sensor-based approach for real-time detection of cognitive workload (CWL). Studies have explored its application in various domains, including aviation (Çakır et al., 2016) and medical training (Bracken et al., 2020). Physiological data can also assess affective CWL components in real-time, a viable alternative or supplement to task performance measures (Ikehara & Crosby, 2005). However, no single measure is optimal across all contexts (Hankins & Wilson, 1998; Miyake et al., 2009; Wilson, 1993). Although multiple measures detect CWL changes, there is a need to move beyond neurological sensors (e.g., EEG and fNIRS) that are often impractical to use in ecologically valid settings. Instead, effective real-time CWL sensing is needed for high-stakes environments (e.g., manned-unmanned air vehicles), using embedded or wearable sensors capable of collecting physiological data.

Across contexts, an invariant challenge remains the development of more sophisticated models capable of explaining and predicting variations in workload. Further, these require conceptual and analytic techniques that can integrate multimodal data. We address this gap by advancing interpretable models that combine physiological and behavioral signals to determine signatures of team workload states in complex settings. Specifically, we utilize fNIRS data as a neurological “ground truth” indicator of CWL. This is based on prior research showing fNIRS data can provide objective measures of CWL (Aghajani et al., 2017; Cao et al., 2022; Causse et al., 2017; Coffey et al., 2012; Liu et al., 2017). We use multidimensional recurrence quantification analysis (MdRQA; e.g., Amon et al., 2019; Wallot et al., 2016) with a novel pattern analysis extension, to develop interpretable models that enable deeper insights into the multimodal markers indicative of high and low CWL. Altogether, this work provides a novel approach for identifying cognitive states in complex settings. This can enable new technologies such as automated adaptation based on operator needs or human-machine teaming applications requiring real-time user state modeling. Towards this end, we address the following research questions:

RQ1: How do multimodal patterns from non-invasive sensors reflect a neurological sensing measure of CWL?

RQ2: Which multimodal patterns from non-invasive sensors most strongly correspond to high- versus low-CWL according to neurological sensing?

RQ3: Which multimodal patterns from non-invasive sensors most strongly correspond to task performance?

Method

Data Set

We utilized a publicly available dataset (ToMCAT; Pyarelal et al., 2023) capturing teams working together under time pressure to complete shared tasks. This dataset included 40 teams with three members in each team. The key experimental task was a Minecraft-based search and rescue mission involving time constraints and coordination between team members (see Huang et al., 2022 for full description). In addition, three baseline tasks were completed to facilitate data interpretation when compared to data gathered during the more demanding search and rescue scenario. We only utilized one of the baseline tasks “the resting state task” for our analysis.

This dataset simultaneously recorded multiple subjects’ neural activities (hyperscanning) during team interactions (Sinha et al., 2016). The dataset includes EEG and fNIRS as primary neural markers of CWL. As described, the use of fNIRS as a CWL baseline has been shown in other task settings, like the n-back working memory task (Aghajani et al., 2017; Liu et al., 2017) and flight simulator (Causse et al., 2017) and validated using other measures such as machine learning and statistical analysis. Additionally, behavioral and physiological measures (electrocardiography, gaze tracking, pupillometry) were collected over the full session. Previous studies showed physiological measures such as heart-rate-variability (HRV), eye tracking, and pupil dilation have been associated with cognitive load (Ikehara & Crosby, 2005; McDuff et al., 2014; Pomplun & Sunkara, n.d.; Wilbanks et al., 2021; Takahashi et al., 2000). Therefore, we utilized these measures (HRV, gaze tracking, and pupil dilation) to derive interpretable real-time indicators of high and low cognitive load based on non-invasive sensing.

Participants

There were 40 teams, each with three members. For the no-show of any participant, they used a confederate as a team member instead of a real participant (n = 18), leaving n = 102 valid participants. We also removed participants from the analysis due to technical issues (device, calibration; n = 22) and missing demographic info (n = 2). Finally, we had 76 real participant data and 33 team/group session data for the final analysis. Among these 76 participants, the mean age was 35.6 years (SD = 10.2), with 45% identifying as males (n = 33), 55.3% as females (n = 42), and 1.3% (n = 1) who preferred not to say. See Table 1 for aggregated demographic information of (a) race, (b) ethnicity, and (c) video gaming experience.

Table 1.

Demographics (a) Race, (b) Hispanic, (c) Video gaming experience.

Demographics	Count	Percentage
(a) Race
African American	5	6.578947
Asian American	9	11.842105
European American	16	21.052632
Non-Hispanic White	22	28.947368
Other	24	31.578947
(b) Hispanic
No	54	71.05263
Yes	22	28.94737
(c) Video gaming experience
Exp
Have played them fairly often	26	34.210526
Have played them occasionally	28	36.842105
Have played them regularly for years	20	26.315789
Never played them	2	2.631579

Data Pre-processing

For the analysis, we used data from the 17-min Minecraft mission Saturn A task (ToMCAT dataset, Pyarelal et al., 2023). We examined multimodal patterns derived from non-invasive sensors of eye movement, heart rate variability, and pupillary response. We investigated how the multimodal patterns corresponded to an fNIRS near-real-time measure of CWL, as well as team performance.

Heart Rate Variability (HRV)

Root mean square of successive differences (RMSSD) is a quantitative measure that reflects short-term heart rate variability (Ciccone et al., 2017; DeGiorgio et al., 2010). For RMSSD calculation we used a 10 s window. Notably, the HRV time window constrained the temporal resolution of our analyses. That is, sensors are commonly used for near-real-time detection, prediction, and intervention, motivating the need for short time windows (i.e., faster sampling rate). However, research suggests time windows shorter than 10s are precarious in their ability to reflect states of interest, like HRV, accurately (Baek et al., 2015). Thus, the HRV measure was calculated at a 10s sampling rate, in turn, guiding the sampling rate used for all sensor measures (also see Amon et al., 2019). We used the Neurokit python package to apply a band-pass filter to the signal (.67–45 Hz; Pyarelal et al., 2023) and then used the BIOSPPY python package for the RMSSD calculation.

Eye Movement

Cognitive workload affects smooth pursuit eye movements, with higher deviations in gaze points at higher workload levels (Kosch et al., 2018). The eye-tracking system used in this study was Pupil Core (Pupil Labs GmbH) with eye-tracking data sampled at 250 Hz (Pyarelal et al., 2023). We calculated the absolute difference between x and y coordinates from each time point and then identified the Euclidean distance between consecutive gaze points. Then we removed outliers ±4 standard deviation from the mean.

Pupillary Response

Increases in pupil size are associated with increased cognitive processing demands (Sibley et al., 2015). We calculated the percentage change in pupil size from pupil diameter. Afterward, we calculated the average diameter change during the rest state task and applied the average diameter change of the rest state task as the baseline diameter for the pupillary response calculation while in the Minecraft mission task.

Functional Near-infrared Spectroscopy (fNIRS)

Increase in HbO and decrease in HbR in the prefrontal cortex have been shown to indicate higher cognitive workload (Causse et al., 2017; Toichi et al., 2004). For analysis, fNIRS reliability was improved by averaging across a small number of channels overlying a cortical region of interest (Luke et al., 2021; Wiggins et al., 2016). In the ToMCAT dataset (Pyarelal et al., 2023), the optodes mainly cover the frontal lobe area. Therefore, we collected all the frontal lobe optics and prefrontal cortex as our region of interest. We then calculated the average of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) from all the channels from the above region of interest. Deoxygenated hemoglobin (HbR) performs better than HbO in CWL measure (Bracken et al., 2020; Hirshfield et al., 2023; Zhuang et al., 2021), hence we only presented results from HbR for our study. fNIRS was recorded using NIRSport2 (NIRx Medical Technology LLC), which records and streams fNIRS signals at 10.2 Hz (Pyarelal et al., 2023).

Analysis

MdRQA is an extension of recurrence quantification analysis in that it incorporates unsupervised learning to classify different time points as recurrent (repeating) or not. We completed MdRQA along with pattern analysis via the following three main steps:

(1) First, we created categorical versions of the HRV, eye movement, and pupillary response timeseries, where time points within each measure were split into high (H) or low (L) values, based on the median value per measure across participants. This initial step allowed us to derive easy-to-understand multimodal clusters (Necaise et al., 2021).

(2) Next, we followed the MdRQA procedure, comparing the similarity of every timepoint to every other timepoint within a time series and marking it as recurrent or non-recurrent based on the relative similarity of compared points (see Necaise et al., 2021 for parameter selection procedure). When performing this step with multiple time series per participant (HRV, eye movement, and pupillary response), all the time series measures must revisit a similar state for it to be marked as recurrent, even if the values of the different measures are dissimilar. For example, in this case, a recurrent pattern might include a minimal eye movement with a minimal pupillary response but a maximal HRV value, reflecting high-intensity focus. This step yields a binarized recurrence matrix indicating which time points are recurrent or not.

(3) Next, we projected the pattern assignments back onto the original recurrence plot to produce a color-coded plot distinguishing the different classes of recurrent states (Necaise et al., 2021). In this case, the color-coded points reflect recurrent patterns of an individual’s multichannel behavioral and physiological activity that occur over time during the collaborative task. Feature extraction of patterned recurrent states derived from non-invasive sensing was then examined in relation to the fNIRS HbR measure of CWL to determine those multimodal signatures most predictive of high and low CWL, as well as how the multimodal patterns predicted performance.

From the three sensing measures (HRV, eye movement, and pupillary response), we examined eight distinct patterns. For example, pattern H-L-H represents high HRV, low eye movement, and high pupillary response respectively. We coded non-recurrent points as N-R-P and used N-R-P pattern as the reference group for the mixed-effect regression. In this case, N-R-P points represents shifts to novel states (i.e., the time points being compared are unique from one another). Figure 1 shows one example of a multidimensional recurrence plot for the non-invasive sensing measures HRV, eye movement, and pupillary response.

Figure 1.

Example participant multidimensional recurrence matrix, with points representing 10s time windows. Via pattern analysis, recurrent values were color-coded to depict specific patterns that repeat, in this case, combinations of low (L) and high (H) levels of the non-invasive sensing measures, HRV, eye movement, and pupillary response. For this participant, we see a concentration of red points (H-H-H) in the beginning of task (bottom-left), shifting to yellow points (L-H-L) in the middle of the task, followed by blue points (H-H-L).

Results

Multimodal Patterns From Non-Invasive Sensing Reflect a Neurological Measure of CWL

First, we were interested in how the non-invasively sensed multimodal patterns are associated with a more traditional neurological measure of cognitive load like fNIRS-derived HbR (Causse et al., 2017; Toichi et al., 2004).

To address RQ1, we identified patterns from non-invasive sensor measures of HRV, eye movement, and pupillary response that are associated with changes in HbR. To examine the extent to which the multimodal patterns of the non-invasive sensors reflect HbR levels, we applied a mixed-effect regression model with HbR as an outcome variable, pattern type as a predictor, and participant ID and team ID as random intercepts. We also controlled for demographic variables of age, gender, and videogame experience, as well as number of timepoints available for analysis per participant. The results can be found in Table 2. The model shows a significant relationship between the non-invasive sensor patterns and frontal lobe de-oxygenation. Moreover, the multimodal patterns differed in the extent to which they mapped to HbR. Specific patterns (LLL, LLH, LHH, HLL, HLH, HHL, HHH) were significant predictors at the p < .05 level.

Table 2.

Mixed-effect regression analysis examining the association between the neurological measure of CWL (HbR) and multimodal patterns of HRV, eye movement, and pupillary response within 10s windows, controlling for demographics and time points available for analysis.

Predictor Variables		HBr
Predictors	Estimates	Std. error	ρ
(Intercept)	0.71	0.39	0.070
Pattern L-L-L	0.08	0.00	<0.001
Pattern L-L-H	−0.04	0.01	<0.001
Pattern L-H-L	0.01	0.01	0.290
Pattern L-H-H	−0.02	0.01	0.003
Pattern H-L-L	0.02	0.01	<0.001
Pattern H-L-H	−0.02	0.01	<0.001
Pattern H-H-L	0.20	0.01	<0.001
Pattern H-H-H	−0.27	0.01	<0.001
Age	−0.01	0.02	0.721
Gender	−0.18	0.12	0.127
Video gaming experience	0.01	0.07	0.919
Participant total time points	−0.00	0.00	0.056
Random effects
σ²		1.42
τ₀₀ _{participantID}		0.23
τ₀₀ _teamID		0.01
ICC		0.14
N _{participantID}		76
N _teamID		33
Observations		1103631
Marginal R² / Conditional R²		0.013 / 0.154

Note. Results are reported using non-recurrent points as the reference category. Numbers in bold indicate significance at p < .05.

Multimodal Patterns From Non-Invasive Sensing Distinguish Between High and Low CWL as Measured Neurologically

To address RQ2, we identified which patterns were related to high vs low CWL as a first step toward understanding how the patterns differently reflect CWL states. To do this, we calculated the mean HbR for each multimodal pattern and participant. These averages are shown in Table 3. For example, the pattern “H-H-L” reflects high HRV, high eye movement, and low pupillary response, respectively, and was significantly associated with the highest HbR levels. In this way, patterns differently map to the neurological measure of CWL.

Table 3.

Mean HbR per Pattern of Non-invasive Neurological Sensors.

Pattern	Mean HbR
L-L-L	0.113635204
L-L-H	0.018213241
L-H-L	0.005285887
L-H-H	−0.007294927
H-L-L	0.049233581
H-L-H	0.023333814
H-H-L	0.329171657
H-H-H	0.131390698
N-R-P	0.097962393

Multimodal CWL Patterns From Non-invasive Sensing Predict Task Performance

To address RQ3, we examined which multimodal patterns from non-invasive sensors most strongly predicted task performance. We performed two linear regression analyses with either team performance or individual performance (Mine craft mission score) as the outcome variables. Each participant’s multimodal pattern frequency was a predictor and we added demographics (age, gender, video gaming experience) as control variables. Results are shown in Table 4. The findings show that different multimodal patterns map onto team versus individual performance. In both cases, the multimodal pattern frequencies outperform HbR in predicting performance.

Table 4.

Comparison of Logistic Regression Analysis Between Team and Individual Performance Prediction Using Non-invasive Sensor Patterns Frequency and Demographics.

Predictors	Team score			Individual score
Predictors	Estimates	Std. error	p	Estimates	Std. error	p
(Intercept)	826.49	312.90	0.010	173.83	286.08	0.546
Pattern frequency HLL	61.27	35.85	0.092	23.63	32.77	0.474
Pattern frequency HLH	79.09	37.66	0.040	18.75	34.43	0.588
Pattern frequency HHL	42.42	25.48	0.101	−8.72	23.29	0.709
Pattern frequency HHH	94.69	36.80	0.013	−31.86	33.64	0.347
Pattern frequency NRP	85.08	65.29	0.197	33.30	59.69	0.579
Pattern frequency LLL	91.82	50.15	0.072	92.25	45.85	0.049
Pattern frequency LLH	50.40	29.55	0.093	50.63	27.02	0.066
Pattern frequencyLHH	92.76	47.31	0.054	−16.31	43.25	0.707
Pattern frequency LHL	65.94	29.82	0.031	2.15	27.26	0.937
Participant mean HbR	15.22	20.90	0.469	9.71	19.11	0.613
Age	1.28	20.38	0.950	−31.34	18.63	0.098
Gender	−36.90	41.41	0.376	−11.36	37.86	0.765
Video gaming experience	37.75	26.34	0.157	44.05	24.08	0.072
Participant total time points	−3.59	2.64	0.179	−0.85	2.42	0.727
Observations	76			76
R² / R² adjusted	0.238 / 0.063			0.448 / 0.321

Note. Numbers in bold indicate significance at p < .05.

Discussion

This research elucidates how multimodal physiological measures reflect CWL dynamics, providing a method for near-real-time interpretable modeling of CWL in high-stakes environments. Not only does our multimodal measure with non-invasive sensors correspond to a traditional neurological measure of CWL, but our multimodal measure is also more predictive of individual and team performance than the neurological measure HbR. Thus, we showcase the effectiveness of incorporating behavioral and physiological data in detecting CWL within complex sociotechnical environments. We anticipate this approach may prove to be more scalable compared to relying solely on neurological measures.

Further, our emphasis on interpretable models allows us to pinpoint specific multimodal behavioral and physiological patterns that can serve as early indicators of impending workload changes. Notably, we identified different multimodal patterns associated with high versus low CWL, as well as patterns that corresponded most strongly with performance. Broadly speaking, the findings indicate a need for more research that examines how specific multimodal patterns correspond to different types of CWL-relevant contexts, for example physical discomfort versus mental panic. In other words, the multimodal patterns may have the added benefit of differentiating between different types of CWL states, with the potential to advance deeper understandings of CWL.

It is important to note some limitations with our approach. In particular, the pupillary response is to some extent confounded by eye movement, which can evoke a light reflex response. Still, we include pupillary response as a common CWL indicator and complement to the other CWL measures. Consistent with the brief format of the paper, we leave it to future work to include a detailed examination of specific patterns.

In conclusion, by integrating behavioral and physiological dynamics through MdRQA along with pattern analysis, our work has the potential to advance both understanding and measurement of CWL, while pointing the way for practical applications in team-based, high-stakes environments. These insights can inform the design of intelligent systems capable of timely interventions to manage CWL, thereby enhancing human-machine teaming and overall performance in complex sociotechnical settings. As such, they offer complementary capabilities to those being developed in human-AI teaming research, such as artificial social intelligence as a method for augmenting team cognitive processes (e.g., Bendel et al., 2024; Williams et al., 2022).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Writing of this paper was partially supported by funding from Lockheed Martin Corporation contract LMC#MRA20-001-RPP006, Cognitive Offloading for Manned-Unmanned Teaming (MUM-T) awarded to the second and third author, and by the Defense Advanced Research Projects Agency (DARPA), under Contract No. W911NF-20-1-0008, Artificial Social Intelligence to Support Macrocognition in Teams, awarded to the third author. Any opinions, findings, and conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of these organizations or the authors’ affiliated University. We also thank Dr. Adarsh Pyarelal for his valuable insights into the comprehensive ToMCAT dataset used in this paper. Dr. Mary Jean Amon is now with Indiana University.

ORCID iDs

Nurun Naher

Stephen M. Fiore

References

Aghajani

Garbey

Omurtag

(2017). Measuring mental workload with EEG+fNIRS. Frontiers in Human Neuroscience, 11, 359. https://doi.org/10.3389/fnhum.2017.00359

Amon

M. J.

Vrzakova

D’Mello

S. K.

(2019). Beyond dyadic coordination: multimodal behavioral irregularity in triads predicts facets of collaborative problem solving. Cognitive Science, 43(10), e12787. https://doi.org/10.1111/cogs.12787

Baek

H. J.

Cho

C.-H.

Cho

Woo

J.-M.

(2015). Reliability of ultra-short-term analysis as a surrogate of standard 5-min analysis of heart rate variability. Telemedicine and E-Health, 21(5), 404–414.

Bendell

Williams

Fiore

S. M.

Jentsch

(2024). Individual and team profiling to support theory of mind in artificial social intelligence. Scientific Reports, 14(1), 12635.

Bracken

Houssan

Broach

Milsten

Leather

Tobyne

Winder

Farry

(2020). Evaluation of a new functional near infrared spectroscopy (fNIRS) sensor, the fNIRS explorer^TM, and software to assess cognitive workload during ecologically valid tasks: Proceedings of the 13th international joint conference on biomedical engineering systems and technologies (pp. 179–186). https://doi.org/10.5220/0008902701790186

Çakır

M. P.

Vural

Koç

S. Ö.

Toktaş

(2016). Real-time monitoring of cognitive workload of airline pilots in a flight simulator with fNIR optical brain imaging technology. In Schmorrow

D. D.

Fidopiastis

C. M.

(Eds.), Foundations of augmented cognition: Neuroergonomics and operational neuroscience (Vol. 9743, pp. 147–158). Springer International Publishing.

Cao

Garro

E. M.

Zhao

(2022). EEG/fNIRS based workload classification using functional brain connectivity and machine learning. Sensors, 22(19), 7623. https://doi.org/10.3390/s22197623

Causse

Chua

Peysakhovich

Del Campo

Matton

(2017). Mental workload and neural efficiency quantified in the prefrontal cortex using fNIRS. Scientific Reports, 7(1), 5222. https://doi.org/10.1038/s41598-017-05378-x

Charles

R. L.

Nixon

(2019). Measuring mental workload using physiological measures: A systematic review. Applied Ergonomics, 74, 221–232.

10.

Ciccone

A. B.

Siedlik

J. A.

Wecht

J. M.

Deckert

J. A.

Nguyen

N. D.

Weir

J. P.

(2017). Reminder: RMSSD and SD1 are identical heart rate variability metrics. Muscle & Nerve, 56(4), 674–678.

11.

Coffey

E. B. J.

Brouwer

A.-M.

Van Erp

J. B. F.

(2012). Measuring workload using a combination of electroencephalography and near infrared spectroscopy. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 1822–1826. https://doi.org/10.1177/1071181312561367

12.

DeGiorgio

C. M.

Miller

Meymandi

Chin

Epps

Gordon

Gornbein

Harper

R. M.

(2010). RMSSD, a measure of vagus-mediated heart rate variability, is associated with risk factors for SUDEP: The SUDEP-7 Inventory. Epilepsy & Behavior, 19(1), 78–81.

13.

Fiore

S. M.

Wiltshire

T. J.

(2016). Technology as teammate: Examining the role of external cognition in support of team cognitive processes. Frontiers in Psychology, 7, 1531. https://doi.org/10.3389/fpsyg.2016.01531

14.

Flin

Salas

Straub

Martin

(2016). Decision-making under stress: Emerging themes and applications. Routledge.

15.

Hankins

T. C.

Wilson

G. F.

(1998). A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight. Aviation, Space, and Environmental Medicine, 69(4), 360–367.

16.

Hirshfield

L. M.

Wickens

Doherty

Spencer

Williams

Hayne

(2023). Toward workload-based adaptive automation: The utility of fNIRS for measuring load in multiple resources in the brain. International Journal of Human–Computer Interaction, 1–27.

17.

Huang

Freeman

Cooke

Colonna-Romano

Wood

Buchanan

Caufman

S. J.

(2022). Exercises for artificial social intelligence in minecraft search and rescue for teams. OSF. https://doi.org/10.17605/OSF.IO/JWYVF

18.

Ikehara

C. S.

Crosby

M. E.

(2005). Assessing cognitive load with physiological sensors. Proceedings of the 38th annual hawaii international conference on system sciences (pp. 295a–295a). 38th Annual Hawaii International Conference on System Sciences. https://doi.org/10.1109/HICSS.2005.103

19.

Kosch

Hassib

Woźniak

P. W.

Buschek

Alt

(2018). Your eyes tell: Leveraging smooth pursuit for assessing cognitive workload. Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1–13). https://doi.org/10.1145/3173574.3174010

20.

Liu

Ayaz

Shewokis

P. A.

(2017). Multisubject “learning” for mental workload classification using concurrent EEG, fNIRS, and physiological measures. Frontiers in Human Neuroscience, 11, 389. https://doi.org/10.3389/fnhum.2017.00389

21.

Luke

Larson

Shader

M. J.

Innes-Brown

Van Yper

Lee

A. K. C.

Sowman

P. F.

McAlpine

(2021). Analysis methods for measuring passive auditory fNIRS responses generated by a block-design paradigm. Neurophotonics, 8(02), 025008–025008. https://doi.org/10.1117/1.NPh.8.2.025008

22.

McDuff

Gontarek

Picard

(2014). Remote measurement of cognitive stress via heart rate variability. 2014 36th annual international conference of the IEEE engineering in medicine and biology society (pp. 2957–2960). https://doi.org/10.1109/EMBC.2014.6944243

23.

Miyake

Yamada

Shoji

Takae

Kuge

Yamamura

(2009). Physiological responses to workload change. A test/retest examination. Applied Ergonomics, 40(6), 987–996. https://doi.org/10.1016/j.apergo.2009.02.005

24.

Necaise

Williams

Vrzakova

Amon

M. J.

(2021). Regularity versus novelty of users’ multimodal comment patterns and dynamics as markers of social media radicalization. Proceedings of the 32st ACM conference on hypertext and social media (pp. 237–243). https://doi.org/10.1145/3465336.3475095

25.

Pomplun

Sunkara

(n.d.). Pupil dilation as an indicator of cognitive workload in human-computer interaction. CRC Press.

26.

Pyarelal

Duong

Shibu

Soares

Boyd

Khosla

Pfeifer

V. A.

Zhang

Andrews

Champlin

Raymond

Krishnaswamy

Morrison

Butler

Barnard

(2023). The ToMCAT dataset. Advances in Neural Information Processing Systems, 36, 40872–40923. https://proceedings.neurips.cc/paper_files/paper/2023/hash/803d8d4b4a549d0d062fc704f8659ce3-Abstract-Datasets_and_Benchmarks.html

27.

Sibley

C. M.

Coyne

J. T.

Doddi

Jasper

(2015, May 7). Pupillary response as an indicator of processing demands within a supervisory control simulation environment. https://www.semanticscholar.org/paper/Pupillary-Response-as-an-Indicator-of-Processing-a-Sibley-Coyne/bb87a5f0e394ddae1f46bdfd518801241244def6

28.

Sinha

Maszczyk

Zhang Wanxuan Tan

Dauwels

(2016). EEG hyperscanning study of inter-brain synchrony during cooperative and competitive interaction. 2016 IEEE international conference on systems, man, and cybernetics (SMC) (pp. 004813–004818). https://doi.org/10.1109/SMC.2016.7844990

29.

Takahashi

Nakayama

Shimizu

(2000). The response of eye-movement and pupil size to audio instruction while viewing a moving target. Proceedings of the Symposium on Eye Tracking Research & Applications - ETRA ’00, 131–138. https://doi.org/10.1145/355017.35504

30.

Toichi

Findling

R. L.

Kubota

Calabrese

J. R.

Wiznitzer

McNamara

N. K.

Yamamoto

(2004). Hemodynamic differences in the activation of the prefrontal cortex: Attention vs. higher cognitive processing. Neuropsychologia, 42(5), 698–706. https://doi.org/10.1016/j.neuropsychologia.2003.08.012

31.

Wallot

Roepstorff

Mønster

(2016). Multidimensional recurrence quantification analysis (MdRQA) for the analysis of multidimensional time-series: A software implementation in MATLAB and its application to group-level data in joint action. Frontiers in Psychology, 7, 1835. https://doi.org/10.3389/fpsyg.2016.01835

32.

Wickens

C. D.

(1979). Measures of workload, stress and secondary tasks. Moray

(Ed.) (pp. 79–99). Springer US. https://doi.org/10.1007/978-1-4757-0884-4_6

33.

Wiggins

I. M.

Anderson

C. A.

Kitterick

P. T.

Hartley

D. E. H.

(2016). Speech-evoked activation in adult temporal cortex measured using functional near-infrared spectroscopy (fNIRS): Are the measurements reliable? Hearing Research, 339, 142–154. https://doi.org/10.1016/j.heares.2016.07.007

34.

Wilbanks

B. A.

Aroke

Dudding

K. M.

(2021). Using eye tracking for measuring cognitive workload during clinical simulations: Literature review and synthesis. CIN: Computers, Informatics, Nursing, 39(9), 499–507. https://doi.org/10.1097/CIN.0000000000000704

35.

Williams

Fiore

S. M.

Jentsch

(2022). Supporting artificial social intelligence with theory of mind. Frontiers in Artificial Intelligence, 5, 750763.

36.

Wilson

G. F.

(1993). Air-to-ground training missions: A psychophysiological workload analysis. Ergonomics, 36(9), 1071–1087. https://doi.org/10.1080/00140139308967979

37.

Zhang

Kumada

(2017). Relationship between workload and mind-wandering in simulated driving. PLOS ONE, 12(5), e0176962. https://doi.org/10.1371/journal.pone.0176962

38.

Zhuang

Meidenbauer

K. L.

Kardan

Stier

A. J.

Choe

K. W.

Cardenas-Iniguez

Huppert

T. J.

Berman

M. G.

(2021). Scale Invariance in fNIRS as a Measurement of Cognitive Load. Cortex, 154, 62–76. https://doi.org/10.1101/2021.08.31.458427

Interpretable Models for Near-real-time Prediction of Team Cognitive Workload in Complex Sociotechnical Environments Using Behavioral and Physiological Data

Abstract

Keywords

Introduction

Method

Data Set

Participants

Data Pre-processing

Heart Rate Variability (HRV)

Eye Movement

Pupillary Response

Functional Near-infrared Spectroscopy (fNIRS)

Analysis

Results

Multimodal Patterns From Non-Invasive Sensing Reflect a Neurological Measure of CWL

Multimodal Patterns From Non-Invasive Sensing Distinguish Between High and Low CWL as Measured Neurologically

Multimodal CWL Patterns From Non-invasive Sensing Predict Task Performance

Discussion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

References