Abstract
Objective
We demonstrate and discuss the use of mobile electroencephalogram (EEG) for neuroergonomics. Both technical state of the art as well as measures and cognitive concepts are systematically addressed.
Background
Modern work is increasingly characterized by information processing. Therefore, the examination of mental states, mental load, or cognitive processing during work is becoming increasingly important for ergonomics.
Results
Mobile EEG allows to measure mental states and processes under real live conditions. It can be used for various research questions in cognitive neuroergonomics. Besides measures in the frequency domain that have a long tradition in the investigation of mental fatigue, task load, and task engagement, new approaches—like blink-evoked potentials—render event-related analyses of the EEG possible also during unrestricted behavior.
Conclusion
Mobile EEG has become a valuable tool for evaluating mental states and mental processes on a highly objective level during work. The main advantage of this technique is that working environments don’t have to be changed while systematically measuring brain functions at work. Moreover, the workflow is unaffected by such neuroergonomic approaches.
Introduction
The guiding principle of neuroergonomics is that understanding how the brain carries out the complex tasks of everyday life—and not just the simple, artificial tasks of the research laboratory—can provide important benefits for both ergonomics research and practice. (Parasuraman, 2003)
Information processing is a central aspect of modern work life. In piloting, monitoring or any other task that requires the intake and processing of (as well as adequate responses to) information, fluctuations in task engagement or mental resources may lead to fatal accidents or mental strain that might end up in stress-related diseases. Thus, the evaluation or even the continuous monitoring of mental states and/or cognitive processing may help to improve work safety and well-being (Parasuraman, 2011). The measurement of neurophysiological parameters in work environments can help to understand the neural basis of cognitive processing during common activities and actions (Hancock, 2019; McKeown, 2014; Mehta & Parasuraman, 2013; Rahman et al., 2019). Furthermore, neurophysiological measures do not only unveil cognitive aspects (i.e., mental load) of modern work. They bear the potential to improve safety and efficiency of work environments and thus to raise motivation and productivity (Cinel et al., 2019; Giraudet et al., 2015).
So far, the main approach of neuroergonomic workplace analysis has been to (1) derive cognitive construct entities of interest (e.g., motivation or mental load) and (2) transfer these constructs to controlled settings in the laboratory where neurophysiological measurements can be conducted in a highly controlled, artificial environment (Arnau et al., 2019; Kramer et al., 1985). In other studies, work-related situations or tasks were partially reconstructed to investigate, for example, the impact of light upon fatigue (Baek & Min, 2015), adaptive automation in flight control (Aricò et al., 2016; Freeman et al., 1999), or high mental demands that should evoke a stress response (Ahn et al., 2019). These work-related studies varied in the realism of the settings, even incorporating real-life experimentation (Dehais et al., 2018; Dehais, Somon, et al., 2020; McKendrick et al., 2016; Shamay-Tsoory & Mendelsohn, 2019). Over the years, the impact of the complex interaction of countless (and so far even hardly identified) mental factors upon information processing within real-life environments has been addressed in neuroscience-related studies (Gramann, Ferris, et al., 2014), especially in the field of automobile driving research (c.f. Borghini et al., 2014; Lohani et al., 2019; Perrier et al., 2016). Thereby, the application of neurophysiological methods in truly natural working environments—such as the cockpit of an airplane (Dehais et al., 2018; Gateau et al., 2018)—intends to generate a valid description of mental processing during work. No situational characteristics should be changed for the worker—neither the workflow nor the self-perception or social interactions that are inherent to many working environments. Therefore, the measurement equipment used must be unobtrusive and should not interfere with any of the wearer’s actions.
What method is best suited to map these requirements? First of all, it should reflect cognitive processing and mental states reliably, which excludes any methods that uses peripheral parameters. An important tool to investigate cognitive processes noninvasively and with high temporal resolution is the electroencephalogram (EEG). This measurement technique records voltage changes over the scalp—generated by cortical and noncortical sources—using small electrodes. Despite its sensitivity to movement and other workplace-related sources of interference (e.g., electromagnetic intrusions from electronics equipment, engines, and generators), it appears to also fulfill other requirements for a suitable system. Since evaluating the time course of mental states is one core objective of this approach, the measures used should remain sensitive over time in order to reflect, for example, increasing fatigue of the worker. Also, the equipment used should be able to record a participant’s mental activity in a measurement lasting for hours and should not become uncomfortable to wear in this time. Remote access to the data might enable its use for controlling or even adapting work situations. Besides measurement reliability, the measurement validity of the mental aspects of work needs to be ensured as well.
The EEG’s selection as the measurement technique of choice is based on several objective reasons. First, it promises high mobility due to its sensors’ properties: electrodes are lightweight, small, and can be positioned to one’s liking without interfering with other sensors. Also, EEG systems are quite inexpensive in comparison to other types of neurocognitive measurement devices (e.g., functional Near InfraRed Spectroscopy [fNIRS]), especially nowadays with an increasing number of devices marketed to end-consumers. The last advantage lies within the EEG’s temporal resolution with up to more than 1 kHz, which boosts its practicability for many use-cases that need to be analyzed in the time domain (Makeig et al., 2009; for an overview of the many available devices, see Mehta & Parasuraman, 2013).
The present paper gives an overview of the development and current trends in the use of mobile systems for the acquisition of EEG parameters in situations that are close to real-life behavior. Requirements, limitations, and possible approaches to overcome limitations are addressed and current EEG measurement systems are presented. Common EEG measures in the frequency and time domains are discussed in detail. Finally, conceptual questions and future goals and challenges are addressed.
Going Mobile
Rationale, Prerequisites, and Restrictions
Cognitive neuroergonomics intends to investigate cognitive states and their impact on information processing in the workplace based on neurophysiological data. While it is also possible to infer on cognitive states by assessing peripheral psychophysiological measures (e.g., heart rate variability or electrodermal activity), cognitive states and mental processing can be addressed most accurately with the use of high temporal resolution measurements by means of EEG (c.f. Hogervorst et al., 2014; St. John et al., 2004). Modulations of information processing, for example, due to cognitive load, can then be described in a temporally specific way and with a high signal sensitivity. However, these measures have seldomly been applied in uncontrolled environments—for example, real-life or high-fidelity simulated driving or aircraft maneuvering (for a review, see Borghini et al., 2014)—that provide reliable information about both the environment and the action of the participant (Gramann et al., 2017). Thus, most neurophysiological research was conducted in sparsely lit chambers where the participant sat inertly in a chair and responded mostly with button presses. The tasks used were simplified experimental approaches toward single cognitive mechanisms. To circumvent this artificiality, realistic work settings were either combined with experimental tasks (Mijović et al., 2016) or behavioral restrictions (i.e., participants were seated on a sofa and asked to minimize body motion to avoid movement artifacts), in order to guarantee proper processing of incoming information and high-quality EEG measurements (Chang & Chen, 2005). These studies were the first steps toward more ecologically valid settings (Ladouce et al., 2016). In later studies, the simulation of workplaces came closer to real-world situations (Funke et al., 2017; Matthews et al., 2015, Mijović et al., 2016; Wascher, Heppner, et al., 2014; Wascher, Heppner, Kobald, et al., 2015), while maintaining experimental control, for example, in virtual reality settings (Banaei et al., 2017) or high-fidelity simulations of military applications (Wilson & Russell, 2007), driving (Getzmann et al., 2018), or piloting (Wilson & Hankins, 1994).
Neurocognitive studies in workplace simulations that come close to real environments are still rare in spite of their relevance. Considering the psychological strain imposed by certain tasks and environments on the worker without relying on overt behavior (e.g., units produced per hour, deviations from the optimal route), well-being and safety could be enhanced significantly (Parasuraman, 2011). The use-cases are manifold, as a sociotechnical system could be extended by a passive brain–computer interface that regulates the distribution of demand adaptively between the operator and the machine depending on the individual’s attentional state. Another vision for future applications of neuroergonomic measures could lead in the direction of neurophysiological workload quantification using a mobile device and a lightweight set of electrodes. Still, in order to be able to quantify cognitive constructs, one needs to understand how brain correlates are related to cognitive states.
Besides the experimental restrictions outlined above, one main restriction in the past were the problems inherent to the measurement of EEG during ongoing bodily motion. When workplaces are considered that do not only comprise seated task execution like driving or air traffic management, both the restricted availability of truly mobile recording equipment and the detrimental influence of motion artifacts (Gale et al., 2007) rendered reliable measurements of cognitive processing almost impossible. However, recent technical advances, in particular the miniaturization of EEG amplifiers, created new possibilities and therefore enabled neuroscientific research to enter the real workplace without changing the naturality of the environment. This includes that workers are restricted neither by the measurement equipment in their work nor with respect to their ability to move or during social interactions.
State of the Art
Research interest in mobile EEG has increased enormously over the last decade. A systematic search in Web of Science (August 2019) revealed 305 studies employing mobile EEG (Search topics: “mobile EEG,” “portable EEG,” “wearable EEG,” “wireless EEG”) starting from 1998 (duplicates were discarded). However, it took until 2014 before a substantial number of studies (>10) was published per year. About 18% of these studies reported the application of mobile EEG in clinical contexts, applying this method to patient monitoring or diagnostic procedures (Dash et al., 2012). Ten studies (3.3%) reported on sleep staging. As would be expected in such a new field, a substantial part of the publications dealt with hardware development and evaluation (17%). Another 17% of studies report the evaluation of methods, in particular regarding the removal of EEG artifacts that arise with mobile application (Blum et al., 2019; Snyder et al., 2015).
The amount of neuroergonomic literature (i.e., studies with a clear focus on workplace-related issues) that incorporates mobile EEG remained relatively sparse with 28 published studies until late 2019. As expected, (mental) fatigue and drowsiness constituted a high fraction of these studies, in particular for different aspects of aircraft piloting (Guo et al., 2018; Hankins & Wilson, 1998; Lin et al., 2014; Rohit et al., 2017, Wilson, 2002. Zhang et al., 2017; Zhou et al., 2018), manual assembly processes (Xiao et al., 2018), or in the context of a logistics workplace (Wascher, Heppner, et al., 2014; Wascher, Heppner, Kobald, et al., 2015). These studies included user state examinations, the development of countermeasures to critical aspects of safety in the workplace (Zhou et al., 2018), or a brain–computer interface (BCI)-based control of driving speed (Zhang et al., 2016).
Mental workload is another core construct of cognitive ergonomics. Studies investigating mental workload applied well-established cognitive paradigms in order to reliably manipulate the construct of task load while using mobile EEG, for example during walking movements, using machine-learning algorithms and phase-locked information (Wang et al., 2016; Yokota et al., 2017). As an approach toward studying even more natural situations, at least one study investigated acute stress in construction workers with the help of features in the time and frequency domain (Jebelli et al., 2018), while another study looked more generally into the emotional state in construction workers as an important factor affecting safety in the workplace using feature extraction and classification techniques (Hwang et al., 2018).
Until now, only a few cognitive aspects of work have been investigated with the help of mobile EEG, despite its large potential for doing so. Liu et al. (2013) investigated to what degree alertness could be modulated by music. More directly geared at basic cognitive mechanisms, some studies have investigated the allocation of attention in the work process (Liu et al., 2013; Mijović et al., 2017; Wang et al., 2017). Beside these basic cognitive aspects of work, another group of studies addressed the question of user experience in cognitive neuroergonomics. They deal with economic decisions or more generally with consumer behavior (Khushaba et al., 2013; Muñoz et al., 2019; Roberts et al., 2018), user interaction (Sargent et al., 2020), or user satisfaction (Keum et al., 2018). On a more macroscopic level, several studies dealt with spatial cognition and sensation in real-life situations in urban environments (Al-Barrak et al., 2017; Djebbara et al., 2019; Mavros et al., 2016) or while walking through an art museum (Cruz-Garza et al., 2017).
In conclusion, the above studies aimed to use brain activity measurement in natural environments to investigate neural aspects of work-related behavior. The technical prerequisites to establish these methods on a larger scale in the field of neuroergonomics seem to be fulfilled. However, further development is still needed.
Measurement Equipment
The possibility of recording EEG signals from a few single electrodes with portable amplifiers has existed since decades (e.g., Vitaport, TEMEC Technologies B.V., Netherlands; Biopac, BIOPAC Systems, Inc., USA). These single electrode systems still exist, but are either restricted with respect of the number of channels available for the EEG, lacking the possibilities of recording exactly timed trigger signals or of not being as portable as recent EEG systems specifically designed for mobile EEG. They are predominantly used for the measurement of peripheral electrophysiology in connection with some isolated EEG measures.
Problems of early systems were solved by introducing wireless transmission of EEG signals. This wireless EEG signal transmission allowed the collection of data from a regular electrode cap (up to 128 channels) in mobile settings. The recorded signal was transmitted to amplifiers that were implemented in a regular laboratory setting and allowed for the integration of stimulus presentation and exact temporal triggering. Given the possibility to evenly cover the entire head surface with electrodes, motion artifacts (in particular neck muscle artifacts) could be identified to clean the data (Gramann et al., 2010, Gramann, Jung, et al., 2014; Jungnickel & Gramann, 2016; Thompson et al., 2008).
Another major step in mobile EEG system development (Bateson et al., 2017) was the introduction of miniaturized mobile amplifiers that can be attached to the back of the head (Smarting, mBrainTrain, Serbia; LiveAmp, Brain Products GmbH, Germany; B-Alert X, Advanced Brain Monitoring, USA). Due to the fact that these amplifiers have integrated gyro-sensors, gait parameters can be recorded to identify movement patterns and motion-induced artifacts. These systems differ particularly in the way the data is stored. The first kind of system is able to stream the data wirelessly to a mobile device (i.e., smartphone), while other systems provide the possibility to store the data on an SD card that is inserted into the amplifier itself or they even combine the two. Both ways to store data have their advantages: whereas wireless transmission allows direct access to the data, for example, for BCI, on-board storage provides the participant with higher range of movement and eliminates problems concerning transmission errors.
With respect to neuroergonomics, these systems face two main problems: (1) EEG caps are obtrusive since the electrode caps cannot easily be hidden. (2) The use of cables induces electromagnetic noise by induction, which is an issue especially when the participants’ activity requires full body motion. It has to be mentioned here that in contrast to walking experiments, in which the walking movements are more or less regular, movements during varying work tasks are neither rhythmic nor repetitive and cannot be corrected for as easily as for walking.
A step toward less obtrusive measurements was achieved by applying adhesive film electrodes affixed behind the ear (Bleichner & Debener, 2017) and by using in-ear electrodes (Bleichner et al., 2015). The amplifiers can be hidden by a headband or a base cap. This way, the participant can act without attracting unwanted attention even in social situations. Still, this method comes with some shortcomings. First, skin texture can vary substantially with age, often resulting in lower signal quality and impaired contact retention of the adhesives for elderly persons. However, the main problem with this type of electrodes is the restricted scalp surface area covered by the implemented electrodes. This makes cortical activity distant to the covered area hard to capture. The same holds true for integrated systems that were developed either for rapid research application or end-consumer use (e.g., for meditation or gaming). These systems mainly consist of a head mount or headband with dry electrodes and an amplifier integrated into the cap. While integrated systems reduce the risk of artifacts due to cable motions, both the restricted number of channels and the poor technical specifications of the amplifiers limit their value for research. Nevertheless, they may still provide reliable data (Barham et al., 2017). Dry electrodes do not make contact to the scalp surface using electrolyte gel like regularly used wet EEG electrodes. The montage is faster but they are potentially less comfortable compared to wet electrodes, and there is not much experience with respect to long-term testing (Di Flumeri et al., 2019).
To sum up, finding the right recording method and devices can be quite daunting, but with a clear research question, the choice of the apparatus can be narrowed down. Table 1 lists most of the currently available mobile EEG systems along with their specifications. Still, this is not an exhaustive collection of all systems available and new systems are being developed and brought to market continuously (Brouwer et al., 2015).
List of Available Mobile EEG Systems and Their Specifications
As can be deduced from the previous paragraphs, there are many sources of artifacts that can interfere with the cortical signal. These artifacts may be inherent to the individual—for example, eye or bodily movements, ECG signal, or sweat—or due to external sources—for example, line noise, magnetic fields, or electrode movement. The artifacts inherent to the subject can hardly be prevented when recording in out-of-the-laboratory environments, but there are numerous ways to clean the signal. Using statistical procedures and threshold values during the offline preprocessing, artefactual segments can be detected and excluded from the signal. With the help of more sophisticated statistical procedures like the independent component analysis (ICA), certain artifacts with prototypical features (topography, spectral, or temporal properties) can be identified and excluded from the signal without having to discard parts of the data. Contributions of such noisy components can be subtracted from the signal. External noise sources should be avoided generally. Therefore, electrodes should be tightly fixed to the cap to avoid current induction in the cables and disconnection from the scalp (Symeonidou et al., 2018) . Also, newly developed dual-electrode systems are able to reduce motion-induced noise in the recorded signal (Nordin et al., 2018).
Besides choosing appropriate sensors and amplifiers, a proper setup and a well-controlled experimental framework are also crucial for integrating neurophysiological and behavioral data in studies of information processing. To clarify the framework of a project, it must be clear which parameters are of interest. Before measuring, an a priori roadmap should define which mental states are under investigation and how those states relate to neurophysiology (for some useful guidance, see Brouwer et al., 2015). In case the exact timing of events is not crucial—for example when longer-lasting mental states are considered—an approximate assignment of time codes to a time period is sufficient for data analyses. In order to implement any temporally more critical external stimulation, trigger signals have to be embedded into the experimental scenario using lightweight electronic stimulus presentation devices like a Raspberry Pi (Kuziek et al., 2017) or Smartphones (e.g., “Presentation Mobile,” Neurobehavioral Systems, Inc., USA). Integration of experimental information into a more complex laboratory setting with multiple measurement devices can be done by embedding the environment into the lab-streaming layer (LSL) framework that collects and synchronizes information from multiple sources (e.g., https://github.com/sccn/labstreaminglayer).
Available Physiological Parameters
Measures in the Frequency Domain: Spectral Power
The analysis of spectral power of the EEG (for a review see Al-Fahoum & Al-Fraihat, 2014), most times via Fast Fourier Transform (FFT), has proven to provide meaningful information with respect to human cognition and mental states (Borghini et al., 2014; Wascher et al., 2014). Research identified functionally distinct frequency bands, namely the Delta (~0–2 Hz), Theta (~3–7 Hz), Alpha (~8–12 Hz), Beta (~15–30 Hz), and Gamma (~30–100 Hz) bands.
The phenomenon of mental fatigue has been investigated extensively by means of EEG spectral analysis. Most studies report a shift of spectral power toward lower frequency bands, that is toward Delta, Theta, and Alpha, with increasing time on task (e.g., Fan et al., 2015; Wascher et al., 2014). However, power increases in the Beta band have also been reported (e.g., Craig et al., 2012). A recent review by Tran et al. (2020) found Theta and Alpha power increases to be reported most consistently. Thus, Alpha power has been proposed as a biomarker for mental fatigue and the associated risk of human error (Lal & Craig, 2001). However, the nature of mental fatigue is still under debate. A depletion of resources due to task demands (e.g., Helton & Warm, 2008) as well as cognitive underload (e.g., Pattyn et al., 2008; Smallwood & Schooler, 2006) were proposed as causal factors. The latter is supported by findings that linked an increase of Alpha power to low versus high task demands (Gevins et al., 1997; Wascher et al., 2019), to reactive rather than proactive task engagement (Karthaus et al., 2018), to phases of mind wandering (Arnau et al., 2020; Compton et al., 2019), and to an internally oriented focus of attention (c.f. Cooper et al., 2003; Hanslmayr et al., 2011).
The construct of cognitive workload is also highly relevant for human performance (c.f. Dehais et al., 2020) as attentional resources are limited (Just & Carpenter, 1992; Kahnemann, 1973). Executive functioning and working memory capacity constitute a bottleneck in human information processing (Cowan, 2001). For executive functioning, the prefrontal cortex (PFC) and the anterior cingulate cortex (ACC) act as a hub to orchestrate brain areas involved in a given task via inter-area communication (Botvinick, 2007; Helfrich & Knight, 2016; Hopfinger et al., 2000). Theta band activity, in particular frontal midline theta, has been linked to the exertion of executive control (Cavanagh & Shackman, 2015, Cavanagh & Frank, 2014; Cavanagh et al., 2012). Theta activity assessed via FFT has been reported to be sensitive to task demands (Gevins et al., 1997; Zakrzewska & Brzezicka, 2014) as well as to cognitive effort (e.g.,.Smit et al., 2005). Cognitive effort is crucial for the overall workload since it determines the amount of resources allocated to a task (c.f. Shenhav et al., 2017).
Measures in the Time Domain: Event-Related Potentials
Event-related potentials (ERPs) in the EEG reflect temporally specific and systematic voltage changes evoked by a temporally well-defined event. The basic principle of ERPs is that every act of information processing changes brain-electric activity in a specific way, which can be broken down into temporally distinct cognitive processes (Luck, 2014). The main advantage of using the ERP technique compared to all other neurocognitive approaches is the high temporal resolution, which is only limited by the sampling frequency. Positive and negative voltage deflections are interpreted as indicators of particular stages in cognitive processing.
Negative (N1, N2) and positive deflections (P1, P2, P3) give insights into specific aspects of information processing, such as sensory and attentional processes (Kramer et al., 1995; Pratt et al., 2011; Ullsperger et al., 2001), aspects of subjective and objective stimulus signal intensity (Kramer et al., 1995), and of stimulus probability or relevance (P3) as well as general resource availability. In the applied context, the P3 component (approx. 300–500 ms post-stimulus) is of high interest, as it is correlated with cognitive workload (Allison & Polich, 2008; Kok, 2001).
So far, research on event-related EEG activity is mainly restricted to laboratory settings, testing distinct cognitive factors such as working memory or skill acquisition that might be important for real-world applications. This has been done in highly restricted workplace simulations, for example by Mijović et al. (2017), but also in a realistic logistics workplace simulation (Wascher, Heppner, et al., 2014). Wascher, Heppner, et al. (2014) showed distinct changes in ERP-component amplitudes while participants had to deal with different load-inducing situations. The main reason for the sparse use of these EEG measures in realistic environments is the fact that the signal of interest (namely the modulations of EEG activity) is embedded in stochastic noise and spontaneous oscillatory activity. Movement and other high voltage fluctuations make it hard to extract reliable and meaningful ERP measures to evaluate underlying cognitive processes in naturalistic environments.
To obtain a proper evoked signal, many time-distinctive repetitions of equivalent events are needed to perform averaging. In doing so, the aforementioned random, task-unrelated noise is removed through the averaging process, leaving only the task-related variability of the event-related signal. Unfortunately, natural environments and tasks lack such repeating events. Thus, there are certain fields of research where it seems necessary to focus on laboratory experiments, especially when considering the research on finely grained cognitive processing. However, there are possibilities to overcome these shortcomings that will be discussed later in this review.
With amplifiers becoming much smaller and more mobile than just some years ago, study designs were enabled that allowed participants to move in full-body motion, either in laboratory facilities (Gramann et al., 2010; Shaw et al., 2018) or in real-life environments (Ladouce et al., 2016, 2019; Reiser et al., 2019, 2020; Scanlon et al., 2019). Studying the influence of real-world motion on cognitive processes, these studies found that even locomotion impairs the availability of cognitive resources.
And finally, while ERPs are a valid tool for evaluating cognitive processes in high temporal resolution, they may lack some informative content of the frequency domain. There are certain task-related processes that are indicated by modulations of oscillatory activity in the Theta or Alpha frequency range explained earlier, that ERPs simply cannot reveal.
Measures in the Time-Frequency Space: Time-Frequency Analyses
Time-frequency analysis allows for analyzing temporal and spectral properties of the EEG simultaneously. The most commonly used methods to perform time-frequency decomposition of the signal are (1) convolving the data with complex Morlet wavelets, (2) the filter-Hilbert-transform, and (3) the short-time FFT (Cohen, 2014). The result of such a signal decomposition is a set of frequency specific time-series of oscillatory power from which event-related spectral perturbations (ERSPs) may be calculated. The ERSP reflects event-related variance in the signal in time-frequency space. Several correlates of psychological constructs and cognitive processes that are relevant to the field of neuroergonomics have been identified using time-frequency analysis.
Event-related frontal Theta activity has been described as indicating stimulus- or response-related cognitive control functioning (Cavanagh & Frank, 2014). It was consequently found to reflect cognitive interference (Nigbur et al., 2011), dynamic changes in mental workload in motor and cognitive tasks with varying difficulty (So et al., 2017), or cognitive effort (Onton et al., 2005). Furthermore, frontal midline Theta was found to be modulated by task load, multitasking, and prolonged focused attention in several basic research and workplace simulation tasks (for an overview, see Borghini et al., 2014). In mobile EEG recordings, different patterns of Theta activation were found for experimental conditions in which participants were in motion compared to standing, all while performing a cognitive task (Pizzamiglio et al., 2017; Reiser et al., 2019, 2020; Shaw et al., 2018). These results highlight the role of frontal midline Theta as a correlate of central executive mechanism during locomotion that controls resource allocation in situations of cognitive-motor interference.
Modulations in the Alpha band (~8–12 Hz) have been assigned to the orientation of attention and working memory processes. Changes in Alpha power in posterior (visual) brain areas were already observed at the very beginning of EEG research when it was found that Alpha power was suppressed in mental states of open versus closed eyes (compare Berger, 1929). This suggests that the oscillatory response of large (visual) neuronal populations were desynchronized. Importantly, the effect is not limited to the bottom-up processing of sensory signals, but can also be used in a top-down way. As proposed by Klimesch et al. (2007), the top-down synchronization of Alpha oscillations is important for the inhibitory control and timing of information processing. In contrast, a relative desynchronization of Alpha power can be seen as a release from the cortical inhibition and thus promotes goal-oriented processes (Haegens et al., 2011; Rösner et al., 2020; Wöstmann et al., 2016).
Finally, central and centro-parietal Mu (~8–12 Hz) and accompanying Beta (~15–30 Hz) oscillations are related to motor planning and the body-centered shifting of attention (see Hari & Salmelin, 1997). When planning an action or awaiting a somato-sensation, a desynchronization of Mu/Beta power can be observed in the sensorimotor cortical areas coding for the respective body part (Llanos et al., 2013; Neuper et al., 2006; van Ede et al., 2011). Thus, while changes in the topography of Alpha oscillations over parietal and parieto-occipital brain areas are linked to the orienting of attention in external space, modulations of Mu oscillations over central and centro-parietal regions might be linked to the body-centered focusing of attention. Measuring modulations of Mu and Beta oscillations with mobile EEG setups could thus prove a valuable approach for assessing the planning and execution of actions or the focusing of attention to certain body regions, since the respective oscillatory effects should not significantly be affected by changing environmental conditions. For example, the measurement of topographic distributions of Mu and Beta oscillations via mobile EEG has recently been shown to be a reliable indicator for motor asymmetries in a realistic environment (Packheiser et al., 2020).
The findings reported above indicate that there are reliable electrophysiological correlates of constructs relevant for the field of neuroergonomics in time-frequency space, and that time-frequency decomposition constitutes a valuable tool in mobile EEG research. In the context of driving, for example, mental states could already be categorized with the help of Alpha- and Theta-band activity using small around-the-ear electrode arrays (Wascher et al., 2019). In future studies, these methods could also be deployed to improve interactive and collaborative human–robot or human–machine scenarios, for example, by using a brain–computer interface (e.g., Berka et al., 2005, Blankertz et al., 2010; Dorneich et al., 2005, 2007).
Measures Using Machine Learning and Artificial Intelligence
All of the introduced measures dealt with univariate approaches: a single electrode, or an averaged electrode batch, always concentrating on a narrow topographical field (e.g., frontal midline theta, parietal P3). With advances in computational power, multivariate approaches were made possible so that information about whole scalp activations could be taken into consideration to explain human cognition. Machine learning algorithms have been used to decode multi-dimensional spatial and/or temporal data series. For example, EEG microstates have been used to decipher cognitive states by integrating the information from all electrodes to generate prototypical quasi-steady patterns, giving way to the analysis of cortical interaction activity (Michel & Koenig, 2018). These prototypical states have been used to successfully categorize functionally useful attentional states of individuals (c.f. Bréchet et al., 2019; Milz et al., 2016).
Methodological Restrictions and Work-Arounds: Event-Related Analysis in Realistic Settings
When addressing EEG measurements at workplaces, the core problem for the event-related analysis is the absence of precisely timed repetitive stimulation. In a workplace simulation, probe stimuli might be added to the scenery as noticeable, reappearing events. When presenting task irrelevant probe tones, for example, event-related activity of the EEG indicates whether the tone was attended to or not (Allison & Polich, 2008; Scheer et al., 2016). The same applies to the visual domain as EEG activity indicates the spatial distribution of attention. Attention to a particular part of a scene should be reflected in modulated ERP responses to randomly presented light flashes at different locations of a scene. With the help of augmented reality via data glasses and the use of scene cameras (e.g., implemented into a mobile eye tracker), such an approach could also be applied to realistic settings. However, the processing of the probe stimulus may exacerbate the assessment of the work-related cognitive processes of interest.
An alternative approach to get precise timing in natural settings might be to use events that are inherent to the environment or to the behavior of the participant. Systematic external events with a high number of repetitions are hardly ever found in a natural environment and often lack a sudden onset needed for exact time-locked averaging. Visual events normally build up as smoothly arising changes in the scene, at least when the observer is moving. A solution to this problem that relies on the inherent behavior of the acting person is the measurement of eye movement parameters. Eye movements are strongly connected to visual information processing and temporally well defined. This holds true both for saccadic eye movements and for eye blinks.
Saccades (see Viviani, 1990) are brief, fast movements of the eyes that change the point of fixation toward an object of interest. They reflect the overt orienting of attention. Using this behavior for EEG research, fixation-related potentials (FRPs) have repeatedly been reported in studies that investigate visual search (Hiebel et al., 2018; Kamienkowski et al., 2012, 2018; Kaunitz et al., 2014). Kaunitz et al. (2014) demonstrated that sensory and cognitive FRP components were similar to stimulus-driven ERPs when time-locked to the onset of a fixation. Moreover, effects of task load are reflected in FRPs in a similar manner as in ERPs (Ries et al., 2016)
Modulations in FRPs may predict whether fixated content will be encoded or not. This could be shown for pictorial content (Nikolaev et al., 2011) as well as the meaning of words during reading (Frey et al., 2018; Sato & Mizuhara, 2018). Furthermore, the decoding of emotional expressions has been studied in regular lab settings (Guérin-Dugué et al., 2018; Simola et al., 2015) and with mobile EEG in natural settings (Soto et al., 2018).
However, there is a substantial problem of saccade-related EEG analyses outside of the laboratory, namely motion. Humans are not able to move their eyes voluntarily in a slow, continuous, and steady motion. Saccades are discrete events that occur when information processing is finished at a particular spatial location and new content is attended to. When motion is involved, our eyes follow moving objects smoothly, even for longer time intervals, when aspects of this object are relevant to the task (Robinson, 1965). Thus, when motion is part of participants’ behavior or the scene, reorienting of attention is not necessarily a discrete shift of gaze. Consequently, saccades are losing their temporal information with respect to attention allocation.
Blinking appears to be more interesting for real-life evaluations for several reasons. Beside saccades, blinking and eye closure play a large role in mental state detection. In fatigue detection for example, average blink duration has been shown to correlate with operator state (Funke et al., 2017; Papadelis et al., 2007; Rohit et al., 2017).
Besides being a measure on its own, eye blinks can also be useful to create events inherent to the experimental situation without further stimulation. It is generally accepted that blinks may occur either voluntarily, due to startling signals (in order to protect the eye), or occur stochastically distributed in order to maintain the tear film on the cornea. The latter, however, is to be questioned. First of all, blink frequency strongly varies with the neurotransmitter dopamine (Karson, 1983) and thus also with motivational characteristics (Colzato et al., 2008). Second, blinks occur predominantly at the end of an information stream, for example, at the end of scenes when watching movies, or at the end of a sentence when reading (Fogarty & Stern, 1989; Nakano et al., 2009). In laboratory settings, blinks have been observed after a decision was made or at the end of an experimental trial, when information processing is finished (Wascher et al., 2015). Even when auditory information is processed, the same timing of eye blinks occurs (Kobald et al., 2019). Thus, eye blinks may be reliable markers of information segmentation.
This assumption can be supported by event-related activity of the brain. Following the eye blink, an N1-like component can be observed, but this only happened when blinking in an illuminated surrounding—in darkness no ERP occurred (Heuser-Link et al., 1992). The same can be observed for the cognitive P3 component of the ERP (Berg & Davies, 1988). MEG studies showed that eye blinks activate the precuneus regions, indicating increased environmental monitoring and awareness (Liu et al., 2017). Additionally, blink-evoked modulations have been reported in the context of ERSPs (Bonfiglio et al., 2009, 2013). In a more applied setting, namely the simulation of a logistics workplace, Wascher, Heppner, et al. (2014) have shown that both blink-evoked ERSPs and ERPs provide valuable information about cognitive load measured with mobile EEG.
Hands-On Neuroergonomics
As can be deduced from the previous sections, (mobile) EEG is a viable and cost-effective method to get a grasp of cognitive processes underlying many aspects of working life. When no overt behavior is available to make assumptions about an individual’s cognitive state, EEG has the potential to elucidate processes unknown to the distant observer. Attention- and fatigue-related EEG investigations have proven the measurement’s usefulness in many areas of life, such as driving (e.g., Borghini et al., 2014; Getzmann et al., 2018; Lin et al., 2014) or operating a machine (Aricò et al., 2016). By decoding cognitive states of individuals, adaptive technological procedures can be regulated either to boost the individual’s attentional focus toward the task or to intercept decreasing wakefulness by enhancing aid by the system. The EEG can also be used to determine aspects of higher-order constructs, such as situational awareness, to be used in the previously mentioned interventions.
Besides the pure attentional state, there are plenty of psychological constructs relevant for the positive outcome in sociotechnical systems, for example, emotion and affect (for a short overview, see Frey et al. (2014)) that can also be classified by using neurophysiological measures (Zander & Jatzev, 2009). This is crucial for situations that could entail hazardous results in case a system user is not able to respond to situational requirements due to maladjusted states of arousal or affect, as for example during driving (Thirunavukkarasu et al., 2016). It is also possible to improve usability evaluations using EEG emotion classification during the interaction with everyday utensils (Sargent et al., 2020).
With the increasing mobility of EEG devices, these use-cases can be extended from seated conditions, such as driving or static machine interaction, to naturalistic scenarios. Several correlates could be classified in such environments, for example, workload and cognitive processing while riding a bike (McLean et al., 2017; Scanlon et al., 2019), or affective and recreational states while walking through three different urban areas (Aspinall et al., 2015). By using a mobile phone running a software to synchronize data-streams of motion-tracking, EEG, and eye-tracking (e.g., Lab Streaming Kothe, 2014), it would be possible to get insights into everyday interaction in socio-technical systems without social, environmental, or behavioral constraints, of course while minding the pitfalls of the involved techniques (Brouwer et al., 2015).
Discussion
The aim of cognitive neuroergonomics is to improve safety and well-being in workplaces or everyday environments with the help of neurophysiological measures that allow an understanding of the mental mechanisms of workers under specific work requirements (Mehta & Parasuraman, 2013; Parasuraman, 2011). Based on this knowledge, working and everyday environments can be modified for enhanced safety and work efficiency (Berka et al., 2004; Dorneich et al., 2005, 2007; Wilson & Russell, 2007). To this end, the following issues need to be addressed—in convergence and addition to those points made by Brouwer et al. (2015):
Concepts and measures of cognitive neuroergonomics should allow to track mental states in order to evaluate specifics of a particular workplace.
The continuous tracking of worker states should allow the identification of individual load in specific work situations, that is, detecting moments of mental strain.
Cognitive neuroergonomics aims at the evaluation of information processing in regular and ambulatory working situations under consideration of the aspects mentioned in (1) and (2).
Based on this approach, countermeasures can be developed that protect the worker from adverse health issues and ensure work quality in safety-critical workplaces (e.g., Huang et al., 2012).
Along this path, the first steps have been taken since mobile EEG measurement became feasible and the extracted parameters demonstrated high reliability despite the challenges that come with mobile EEG recordings. Technically, the online evaluation of the data is now possible in conjunction with closed loop systems that feed mental state-related parameters back to the worker. However, the database of mobile EEG evaluations in free body motion is sparse. Data from driving or flight simulations cannot be considered, because these data are recorded in rather static environments under highly controlled environmental conditions. Also, regular gait studies have limited explanatory power since treadmill walking is a rather rhythmic movement that represents a well-learned and automated motion and is assumed to not consume many cognitive resources. Unpredictable movement behavior, as is inherent to many workplaces or walking in nature, may interact with cognition in a differential manner and, with respect to the measure, provoke a yet unknown pattern of artifacts.
So far, most neuroergonomics studies addressed a description of the subject’s state. In particular, measures in the frequency domain provide reliable access to cognitive constructs such as mental fatigue, attention allocation, or mental load. These measures form the basis of neuroergonomics research since they are relatively robust against singular artifacts, that are unavoidable when participants are moving freely. Results from previous simulator and laboratory studies show a high similarity to the results in natural settings and therefore allow to transfer the existing knowledge to the field. In addition to the rather objective access to user states in almost any environment by using response times or measures of primary task performance, mobile EEG allows to establish neuronal models in order to gain an understanding of the implementation of worker states into neural interactions in a broader sense. Based on these neuronal models, well-founded countermeasures to avoid undesired worker states can be developed.
Neurophysiological measures of cognitive processing, that is, the entire cascade of information processing from signal input to action execution become accessible with mobile EEG. However, work-related environments that require unrestricted movement need the design and evaluation of new study approaches (such as eye blink or fixation-related analyses) to enhance mobile EEG measurements. In this way, all parameters that have previously been established in the laboratory or in restricted simulation environments may be accessible. The application of mobile EEG methods in real workplaces, however, still faces some unresolved problems. It would be recommended that the equipment used for recording the EEG should be unobtrusive and (almost) invisible to others to not hinder the worker to behave normally. This can be achieved with headband-based systems or with the use of adhesive electrodes (e.g., cEEGrids, University of Oldenburg, Germany). However, these systems cover only a small area of the skull and therefore do not allow for a clear separation of cortical sources or, more specifically, of the sources’ origins. Thus, a validation of such constrained measures with data that is collected with extended multi-channel systems in a realistic simulation appears necessary. In general, all the psychological constructs outlined above need to be evaluated systematically in real-life environments in order to understand how the human brain processes information in working situations. For the validation, other physiological measures with a longer history of mobile measurements of mental states need to be considered (e.g., HRV, EDA, or salivary cortisol; Dirican & Göktürk, 2011; Fairclough & Venables, 2006).
Conclusion
Building on decades of knowledge about cognitive functioning obtained using EEG in laboratory conditions, the application in real-world paradigms in order to bridge the gap between behavior and mind is very compelling as recording technologies become smaller and smaller. Studies in workplace and real-life simulations have already shown that the EEG is suitable to quantify mental workload with very high accuracy—maybe higher than the accuracy of other physiological measures, for example, the electrocardiogram or eye movements (Ahn et al., 2016; Hankins & Wilson, 1998; Hogervorst et al., 2014; Wilson, 2002). With recent technical developments allowing increasingly smaller recording devices, mobile EEG can be applied in many fields. In particular, considering the field of ergonomics, mobile EEG can be downsized to the point of being almost unobtrusive in regular working environments. Thus, it provides the potential to track mental aspects of diverse workplaces. Neurophysiological correlates of cognitive processes during work can be assessed in a depth comparable to laboratory studies. And while there still may be some shortcomings of mobile EEG compared to its laboratory counterpart, such as the potential for noisier data and the lack of temporally precise stimulation, all of these can be overcome and will surely be overcome in near future. And last but not least, some interesting approaches to the mobile measurement of EEG have been outlined in this overview, which provide completely new insights into cognition at the workplace.
Key Points
Mobile EEG provides objectifiable access to mental states and cognitive processing during work
Recent developments both with respect to hardware and data processing techniques allow concept-based research in almost any natural environment.
Measures of brain activity, however, need to be carefully adjusted to the mental concepts addressed.
Footnotes
Acknowledgments
Edmund Wascher and Julian Reiser have contributed equally to this study and should be considered as co-first authors. E.W., J.R. and S.A. developed the outline of the manuscript and prepared the initial version. All other authors contributed by specific contributions mainly to the methods and cognitive aspects of the EEG. All authors were involved in the preparation of the final version.
Author Biographies
Edmund Wascher is scientific director at the Leibniz Research Centre for Working Environments and Human Factors and professor of ergonomics at the Technical University of Dortmund. He earned his PhD in psychology in 1992 from the University of Graz (Austria).
Julian Reiser is a post-doc researcher at the Leibniz Research Centre for Working Environment and Human Factors. He earned his PhD from Ruhr-University Bochum in 2021.
Gerhard Rinkenauer is senior researcher at the Leibniz Research Centre for Working Environments and Human Factors, head of the research area of work design at the department of ergonomics, and associate professor of Cognitive Psychology at the Technical University of Dortmund. He received his PhD in psychology from the University of Wuppertal in 1999.
Mauro Larrá is a post-doc researcher at the Leibniz Research Centre for Working Environments and Human Factors. He received his PhD in Psychobiology in 2015 from the University of Trier (Germany).
Felix A. Dregeris researcher and PhD candidate at the Department of Ergonomics of the Leibniz Research Centre for Working Environments and Human Factors. He received his MSc in psychology from the University of Tubingen in 2016.
Daniel Schneider is a research group leader at the Leibniz Research Centre for Working Environments and Human Factors. He received his PhD in Psychology in 2012 from Ruhr-University Bochum (Germany).
Melanie Karthaus is research assistant at the Leibniz Research Centre for Working Environment and Human Factors. She earned her PhD in Psychology in 2008 from the University of Trier (Germany).
Stephan Getzmann is head of the Networking Group Aging at the Leibniz Research Centre for Working Environments and Human Factors. He received his PhD in Psychology in 2004 from the Ruhr-University Bochum and qualified 2010 as a university lecturer at the Technical University of Dortmund.
Marie Gutberlet is the owner and CEO of BlindSight GmbH, a company specializing in the development of biosignal devices, accessories and related analysis methods, with special emphasis on EEG, ExG, and fMRI. She earned her PhD in physiological psychology in 1994 from the University of Trier (Germany).
Stefan Arnau is a post-doctoral researcher at the Leibniz Research Centre for Working Environments and Human Factors at the Technical University of Dortmund. He received his PhD from the Ruhr-University Bochum in 2019.