We use a set of unobtrusive measures to estimate subjectively reported trust, mental workload, and situation awareness (henceforth “TWSA”).
Background
Subjective questionnaires are commonly used to assess human cognitive states. However, they are obtrusive and usually impractical to administer during operations. Measures derived from actions operators take while working (which we call “embedded measures”) have been proposed as an unobtrusive way to obtain TWSA estimates. Embedded measures have not been systematically investigated for each of TWSA, which prevents their operational utility.
Methods
Fifteen participants completed twelve trials of spaceflight-relevant tasks while using a simulated autonomous system. Embedded measures of TWSA were obtained during each trial and participants completed TWSA questionnaires after each trial. Statistical models incorporating our embedded measures were fit with various formulations, interaction effects, and levels of personalization to understand their benefits and improve model accuracy.
Results
The stepwise algorithm for building statistical models usually included embedded measures, which frequently corresponded to an intuitive increase or decrease in reported TWSA. Embedded measures alone could not accurately capture an operator’s cognitive state, but combining the measures with readily observable task information or information about participants’ backgrounds enabled the models to achieve good descriptive fit and accurate prediction of TWSA.
Conclusion
Statistical models leveraging embedded measures of TWSA can be used to accurately estimate responses on subjective questionnaires that measure TWSA.
Application
Our systematic approach to investigating embedded measures and fitting models allows for cognitive state estimation without disrupting tasks when administering questionnaires would be impractical.
AkashK.HuW.-L.JainN.ReidT. (2018). A classification model for sensing human trust in machines using EEG and GSR. ACM Transactions on Interactive Intelligent Systems, 8(4), 1–20. https://doi.org/10.1145/3132743
3.
AkashK.McMahonG.ReidT.JainN. (2020). Human trust-based feedback control: dynamically varying automation transparency to optimize human-machine interactions. IEEE Control Systems Magazine, 40(6), 98–116. https://doi.org/10.1109/MCS.2020.3019151
4.
BasnerM.SavittA.MooreT. M.PortA. M.McGuireS.EckerA. J.NasriniJ.MolliconeD. J.MottC. M.McCannT.DingesD. F.GurR. C. (2015). Development and validation of the cognition test battery for spaceflight. Aerospace Medicine and Human Performance, 86(11), 942–952. https://doi.org/10.3357/AMHP.4343.2015
BessonP.MaïanoC.BringouxL.MarquesteT.MestreD. R.BourdinC.DoussetE.DurandM.VercherJ. (2012). Cognitive workload and affective state: A computational study using Bayesian networks. 2012 6th IEEE International Conference Intelligent Systems, 1(1), 140–145. https://doi.org/10.1109/IS.2012.6335127
7.
CasaliJ. G.WierwilleW. W. (1983). A comparison of rating scale, secondary-task, physiological, and primary-task workload estimation techniques in a simulated flight task emphasizing communications load. Human Factors, 25(6), 623–641. https://doi.org/10.1177/001872088302500602
8.
CasnerS. M.GoreB. F. (2010). Measuring and evaluating workload: A primer. NASA Technical Memorandum, 216395, 2010.
9.
ChristensenR. H. B. (2019). ordinal—Regression models for ordinal data.
10.
CraigS. D.ChiouE. K.SchroederN. L. (2019). The impact of virtual human voice on learner trust. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 2272–2276. https://doi.org/10.1177/1071181319631517
11.
de WinterJ. C. F.EismaY. B.CabrallC. D. D.HancockP. A.StantonN. A. (2019). Situation awareness based on eye movements in relation to the task environment. Cognition, Technology & Work, 21(1), 99–111. https://doi.org/10.1007/s10111-018-0527-6
12.
DingY.CaoY.DuffyV. G.WangY.XuefengZ. (2020). Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning. Ergonomics, 1–32, 896–908, https://doi.org/10.1080/00140139.2020.1759699
13.
DursoF. T.HackworthC. A.TruittT. R.CrutchfieldJ.NikolicD.ManningC. A. (1998). Situation awareness as a predictor of performance for en route air traffic controllers. Air Traffic Control Quarterly, 6(1), 1–20. https://doi.org/10.2514/atcq.6.1.1
14.
DzindoletM.PierceL.BeckH.DaweL. (2002). The perceived utility of human and automated aids in a visual detection task. Human Factors, 44, 79–94. https://doi.org/10.1518/0018720024494856
15.
EndsleyM. R. (1988a). Situation awareness global assessment technique (SAGAT). Proceedings of the IEEE 1988 National Aerospace and Electronics Conference, Dayton, OH, USA, 23–27 May 1988, pp. 789–795 vol.3. https://doi.org/10.1109/NAECON.1988.195097
16.
EndsleyM. R. (1988b). Design and evaluation for situation awareness enhancement. Proceedings of the Human Factors Society Annual Meeting, 32(2), 97–101. https://doi.org/10.1177/154193128803200221
EndsleyM. R. (2021). A systematic review and meta-analysis of direct objective measures of situation awareness: a comparison of SAGAT and SPAM. Human Factors, 63(1), 124–150. https://doi.org/10.1177/0018720819875376
EzekielM. (1930). Methods of correlation analysis (pp. xiv, 427). Wiley.
21.
FeighK. M.DorneichM. C.HayesC. C. (2012). Toward a characterization of adaptive systems: a framework for researchers and system designers. Human Factors, 54(6), 1008–1024. https://doi.org/10.1177/0018720812443983
22.
GaudetteL.JapkowiczN. (2009). Evaluation methods for ordinal classification. In GaoY.JapkowiczN. (Eds.), Advances in Artificial Intelligence (pp. 207–210). Springer. https://doi.org/10.1007/978-3-642-01818-3_25
23.
GhazaliA. S.HamJ.BarakovaE. I.MarkopoulosP. (2018). Effects of robot facial characteristics and gender in persuasive human-robot interaction. Frontiers in Robotics and AI, 5. 73, https://doi.org/10.3389/frobt.2018.00073
24.
GombolayM.YangX. J.HayesB.SeoN.LiuZ.WadhwaniaS.YuT.ShahN.GolenT.ShahJ. (2018). Robotic assistance in the coordination of patient care. The International Journal of Robotics Research, 37(10), 1300–1316. https://doi.org/10.1177/0278364918778344
25.
GuoY.YangX. J. (2020). Modeling and predicting trust dynamics in human–robot teaming: A Bayesian inference approach. International Journal of Social Robotics, 13(8), 1899–1909. https://doi.org/10.1007/s12369-020-00703-3
26.
GutzwillerR. S.ChiouE. K.CraigS. D.LewisC. M.LemattaG. J.HsiungC.-P. (2019). Positive bias in the ‘Trust in Automated Systems Survey’? An examination of the Jian et al. (2000) scale. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 217–221. https://doi.org/10.1177/1071181319631201
27.
HainleyC. J.DudaK. R.OmanC. M.NatapoffA. (2013). Pilot performance, workload, and situation awareness during lunar landing mode transitions. Journal of Spacecraft and Rockets, 50(4), 793–801. https://doi.org/10.2514/1.A32267
28.
HancockP. A.MatthewsG. (2018). Workload and performance: Associations, insensitivities, and dissociations: Human Factors. https://doi.org/10.1177/0018720818809590
29.
HancockP. A.WulfG.ThomD.FassnachtP. (1990). Driver workload during differing driving maneuvers. Accident; Analysis and Prevention, 22(3), 281–290. https://doi.org/10.1016/0001-4575(90)90019-h
30.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In HancockP. A.MeshkatiN. (Eds.), Advances in Psychology (Vol. 52, pp. 139–183). North-Holland. https://doi.org/10.1016/S0166-4115(08)62386-9
31.
HeardJ.AdamsJ. A. (2019). Multi-dimensional human workload assessment for supervisory human–machine teams. Journal of Cognitive Engineering and Decision Making, 13(3), 146–170. https://doi.org/10.1177/1555343419847906
32.
HeardJ.FortuneJ.AdamsJ. A. (2020). SAHRTA: A supervisory-based adaptive human-robot teaming architecture. 2020 IEEE Conference on Cognitive and Computational Aspects of Situation. Management (CogSIMA), 1–8. https://doi.org/10.1109/CogSIMA49017.2020.9215996
33.
HeardJ.HarriottC. E.AdamsJ. A. (2018). A survey of workload assessment algorithms. IEEE Transactions on Human-Machine Systems, 48(5), 434–451. https://doi.org/10.1109/THMS.2017.2782483
34.
HicksT. G.WierwilleW. W. (1979). Comparison of five mental workload assessment procedures in a moving-base driving simulator. Human Factors, 21(2), 129–143. https://doi.org/10.1177/001872087902100201
35.
HooeyB. L.KaberD. B.AdamsJ. A.FongT. W.GoreB. F. (2018). The underpinnings of workload in unmanned vehicle systems. IEEE Transactions on Human-Machine Systems, 48(5), 452–467. https://doi.org/10.1109/THMS.2017.2759758
JianJ.-Y.BisantzA. M.DruryC. G. (2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71. https://doi.org/10.1207/S15327566IJCE0401_04
38.
KantowitzB. H.HanowskiR. J.KantowitzS. C. (1997). Driver acceptance of unreliable traffic information in familiar and unfamiliar settings. Human Factors, 39(2), 164–176. https://doi.org/10.1518/001872097778543831
39.
KarasinskiJ. A.RobinsonS. K.DudaK. R.PrasovZ. (2016). Development of real-time performance metrics for manually-guided spacecraft operations. 2016 IEEE Aerospace Conference,1–9. https://doi.org/10.1109/AERO.2016.7500734
40.
KarasinskiJ. A.RobinsonS. K.HandleyP.DudaK. R. (2017). Real-time performance feedback in a manually-controlled spacecraft inspection task. In AIAA Modeling and Simulation Technologies Conference. American Institute of Aeronautics and Astronautics. https://doi.org/10.2514/6.2017-1314
KunzeA.SummerskillS. J.MarshallR.FiltnessA. J. (2019). Automation transparency: Implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics, 62(3), 345–360. https://doi.org/10.1080/00140139.2018.1547842
LinL.-W.LuM.-S. (2017). Empirical research regarding effects of pilot age and flight hours on pilot workload and situation awareness. Journal of Aeronautics, Astronautics and Aviation, 49(1), 31–47. https://doi.org/10.6125/16-1114-910
46.
LiuS.WanyanX.ZhuangD. (2014a). Modeling the situation awareness by the analysis of cognitive process. Bio-Medical Materials and Engineering, 24(6), 2311–2318. https://doi.org/10.3233/BME-141044
47.
LiuS.WanyanX.ZhuangD. (2014b). A quantitative situational awareness model of pilot. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 3(1), 117–122. https://doi.org/10.1177/2327857914031019
48.
MangiaficoS. S. (2016). Summary and analysis of extension program evaluation in R. Rutgers Cooperative Extension: New Brunswick, NJ, USA, 125, 16–22.
49.
MerrittS. M.Ako-BrewA.BryantW. J.StaleyA.McKennaM.LeoneA.ShiraseL. (2019). Automation-induced complacency potential: Development and validation of a new scale. Frontiers in Psychology, 10. 225, https://doi.org/10.3389/fpsyg.2019.00225
50.
NagelkerkeN. J. D. (1991). A note on a general definition of the coefficient of determination. Biometrika, 78(3), 691–692. https://doi.org/10.1093/biomet/78.3.691
51.
NunnallyJ. C. (1978). Psychometric Theory/Jum C. Nunnaly.
ParasuramanR.SheridanT. B.WickensC. D. (2008). Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making, 2(2), 140–160. https://doi.org/10.1518/155534308X284417
54.
PetersenL.RobertL.YangX. J.TilburyD. (2019). Situational awareness, driver’s trust in automated driving systems and secondary task performance. SAE International Journal of Connected and Automated Vehicles, 2(2), 12. https://doi.org/10.4271/12-02-02-0009
55.
QuanN. T. (1988). The prediction sum of squares as a general measure for regression diagnostics. Journal of Business & Economic Statistics, 6(4), 501–504. https://doi.org/10.2307/1391469
56.
R Core Team. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/.
57.
RajuN. S.BilgicR.EdwardsJ. E.FleerP. F. (1997). Methodology review: Estimation of population validity and cross-validity, and the use of equal weights in prediction. Applied Psychological Measurement, 21(4), 291–305. https://doi.org/10.1177/01466216970214001
RoscoeA. H. (1979). Handling qualities, workload and heart rate. In AGARDograph No. 246; Survey of Methods to Assess Workload. AGARD.
60.
RoscoeA. H. (1984). Assessing pilot workload in flight. Flight test techniques (AGARD-CP373). In Proceedings of NATO Advisory Group for Aerospace Research and Development (AGARD). AGARD.
61.
RoscoeA. H.EllisG. A. (1990). A subjective rating scale for assessing pilot workload in flight: A decade of practical use. Royal Aerospace Establishment Farnborough (United Kingdom). https://apps.dtic.mil/sti/citations/ADA227864
62.
SandersT.KaplanA.KochR.SchwartzM.HancockP. A. (2019). The relationship between trust and use choice in human-robot interaction. Human Factors, 61(4), 614–626. https://doi.org/10.1177/0018720818816838
63.
SchwarzJ.FuchsS. (2018). Validating a “Real-Time Assessment of Multidimensional User State” (RASMUS) for adaptive human-computer interaction. 2018IEEE International Conference on Systems, Man, and Cybernetics (SMC),704–709. https://doi.org/10.1109/SMC.2018.00128
64.
SelconS. J.TaylorR. (1990). Evaluation of the Situational Awareness Rating Technique (SART) as a tool for aircrew systems design. Situational Awareness in Aerospace Operations (AGARD-CP-478), 478, 1–8.
65.
SelconS. J.TaylorR. M.KoritsasE. (1991). Workload or situational awareness?: TLX vs. SART for aerospace systems design evaluation. Proceedings of the Human Factors Society Annual Meeting, 35(2), 62–66. https://doi.org/10.1518/107118191786755706
66.
SpainR. D.BustamanteE. A.BlissJ. P. (2008). Towards an empirically developed scale for system trust: Take two: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. https://doi.org/10.1177/154193120805201907
SzalmaJ. L. (2009). Individual differences in human–technology interaction: Incorporating variation in human characteristics into human factors and ergonomics research and design. Theoretical Issues in Ergonomics Science, 10(5), 381–397. https://doi.org/10.1080/14639220902893613
69.
TaylorR. M. (1990). Situational awareness rating technique (SART): The development of a tool for aircrew systems design. Situational Awareness in Aerospace Operations (AGARD-CP-478), 478, 1-17.
70.
Van AckerB. B.ParmentierD. D.VlerickP.SaldienJ. (2018). Understanding mental workload: From a clarifying concept analysis toward an implementable framework. Cognition, Technology & Work, 20(3), 351–365. https://doi.org/10.1007/s10111-018-0481-3
WickensC. D.DixonS. R. (2007). The benefits of imperfect diagnostic automation: A synthesis of the literature. Theoretical Issues in Ergonomics Science, 8(3), 201–212. https://doi.org/10.1080/14639220500370105
73.
WickensC. D.FitzgeraldN. J.CleggB. A.SmithC. A. P.OrthD.KincaidK. (2020). Decision aiding for nautical collision avoidance: Trust, dependence, and implicit understanding of the decision algorithm. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 1950–1954. https://doi.org/10.1177/1071181320641470
74.
WierwilleW. W.EggemeierF. T. (1993). Recommendations for mental workload measurement in a test and evaluation environment. Human Factors, 35(2), 263–281. https://doi.org/10.1177/001872089303500205
75.
XuA.DudekG. (2015). OPTIMo: Online Probabilistic Trust Inference Model for Asymmetric Human-Robot Collaborations. Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, 221-228. https://doi.org/10.1145/2696454.2696492.
76.
YangX. J.SchemanskeC.SearleC. (2021). Toward quantifying trust dynamics: How people adjust their trust after moment-to-moment interaction with automation. Human Factors, 00187208211034716. https://doi.org/10.1177/00187208211034716
77.
YangX. J.UnhelkarV. V.LiK.ShahJ. A. (2017). Evaluating effects of user experience and system transparency on trust in automation. Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, 1(1), 408–416. https://doi.org/10.1145/2909824.3020230
78.
YerkesR. M.DodsonJ. D. (1908). The relation of strength of stimulus to rapidity of habit rormation. Journal of Comparative Neurology & Psychology, 18(1), 459–482. https://doi.org/10.1002/cne.920180503
79.
YouS.RobertL. P.Jr. (2018). Human-robot similarity and willingness to work with a robotic co-worker. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 251–260. https://doi.org/10.1145/3171221.3171281
YoungM. S.StantonN. A. (2002). Malleable attentional resources theory: A new explanation for the effects of mental underload on performance. Human Factors, 44(3), 365–375. https://doi.org/10.1518/0018720024497709
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
