Mental models are characterized as a person’s mental representation of how something works, guiding how they process information, anticipate future events, and interact with devices and tools in complex sociotechnical systems. This paper introduces the Mental Model Matrix (MMM), a novel framework for conceptualizing mental models in the context of human-system interactions. The MMM framework partitions aspects of an individual’s knowledge about a target system into two primary dimensions (system and user), each containing beliefs regarding capabilities and limitations. The system-based dimension includes knowledge for the system’s parts, connections, and causal relationships (capabilities), as well as knowledge for how the system can fail (limitations). The user-based dimension incorporates tacit knowledge for working with the system (capabilities) along with an appreciation for one’s own confusions and misunderstandings (limitations). The MMM provides a conceptual framework and guidance for end-users, trainers, and system designers seeking to elicit and codify different dimensions of an individual’s mental model of the systems they work with. The MMM can be used to reveal knowledge gaps and misalignments between different stakeholders in an organization, which can facilitate the development of human-centered technologies and systems, and the creation of effective training programs.
BartlettF. C. (1932). Remembering: A study in experimental and social psychology. Cambridge University Press.
2.
BordersJ.KleinG.BesuijenR. (2019, June 21-24). An operational account of mental models: A pilot study. [Conference presentation]. International Naturalistic Decision Making Conference,San Francisco, CA, United States.
3.
BransfordJ. D.BrownA. L.CockingR. R. (2000). How people learn. Brain, mind, experience, and school. National Academy Press.
4.
Cannon-BowersJ. A.SalasE.ConverseS. (1993). Shared mental models in expert team decision making. In CastellanN. J. (Ed.), Individual and group decision making: Current issues (pp. 221–246). Lawrence Erlbaum Associates.
ChiM. T. H.WylieR. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243. https://doi.org/10.1080/00461520.2014.965823
7.
ChinnC. A.BrewerW. F. (1993). The role of anomalous data in knowledge acquisition: A theoretical framework and implications for science instruction. Review of Educational Research, 63(1), 1–49. https://doi.org/10.3102/00346543063001001
8.
CookeN. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41(6), 801–849. https://doi.org/10.1006/ijhc.1994.1083
9.
CookeN. J. (1999). Knowledge elicitation. In DursoF. T. (Ed.), Handbook of applied cognition (pp. 479–509). Wiley.
10.
CookeN. J.GormanJ. C.MyersC. W.DuranJ. L. (2013). Interactive team cognition. Cognitive Science, 37(2), 255–285. https://doi.org/10.1111/cogs.12009
11.
CookeN. M. (1992). Modeling human expertise in expert systems. In HoffmanR. R. (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 29–60). Springer Verlag.
12.
CraikK. J. W. (1943). The nature of explanation. Cambridge University Press.
13.
CrandallB.KleinG.HoffmanR. R. (2006). Working minds. A practitioners guide to cognitive task analysis, 2006. MIT Press.
14.
De KeyserV.WoodsD. D. (1990). Fixation errors: Failures to revise situation assessment in dynamic and risky systems. In ColomboA. G.Saiz de BustamanteA. (Eds.), System reliability assessment (pp. 231–251). Kluwer Academic.
15.
DixonR. A.JohnsonS. D. (2011). Expert vs. novices: Differences in how mental representations are used in engineering design. Journal of Technology Education, 23(1), 47–65.
16.
DominguezC. O. (2001). Expertise and metacognition in laparoscopic surgery: A field study. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 45(17), 1298–1302. Sage Publications.
17.
DrayA.PerezP.Le PageC.D’AquinoP.WhiteI.AuatabuT. (2006). The AtollGame experience: From knowledge engineering to a computer-assisted role playing game. Simulation and Gaming, 38(4), 494–511. https://doi.org/10.1177/1046878107300673
18.
DreyfusS. E.DreyfusH. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition. Operations research center: University of California.
19.
EndsleyM. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), 32–64. https://doi.org/10.1518/001872095779049543
FeltovichP. J.CoulsonR. L.SpiroR. J. (2001). Learners’ (mis)understanding of important and difficult concepts: A challenge to smart machines in education. In ForbusK.FeltovichP. J. (Eds.), Smart machines in education (pp. 349–375). MIT Press.
22.
FeltovichP. J.HoffmanR. R.WoodsD.RoeslerA. (2004). Keeping it too simple: How the reductive tendency affects cognitive engineering. IEEE Intelligent Systems, 19(3), 90–94. https://doi.org/10.1109/mis.2004.14
23.
FleurD. S.BredewegB.van den BosW. (2021). Metacognition: Ideas and insights from neuro-and educational sciences. NPJ Science of Learning, 6(1), 13. https://doi.org/10.1038/s41539-021-00089-5
GentnerD.StevensA. L. (2014). Mental models. Psychology Press.
26.
HoffmanR. R.WardP.FeltovichP. J.DiBelloL.FioreS. M.AndrewsD. H. (2013). Accelerated expertise: Training for high proficiency in a complex world. Psychology Press.
27.
HollandJ. H.HolyoakK. J.NisbettR. E.ThagardP. R. (1986). Induction: Processes of inference, learning, and discovery. MIT Press.
Johnson-LairdP. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Harvard University Press.
30.
JonesN. A.RossH.LynamT.PerezP.LeitchA. (2011). Mental models: An interdisciplinary synthesis of theory and methods. Ecology and Society, 16(1), 46. https://doi.org/10.5751/es-03802-160146
31.
KierasD. E.BovairS. (1986). The acquisition of procedures from text: A production-system analysis of transfer of training. Journal of Memory and Language, 25(5), 507–524. https://doi.org/10.1016/0749-596x(86)90008-2
32.
KleinG. (2015). A naturalistic decision making perspective on studying intuitive decision making. Journal of Applied Research in Memory and Cognition, 4(3), 164–168.
33.
KleinG.BaxterH. C. (2009). Cognitive transformation theory: Contrasting cognitive and behavioral learning. In CohnJ. V.SchmorrowD.NicholsonD. (Eds.), The PSI handbook of virtual environments for training and education: Developments for the military and beyond. Vol 1: Learning, requirements, and metrics (1, pp. 50–64). Praeger Security International.
34.
KleinG.BordersJ. (2016). The ShadowBox approach to cognitive skills training: An empirical evaluation. Journal of Cognitive Engineering and Decision Making, 10(3), 268–280. https://doi.org/10.1177/1555343416636515
35.
KleinG.BordersJ.NewsomeE.MilitelloL.KleinH. A. (2018). Cognitive skills training: Lessons learned. Cognition, Technology and Work, 20(4), 681–687. https://doi.org/10.1007/s10111-018-0528-5
36.
KleinG.FeltovichP. J.BradshawJ. M.WoodsD. D. (2005). Common ground and coordination in joint activity. In RouseW. B.KennethR. B. (Eds.), Organizational Simulation (pp. 139–184). John Wiley & Sons.
37.
KleinG.HoffmanR. R. (2008). Macrocognition, mental models, and cognitive task analysis methodology. In SchraagenJ. A.MilitelloL. G.OrmerodT.LipshitzR. (Eds.), Naturalistic decision making and macrocognition (pp. 57–80). Ashgate Publishing Limited.
38.
KleinG.MoonB.HoffmanR. R. (2006a). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70–73. https://doi.org/10.1109/mis.2006.75
39.
KleinG.MoonB.HoffmanR. R. (2006b). Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent Systems, 21(5), 88–92. https://doi.org/10.1109/mis.2006.100
40.
KleinG. A. (1993). A recognition-primed decision (RPD). Model of rapid decision making. Decision Making in Action: Models and Methods, 5(4), 138–147.
41.
KleinG. A.CalderwoodR.MacgregorD. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19(3), 462–472. https://doi.org/10.1109/21.31053
42.
KleinG. A.PhillipsJ. K.RallE. L.PelusoD. A. (2007). A data-frame theory of sensemaking. In HoffmanR. T. (Ed.), Expertise out of context: Proceedings of the 6th international conference on naturalistic decision making (pp. 113–156). Psychology Press.
43.
LiuA. L.NesbitJ. C. (2020). Dashboards for computer-supported collaborative learning. In VirvouM.AlepisE.TsihrintzisG. A.JainL. C. (Eds.), Machine learning paradigms: Advances in learning analytics (pp. 157–182). Springer Link.
44.
LiuD.MacchiarellaN. D.VincenziD. A. (2008). Simulation fidelity. In VincenziD. A.WiseJ. A.MoulouaM.HancockP. A. (Eds.), Human factors in simulation and training (pp. 61–73). CRC Press.
45.
LoftS.SadlerA.BraithwaiteJ.HufS. (2015). The chronic detrimental impact of interruptions in a simulated submarine track management task. Human Factors, 57(8), 1417–1426. https://doi.org/10.1177/0018720815599518
46.
MacIntyreT. E.IgouE. R.CampbellM. J.MoranA. P.MatthewsJ. (2014). Metacognition and action: A new pathway to understanding social and cognitive aspects of expertise in sport. Frontiers in Psychology, 5(1155), 1155. https://doi.org/10.3389/fpsyg.2014.01155
47.
MacKenzieC.HolmstromD.KaszniakM. (2007). Human factors analysis of the BP Texas City refinery explosion. Proceedings of the Human Factors and Ergonomics Society, 51(20), 1444–1448.
48.
McKeownM. G.BeckI. L. (2009). The role of metacognition in understanding and supporting reading comprehension. In HackerD. J.DunloskyJ.GraesserA. C. (Eds.), Handbook of metacognition in education (pp. 19–37). Routledge.
49.
MilitelloL. G.HuttonR. J. (1998). Applied cognitive task analysis (ACTA): A practitioner’s toolkit for understanding cognitive task demands. Ergonomics, 41(11), 1618–1641. https://doi.org/10.1080/001401398186108
50.
MillerJ. E.PattersonE. S.WoodsD. D. (2006). Elicitation by critiquing as a cognitive task analysis methodology. Cognition, Technology and Work, 8(2), 90–102. https://doi.org/10.1007/s10111-005-0023-7
MuellerS. T.TanY.LinjaA.KleinG.HoffmanR. (2021). Authoring guide for cognitive tutorials for artificial intelligence: Purposes and methods. PsyArXiv. https://doi.org/10.31234/osf.io/g7umc
54.
MumawR. J.RothE. M.VicenteK. J.BurnsC. M. (2000). There is more to monitoring a nuclear power plant than meets the eye. Human Factors, 42(1), 36–55. https://doi.org/10.1518/001872000779656651
55.
NewtonO. B.WiltshireT. J.FioreS. M. (2018). Macrocognition in teams and metacognition: Developing instructional strategies for complex collaborative problem solving. In JohnstonJ.SottilareR.SinatraA. M.BurkeC. S. (Eds.), Building intelligent tutoring systems for teams (pp. 33–54). Emerald Publishing Limited.
56.
NormanD. A.DraperS. W. (1986). User-centered system design: New perspectives on human-computer interaction. Lawrence Erlbaum Associates.
57.
PattersonR. E.PierceB. J.BellH. H.KleinG. (2010). Implicit learning, tacit knowledge, expertise development, and naturalistic decision making. Journal of Cognitive Engineering and Decision Making, 4(4), 289–303. https://doi.org/10.1177/155534341000400403
58.
PolanyiM. (1967). The tacit dimension. Doubleday.
59.
RasmussenJ. (1979). On the structure of knowledge-a morphology of mental models in a man-machine context. Riso National Laboratory.
60.
RasmussenJ.RouseW. B. (1981). Human detection and diagnosis of system failures. Plenum Press.
61.
RouseW. B.MorrisN. M. (1986). On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin, 100(3), 349–363. https://doi.org/10.1037/0033-2909.100.3.349
62.
RoweA. L.CookeN. J. (1995). Measuring mental models: Choosing the right tools for the job. Human Resource Development Quarterly, 6(3), 243–255. https://doi.org/10.1002/hrdq.3920060303
63.
RoweA. L.CookeN. J.NevilleK. J.SchachererC. W. (1992). Mental models of metal models: A comparison of mental model measurement techniques. Proceedings of the Human Factors Society Annual Meeting, 36(16), 1195–1199. https://doi.org/10.1177/154193129203601603
64.
SarterN. B.WoodsD. D. (1991). Situation awareness: A critical but ill-defined phenomenon. The International Journal of Aviation Psychology, 1(1), 45–57.
65.
SarterN. B.WoodsD. D. (1994). Pilot interaction with cockpit automation II: An experimental study of pilots’ model and awareness of the flight management system. The International Journal of Aviation Psychology, 4(1), 1–28. https://doi.org/10.1207/s15327108ijap0401_1
66.
SarterN. B.WoodsD. D. (1995). How in the world did we ever get into that mode? Mode error and awareness in supervisory control. Human Factors, 37(1), 5–19.
67.
SchmittJ. (1994). Mastering tactics: Tactical decision game workbook. Marine Corps Association.
68.
SchmittJ. F.KleinG. (1996). Fighting in the fog: Dealing with battlefield uncertainty. Marine Corps Gazette, 80(10), 62–69.
69.
TverskyB. (2019). Mind in motion: How action shapes thought. Basic Books.
70.
VanLehnK.BallW.KowalskiB. (1990). Explanation-based learning of correctness: Towards a model of the self-explanation effect. In Piatelli-PalmariniM. (Ed.), Proceedings of the twelfth annual conference of the cognitive science society (pp. 717–724). Lawrence Erlbaum Associates.
71.
VeldhuyzenW.StassenH. G. (1977). The internal model concept: An application to modeling human control of large ships. Human Factors: The Journal of the Human Factors and Ergonomics Society, 19(4), 367–380. https://doi.org/10.1177/001872087701900405
72.
WilsonJ. R.RutherfordA. (1989). Mental models: Theory and application in human factors. Human Factors, 31(6), 617–634.
73.
WinneP. H.AzevedoR. (2014). Metacognition. In SawyerR. K. (Ed.), The Cambridge handbook of the learning sciences (pp. 63–87). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.006
74.
WinneP. H.PerryN. E. (2000). Measuring self-regulated learning. In BoekaertsM.PintrichP. R.ZeidnerM. (Eds.), Handbook of self-regulation (pp. 531–559). Academic Press.
75.
WoodsD. D. (1993a). Process tracing methods for the study of cognition outside of the experimental psychology laboratory. In KleinG. A.OrasanuJ.CalderwoodR. (Eds.), Decision making in action: Models and methods (pp. 228–251). Ablex.
