Mental Models of Metal Models: A Comparison of Mental Model Measurement Techniques

Abstract

Although use of the mental model construct has proliferated in recent research, the construct lacks a clear definition and an agreed upon method of measurement. Furthermore, the reliability and validity of the different measurement techniques in use have not been established, thereby making generalizations across studies of mental models difficult. The purpose of the current project was to assess several methods of measuring mental models in terms of their reliability/stability over time. Subjects” mental models of the automobile engine system were elicited on two occasions separated by one week, using seven different knowledge elicitation techniques. Subjects” level of experience was also measured to allow comparisons between experts and novices. The results indicate that each of the measurement techniques tended to be reliable for both experts and novices. However, reliability tended to be greater for experts than novices. Additionally, experts tended to agree with each other more than did the novices. Some evidence also indicated that the results from the similarity ratings and subsequent Pathfinder analysis converged with those from the structured interviews.

Get full access to this article

View all access options for this article.

References

Barsalou

L.M.

(1989). Intraconcept similarity and its implications for interconcept similarity. In Vosniadov

Ortony

(Eds.), Similarity and Analogical Reasoning (pp. 76–121). Cambridge: Cambridge University Press.

Carroll

J.M.

Olson

J.R.

(1983). Mental Models in Human-Computer Interaction. In Helander

(Ed.), Handbook of Human-Computer Interaction (pp. 45–65). North-Holland.

Cooke

N.J.

Schvaneveldt

(1988). Effects of computer programming experience on network representations of abstract programming concepts. International Journal of Man-Machine Studies, 29, 407–427.

Gentner

Stevens

A.L.

(Eds.). (1983). Mental Models. Hillsdale, NJ: Lawrence Erlbaum Assoc.

Goldsmith

T.E.

Davenport

D.M.

(1990). Assessing structural similarity of graphs. In Schvaneveldt

, (Ed.), Pathfinder Associative Networks: Studies in Knowledge Organization (pp. 75–87). Norwood, NJ: Ablex.

Halasz

F.G.

Moran

T.P.

(1983). Mental models and problem solving in using a calculator. In Janda

(Ed.), Human factors in computing systems: Proceedings of CHI 1983 conference (pp. 212–216). New York: Association for Computing Machinery.

Hegarty

(1991). Knowledge and processes in mechanical problem solving. In Sternberg

R.J.

Frensch

P.A.

(Eds.), Complex Problem Solving: Principles and Mechanisms (pp. 253–285). Hillsdale, NJ: Lawrence Erlbaum Assoc.

Norman

D.A.

(1983). Some observations on mental models. In Gentner

Stevens

A.L.

(Eds.), Mental Models (pp. 7–14). Hillsdale, NJ: Lawrence Erlbaum Assoc.

Rouse

W.B.

Morris

N.M.

(1986). On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin, 100, 349–363.

10.

Schvaneveldt

(Ed.). (1990). Pathfinder Associative Networks: Studies in Knowledge Organization. Norwood, NJ: Ablex.

11.

Wilson

J.R.

Rutherford

(1989). Mental models: Theory and application in human factors. Human Factors, 31, 617–634.

12.

Young

R.M.

(1981). The machine inside the machine: User's models of pocket calculators. International Journal of Man-Machine Studies, 15, 51–85.