This paper examines fundamental measurement issues that have largely been ignored in scaling techniques for assessing perceived mental workload. It is argued that both theoretical and practical advances in workload assessment have been unnecessarily slowed by a lack of concern for measurement issues. Typical of this impedance are some recently reported “empirical results“ comparing various workload assessment techniques which actually follow directly from some misunderstood statistical properties of the scales. The commonly used subjective workload assessment procedures have distinct psychometric properties that make each procedure sensitive to different kinds of workload measurement. Examination of the properties for the National Aeronautics and Space Administration Task Load Index (NASA TLX) support its potential as a general prediction model for experienced workload and for the Subjective Workload Assessment Technique (SWAT) as a cognitive model sensitive to individual differences.
Get full access to this article
View all access options for this article.
References
1.
Battiste, V., and Bortolussi, M. R. (1988). Transport pilot workload: A comparison of two subjective techniques. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 150–154). Santa Monica, CA: Human Factors Society.
2.
Birnbaum, M. H. (1973). The devil rides again: Correlation as an index of fit. Psychological Bulletin, 79, 239–242.
3.
Byers, J. C., Bittner, A. C., Jr., and Hill, S. G. (1989). Traditional and raw task load index (TLX) correlations: Are paired comparisons necessary? In A. Mital (Ed.), Advances in industrial ergonomics and safety (Vol. 1). London: Taylor & Francis.
4.
Casper, P. A., Shively, R. J., and Hart, S. G. (1987). Decision support for workload assessment: Introducing WC FIELDE. In Proceedings of the Human Factors Society 31st Annual Meeting (pp. 72–76). Santa Monica, CA: Human Factors Society.
5.
Cooper, G. E., and Harper, R. P. (1969). The use of pilot ratings in the evaluation of aircraft handling qualities (NASA tech. Report TN 0–5153). Washington, DC: National Aeronautics and Space Administration.
6.
Dawes, R. M., and Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95–106.
7.
Gopher, D., and Braune, R. (1984). On the psychophysics of workload: Why bother with subjective measures?. Human Factors, 26, 519–532.
8.
Gopher, D., and Donchin, E. (1986). Workload—An examination of the concept. In K. Boff, L. Kauffman, and J. P. Thomas (Eds.), Handbook of perception and human performance (pp. 41. 1–41.9). New York: Wiley.
9.
Hart, S. G. (1986). Theory and measurement of human workload. In J. Zeidner (Ed.), Human productivity enhancement: Training and human factors in systems design (Vol. 1, pp. 396–456). New York: Praeger.
10.
Hart, S. G., and Staveland, L. (1988). Development of the NASA task load index (TLX): Results of empirical and theoretical research. In P. A. Hancock and N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North-Holland.
11.
Haworth, L. A., Atencio, A., Bivens, C. C., Shively, R. J., and Delgado, D. (1987). Advanced helicopter cockpit and control configurations for helicopter combat missions (NASA Tech. Manual 100017). Washington, DC: National Aeronautics and Space Administration.
12.
Haworth, L. A., Bivens, C. C., and Shively, R. J. (1986). An investigation of single-piloted advanced cockpit and control configurations for nap-of-the-earth helicopter combat mission tasks. In Proceedings of the 1986 Meeting of the American Helicopter Society (pp. 657–672). Washington. DC: American Helicopter Society.
13.
Hoffman, P. J. (1960). The paramorphic representation of clinical judgment. Psychological Bulletin, 57, 116–131.
14.
Lysaght, R. J., Hill, S. G., Dick, A. O., Plamondon, B. D., Linton, P. M., Wierwille, W. W., Zaklad, A. L., Bittner, A. C., Jr., and Wherry, R. J. (1989). Operator workload: Comprehensive review and evaluation of operator workload methodologies (ARI Tech. Report 851). Alexandria, VA: Army Research Institute.
15.
Meehl, P. E. (1954). Clinical vs. statistical prediction: A theoretical analysis and review of the literature. Minneapolis: University of Minnesota Press.
16.
National Aeronautics and Space Administration. (1989). WC FIELDE: Workload consultant for field evaluation [computer software]. Moffett Field. CA: NASA Ames Research Center.
17.
Nickerson, C. A., and McClelland, G. B. (1984). Scaling distortion in numerical conjoint measurement. Applied Psychological Measurement, 8, 183–198.
18.
Nygren, T. E. (1985a). Axiomatic and numeric conjoint measurement: A comparison of three methods for obtaining subjective workload (SWAT) rankings. In Proceedings of the IEEE 1985 National Aerospace and Electronics Conference (pp. 878–883). New York: Institute of Electrical and Electronics Engineers.
19.
Nygren, T. E. (1985b). An examination of conditional violations of axioms for additive conjoint measurement. Applied Psychological Measurement, 9, 249–264.
20.
Nygren, T. E. (1986). A two-stage algorithm for assessing violations of additivity via axiomatic and numerical conjoint measurement. Psychometrika, 51, 483–491.
21.
Nygren, T. E. (1990, May). Assessing individual differences in perceived mental workload. Paper presented at the 62nd Annual Meeting of the Midwestern Psychological Association, Chicago, IL.
22.
Nygren, T. E., and Lee, S. (1987, August). Minimal paired comparisons for subject prototypes in conjoint scaling. Paper presented at the 20th Annual Meeting of the Society for Mathematical Psychology, University of California, Berkeley.
23.
O'Donnell, R. D., and Eggemeier, F. T. (1987). Workload assessment methodology. In K. Boff, L. Kauffman, and J. P. Thomas (Eds.), Handbook of perception and human performance (pp. 42. 1–42. 49). New York: Wiley.
24.
Reid, G. B. (1985). The systematic development of a subjective measure of workload. In Proceedings of the 9th Congress of the International Ergonomics Association (pp. 109–111). London: Taylor & Francis.
25.
Reid, G. B., and Nygren, T. E. (1988). The subjective workload assessment technique: A scaling procedure for measuring mental workload. In P. A. Hancock and N. Meshkati (Eds.), Human mental workload (pp. 185–218). New York: North-Holland.
26.
Reid, G. B., Shingledecker, C. A., Nygren, T. E., and Eggemeier, F. T. (1981), The development of multidimensional subjective measures of workload. In Proceedings of the IEEE International Conference on Cybernetics and Society (pp. 403–406). New York: Institute of Electrical and Electronics Engineers.
27.
Sheridan, T. B., and Simpson, R. W. (1979). Toward the definition and measurement of the mental workload of transport pilots (Tech. Report R79–4). Cambridge: Massachusetts Institute of Technology, Flight Transportation Laboratory.
28.
Tein, J.-Y., and Nygren, T. E. (1988, July). Additivity and the resulting conjoint scaling solution. Paper presented at the 21st Annual Meeting of the Society for Mathematical Psychology, Northwestern University, Evanston, IL.
29.
Vidulich, M. A., and Tsang, P. S. (1986). Techniques of subjective workload assessment: A comparison of SWAT and the NASA-bipolar methods. Ergonomics, 29, 1385–1398.
30.
Wierwille, W. W., Casali, J. G., Connor, S. A., and Rahimi, M. (1986). Evaluation of the sensitivity and intrusion of mental workload estimation techniques. In W. Roner (Ed.), Advances in man-machine systems research (Vol. 2, pp. 51–127). Greenwich, CT: JAI Press.
