The Use of Judgment Matrices in Subjective Workload Assessment: The Subjective Workload Dominance (SWORD) Technique

Abstract

One objective of the project was to determine compare two analytic algorithms for converting judgment matrices into subjective workload ratings. The original eigenvector algorithm used in Saaty's Analytic Hierarchy Process (AHP) was compared an algorithm of calculating geometric means. Also, three methods of identifying excessively inconsistent matrices were compared. Data from nine previous experiments were re-examined in the present analysis. There were no differences between the AHP ratings and the geometric mean ratings in terms of their sensitivity to the experimental manipulations. However, two of the inconsistency measures were successfully used to cull the data-sets of inconsistent matrices and improved the statistical sensitivity of one set of ratings. These findings suggest that: (1) the computationally simpler geometric means algorithm can be used as an alternative to the eigenvector algorithm, and (2) culling inconsistent matrices can sometimes improve rating sensitivity. These findings, along with previous research, demonstrate that judgment matrices can be a very valuable workload assessment tool. The essential steps for the proper use of judgment matrices in workload assessment are reviewed. A user's guide and software are also being prepared to aid researchers and practitioners

Get full access to this article

View all access options for this article.

References

Budescu

D. V.

Zwick

Rapoport

(1986). A comparison, of the eigenvalue method and the geometric means procedure for ratio scaling. Applied Psychological Measurement, 10, 69–78

Gopher

Braune

(1984). On the psychophysics of workload: Why bother with subjective measures? Human Factors, 26, 519–532

Gopher

Chillag

Arzi

(1985, February). The influence of voluntary effort, context, and anchor task, on the subjective estimate of load (Tech. Report FY- 84). Haifa, Israel: Research Center for Work Safety and Human Engineering

Hart

S. G.

Staveland

(1988). Development of NASATLX (Task Load Index): Results of empirical and theoretical research. In Hancock

P. A.

Meshkati

(Eds.), Human Mental Workload (pp. 139–183). Amsterdam, The Netherlands: Elsevier

Lidderdale

I. G.

(1987). Measurement of aircrew workload during low-level flight. In Roscoe

A. H.

(Ed.), AGARDograph No. 282 - The practical assessment of pilot workload (pp. 78–82). Lougton, United Kingdom: AGARD

Reid

Nygren

T. E.

(1988). The subjective workload assessment technique: A scaling procedure for measuring mental workload. In Hancock

P. A.

Meshkati

(Eds.), Human Mental Workload (pp. 185–218). Amsterdam, The Netherlands: Elsevier

Saaty

T. L.

(1980). The analytic hierarchy process. New York: McGraw-Hill

Vidulich

M. A.

Bortolussi

M. R.

(1988). A dissociation of objective and subjective workload measures in assessing the impact of speech controls in advanced helicopters. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 1471–1475). Santa Monica, CA: Human Factors Society

Vidulich

M. A.

Tsang

P. S.

(1987). Absolute magnitude estimation and relative judgment approaches to subjective workload assessment. In Proceedings of the Human Factors Society 31st Annual Meeting (pp. 1057–1061). Santa Monica, CA: Human Factors Society

10.

Vidulich

M. A.

Tsang

P. S.

(in press). Evaluating immediacy and redundancy in subjective workload techniques. In Proceedings of the Twenty-Third Annual Conference on Manual Control.

11.

Williams

Crawford

(1980, May). Analysis of Subjective Judgment Matrices (Tech. Report R-2572-AF). Santa Monica, CA: Rand