Overcoming Unreliability in Operational Measures: The Use of Surrogate Measure Systems

Abstract

Improved assessment of operational performance is critical for studies of selection, training and human engineering as well as those examining the performance effects of environmental changes, chemicals or other stressors imposed by military duties. The present discussion focuses on what we consider to be the major problem with such enterprises – the lack of sensitivity of operational measures because of poor reliability. The well-documented impact of low field measure reliability includes the inability to demonstrate differences resulting from experimental treatments and the chronic underrepresentation of validity in predictive studies. This paper describes a general methodology for using specially-developed performance batteries as surrogates for real-world performances, in particular for determining whether such performance may be disrupted by environmental or chemical agents. The logical and metric rationale of surrogate measurement is presented, and the advantages and disadvantages are discussed and compared to alternative approaches (job samples, synthetic tasks, etc.).

Get full access to this article

View all access options for this article.

References

Alluisi

E. A.

Fleishman

E. A.

(Eds.) (1982) Human performance and productivity: Vol. 3; Stress and performance effectiveness. Hillsdale, NJ: Erlbaum.

Biers

D. W.

Sauer

D. W.

(1982). Job sample tests as predictors of Ml gunnery performance (Appendixes A-E) (Report No. SRL-6639-1). Dayton, OH: Systems Research Labs, Inc. (NTIS No. AD A135879).

Cook

S. W.

(Ed.), (1947). Psychological research on radar observer training (Report No. 12, AAF AAP). Washington, DC: Government Printing Office.

Crawford

M. P.

Wollenberger

R. T.

Ward

L. B.

Brown

C. W.

Ghiselli

E. E.

(Eds.), (1947). Psychological research on operational training in the continental air forces (Report No. 16, AAF AAP). Washington, DC: Government Printing Office.

Englund

C. E.

Krueger

G. P.

(Eds.). (1985). Methodological approaches to the study of sustained work/sustained operations. Behavior Research Methods, Instruments and Computers, 17, 3–101. (Special Section).

Flanagan

J. C.

(Ed.), (1948). The aviation psychology program in the Army Air Forces (Report No. 1). Washington, DC: Government Printing Office.

General Accounting Office. (1982). The army needs to modify its system for measuring individual soldier proficiency (Report No. GAO/FPCD-82-28). Washington, DC: Federal Personnel and Compensation Division. (NTIS No. AD A112923).

Green

B. F.

Hall

J. A.

(1984). Quantitative methods for literature reviews. Annual Review of Psychology, 35, 37–58.

Guilford

J. P.

(1954). Psychometric Methods (2nd ed.). New York: McGraw Hill, 400–402.

10.

Headley

D. B.

(1985). Sustained operations in a chemical battlefield protective posture: The Army's P2 NBC2 field test experience. Presentation at 2nd meeting of the DOD HFE Subtag on Sustained/Continuous Operations, 3 October, 1985, Baltimore, Maryland.

11.

Jobe

J. B.

Bandaret

L. E.

(1986). Cognitive testing in military performance research. Proceedings of a Workshop on Cognitive Testing Methodology, National Research Council Committee on Military Nutrition Research, June 1984. Washington, DC: National Academy Press.

12.

Johnson

J. H.

Jones

M. B.

Kennedy

R. S.

(1984). A video test for the prediction of tank commander performance. Proceedings of the 28th Annual Meeting of the Human Factors Society. Santa Monica, CA: Human Factors Society, 502–503.

13.

Krueger

G. P.

Cardenelas-Ortiz

Loveless

C. A.

(1985, May). Human performance in continuous/sustained operations and the demands of extended work/rest schedules: An annotated bibliography. (WRAIR BB-85-1) Washington, DC: Walter Reed Army Institute of Research.

14.

Lane

N. E.

(1986). Issues in performance measurement for military aviation with applications to air combat maneuvering. (NTSC TR-86-008) Orlando, FL: Naval Training Systems Center.

15.

Lintern

Nelson

B. E.

Sheppard

D. J.

Westra

D. P.

Kennedy

R. S.

(1981). Visual technology research simulator (VTRS) human performance research: Phase III (NAVTRAEQUIPCEN 78-C-0060-11). Orlando, FL: Canyon Research Group.

16.

Miller

N. E.

(1947). Summary of main results and recommendations for future work. In Miller

N. E.

(Ed.), Psychological research on pilot training (AAF AAP Report No. 8) (pp. 353–380). Washington, DC: Government Printing Office.

17.

Mixon

T. R.

(1982). An annotated bibliography of objective pilot performance measures, Part II (Technical Report No. NPS55-81-010PR). Monterey, CA: Naval Post Graduate School.

18.

Schmidt

F. L.

Hunter

J. E.

(1981). Employment testing: Old theories and new research findings. American Psychologist, 36, 1128–1137.

19.

Wallace

S. R.

(1965). Criteria for what? American Psychologist, 84, 601–618.

20.

Westra

D. P.

Lintern

Sheppard

D. J.

Thomley

K. T.

Mauk

Wightman

D. C.

Chambers

(1986). Simulator design and instructional features for carrier landing: Transfer study. (NAVTRASYSCEN 85-C-0044-2.) Orlando, Fl: Naval Training Systems Center.

21.

Youtz

R. P.

(1947). Objective measures of flying skill for the primary level of training. In Miller

N. E.

(Ed.), Psychological research on pilot training (AAF AAP Report No. 8) (pp. 111–163). Washington DC: Government Printing Office.