Despite the comprehensive treatment of test validity in most technical manuals, test authors appear to routinely assume that clients and professionals will score their instruments without error. Recently Allard and Faust challenged this assumption by suggesting that error rates “may not be rare or benign” and demonstrated that tests with more complex scoring procedures were associated with a greater number of scoring errors. This study investigated error rates that resulted from hand scoring seven psychometric tests commonly employed in psychological practice. Significant and serious error rates were identified for both psychologist and client scorers across all tests investigated. Scoring complexity was found to predict the base rate of scorer errors. The findings suggest that greater development in and attention to test-scoring procedures is required to restrict the likelihood of scorer error.
Allard, G.
, Butler, J., Shea, M. T., & Faust, D. (1995). Errors in hand scoring objective personality tests: The case of the Personality Diagnostic Questionnaire-Revised (PDQ-R). Professional Psychology: Research and practice, 26(3), 304-308.
2.
Allard, G.
, & Faust, D. (2000). Errors in scoring objective personality tests. Assessment, 7(2), 119-131.
3.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). 1999 standards for educational and psychological testing. Retrieved from http://www.apa.org/science/standards.html#overview
4.
Australian Council of Educational Research
. (1981). ACER Higher Test manual Form ML-MQ. Canberra, Australia: Author.
5.
Australian Psychological Society Ltd. (1997). Guidelines for the use of psychological tests. Retrieved from http://www.psychsociety.com.au/about/testing.pdf
6.
Bass, B. M.
, & Avolio, B. J. (1995). Multifactor leadership questionnaire: Manual leader form, rater, and scoring key for MLQ (Form 5x-Short). Redwood City, CA: Mind Garden.
7.
Beck, A. T.
, Steer, R. A., & Brown, G. K. (1996). Beck Depression Inventory (2nd ed.). San Antonio, TX: Harcourt Brace.
8.
Beck, A. T.
, Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561-571.
9.
Bickham, P. J.
, Miller, M. J., O’Neal, H., & Clanton, R. (1998). Comparison of error rates on the 1990 and 1994 revised Self-Directed Search. Perceptual and Motor Skills, 86, 1168-1170.
10.
Canadian Psychological Association
. (1996). Guidelines for educational and psychological testing. Retrieved from http://www.cpa.ca/guide9.html
11.
Charter, R. A.
, Walden, D. K., & Padilla, S. P. (2000). Too many simple scoring errors: The Rey Figure as an example. Journal of Clinical Psychology, 56(4), 571-574.
12.
Christensen, K. C.
, Gelso, C. J., Williams, R. O., & Sedlacek, W. E. (1975). Variations in the administration of the Self-Directed Search, scoring accuracy and satisfaction with results. Journal of Counseling Psychology, 22(1), 12-16.
13.
Connor, R.
, & Woodall, F. E. (1983). The effects of experience and structured feedback on WISC-R error rates made by student-examiners. Psychology in the Schools, 20, 376-379.
14.
Cummings, R. W.
, & Maddux, C. D. (1987). Self-administration and scoring errors of learning disabled and non-learning disabled students on two forms of the Self-Directed Search. Journal of Counseling Psychology, 34(1), 83-85.
15.
Franklin, M. R.
, Stillman, P. L., Burpeau, M. Y., & Sabers, D. L. (1982). Examiner error in intelligence testing: Are you a source?Psychology in the Schools, 19, 563-569.
16.
Goddard, R. C.
, Patton, W., & Creed, P. (2000). Case manager burnout in the Australian job network. Journal of Applied Health Behaviour, 2(2), 1-6.
17.
Goddard, R. C.
, Patton, W., & Creed, P. (2001). Psychological distress in Australian case managers working with the unemployed. Journal of Employment Counseling, 38, 50-61.
18.
Groth-Marnat, G.
(1990). The handbook of psychological assessment (2nd ed.). New York: John Wiley.
19.
Holland, J. L.
, Powell, A. B., & Fritzsche, B. A. (1994). The Self-Directed Search (SDS) professional user’s guide 1994 edition. Odessa, FL: Psychological Assessment Resources.
20.
Levenson, R. L.
, Golden-Scaduto, C. J., Aiosa-Karpas, C. J., & Ward, A. W. (1988). Effects of examiners’education and sex on presence and type of clerical errors made on WISC-R protocols. Psychological Reports, 62, 659-664.
21.
Lin, H. X.
, & Salvendy, G. (1999). Instruction effect on human error reduction. International Journal of Cognitive Ergonomics, 3(2),115-129.
22.
McGrew, K. S.
, Murphy, S. R., & Knutson, D. J. (1994). The development and investigation of a graphic scoring system for obtaining derived scores for the WJ-R and other tests. Journal of Psychoeducational Assessment, 72, 33-41.
23.
Miller, C. K.
, Chansky, N. M., & Gredler, G. R. (1970). Rater agreement on WISC protocols. Psychology in the Schools, 7, 190-193.
24.
Miller, K. M.
, Tyler, B., & Rothwell, J. W. (1994). Rothwell Miller Interest Blank (Revised). London: Miller & Tyler.
25.
Miller, M. J.
(1997). Error rates on two forms of the Self-Directed Search and satisfaction with results. Journal of Employment Counseling, 34, 98-103.
26.
Myers, I. B.
, & McCaulley, M. H. (1985). Manual: A guide to the development and use of the Myers-Briggs Type Indicator. Palo Alto, CA: Consulting Psychologists Press.
27.
Myers, I. B.
, McCaulley, M. H., Quenk, N. L., & Hammer, A. L. (1988). MBTI manual: A guide to the development and use of the Myers Briggs Type Indicator (3rd ed.). Palo Alto, CA: Consulting Psychologists Press.
28.
Pryor, R. G. L.
(1995). Vocational Interest Survey for Australia (VISA): Professional manual. Sydney, Australia: Psychological Corporation.
29.
Quinn, R. E.
(1988). Beyond rational management: Mastering the paradoxes and competing demands of high performance. San Francisco: Jossey-Bass.
30.
Reason, J. T.
(1990). Human error. Cambridge, MA: Cambridge University Press.
31.
Sherrets, S.
, Gard, G., & Langner, H. (1979). Frequency of clerical errors on WISC protocols. Psychology in the Schools, 16(4), 495-496.
32.
Slate, J. R.
, & Chick, D. (1989). WISC-R examiner errors: Cause for concern. Psychology in the Schools, 26, 78-84.
33.
Sullivan, K.
(2000). Examiners’ error on the Wechsler Memory Scale–Revised. Psychological Reports, 87, 234-240.
34.
Tracey, T. J.
, & Sedlacek, W. E. (1980). Comparison of error rates on the original Self-Directed Search and the 1977 revision. Journal of Counseling Psychology, 27(3), 299-301.
35.
Whitten, J.
, Slate, J. R., Shine, A. E., & Raggio, D. (1994). Examiner errors in administering and scoring the WPPSI-R. Journal of Psychoeducational Assessment, 12, 49-54.
36.
Woltz, D. J.
, & Gardner, M. K. (2000). Negative transfer errors in sequential cognitive skills: Strong-but-wrong sequence application. Journal of Experimental Psychology, 26(3), 601-625.
