One of the most preferred and widely used methods for generating cutoff scores in content-related validity situations is the Angoff procedure. This procedure, however, typically displays low inter-rater reliability. Very little is known about the accuracy of the cutoff scores. The present study compares the effect of two frame-of-reference rater training approaches on the reliability and accuracy of cutoff scores generated by the Angoff procedure. Results indicate that both frame-of-reference rater training strategies resulted in significantly higher levels of interrater reliability and more accurate decisions than a non-frame-of-reference condition.
Get full access to this article
View all access options for this article.
References
1.
Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational Measurement. Washington, DC: American Council on Education.
2.
Arthur, W., Jr. , Woehr, D. J., and Fehrmann, M. L. (1990). Comparison of cutoff cores for content-related and criterion-related validity settings. Paper presented at the 98th Annual Convention of the American Psychological Association, Boston, MA.
3.
Athey, T. R. and McIntyre, R. M. (1987). Effect of rater training on rater accuracy: Level-of-processing theory and social facilitation theory perspectives. Journal of Applied Psychology, 72, 239-244.
4.
Barrett, G. V. (1989). Personnel selection after Watson, Hopkins, Atonio and Martin (WHAM). Paper presented at Personnel Testing Council of Metropolitan Washington Conference, Washington, D.C.
5.
Behuniak, P. , Archambault, F. X., and Gable, R. K. (1982). Angoff and Nedelsky standard setting procedures: Implications for the validity of proficiency test score interpretation. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 42, 247-255.
6.
Berk, R. A. (1986). A consumer's guide to setting performance standards on criterion-referenced tests. Review of Educational Research, 52, 137-172.
7.
Bernardin, H. J. and Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.
8.
Brennan, R. L. (1983). Elements of generalizability theory. Iowa City, IA: ACT Publications.
9.
Brennan, R. L. and Lockwood, R. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
10.
Cascio, W. F. , Alexander, R. A., and Barrett, G. V. (1988). Setting cutoff scores: Legal, psychometric, and professional issues and guidelines. Personnel Psychology, 41, 1-24.
11.
Cronbach, L. J. , Glesser, G. C., Nanda, H., and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
12.
Cross, L. H. , Impara, J. C., Frary, R. B., and Jaeger, R. M. (1984). A comparison of three methods for establishing minimum standards on the National Teachers Examinations. Journal of Educational Measurement, 21, 113-129.
13.
Ebel, R. L. (1972). Essentials of educational measurement. Engle-wood Cliffs, NJ: Prentice Hall.
14.
Fehrmann, M. L. (1989), Standard setting: Frame of reference training, generalizability training, and the Angoff method. Un- published master's thesis, Texas AandM University, Department of Psychology, College Station, TX.
15.
Glass, G. V. (1978). Standards and criterion. Journal of Educational Measurement, 15, 237-261.
16.
Halpin, G. , Sigmon, G., and Halpin, G. (1983). Minimum competency standards set by three divergent groups of raters using three judgmental procedures: Implications for validity. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 43, 185-196.
17.
Hambleton, R. K. and Eignor, D. R. (1979). Competency test development, validation, and standard setting. In R. Jaeger, and C. Tittle (Eds.), Minimum competency testing. Berkeley, CA: McCutchan.
18.
Hauenstein, N. M. A. and Foti, R. J. (1989). From laboratory to practice: Neglected issues in implementing frame-of-reference rater training. Personnel Psychology, 42, 359-378.
19.
Jaeger, R. M. (1979). Measurement consequences of selected standard-setting models. In. M. A. Bunda and J. R. Sanders (Eds.), Practices and problems in competency-based measurement. Washington, DC: National Council on Measurement in Education.
20.
Maurer, T. J. , Alexander, R. A., Bailey, J. J., and Callahan, C. M. (1990). Judgmental methodology for setting cutoff scores: Expert vs. non-expert judges. Paper presented at the 5th annual meeting of the Society of Industrial/Organizational Psychologists, Miami, FL.
21.
McIntyre, R. , Smith, D., and Hasset, C. (1984). Accuracy of performance ratings as affected by rater training and perceived purpose of rating. Journal of Applied Psychology, 69, 147-156.
22.
Meskauskas, J. A. (1976). Evaluation models for criterion-referenced testing: Views regarding mastery and standard setting. Review of Educational Research, 45, 133-158.
23.
Millman, J. (1973). Passing scores and test lengths for domain-referenced measures. Review of Educational Research, 43, 205-216.
24.
Nedelsky, L. (1954). Absolute grading standards for objective tests. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 14, 3-19.
25.
Pulakos, E. D. (1986). The development of training programs to increase accuracy with difficult rating tasks. Organizational Behavior and Human Decision Processes, 38, 76-91.
26.
Pulakos, E. D. (1984). A comparison of rater training programs: Error training and accuracy training. Journal of Applied Psychology, 69, 581-588.
27.
Shepard, L. A. (1980a). Technical issues in minimum competency testing. In D. C. Berliner (Ed.), Review of research in education (Vol.8). Itasca, IL: F. E. Peacock Publishers.
28.
Shepard, L. A. (1980b). Standard setting issues and methods. Applied Psychological Measurement, 4, 447-467.
29.
Srull, T. K. and Wyer, R. S. (1989). Person memory and judgment. Psychological Review, 96, 58-83.
30.
Van der Linden, W. J. (1982). A latent trait method for determining intra-judge inconsistency in the Angoff and Nedelsky techniques of standard setting. Journal of Educational Measurement, 19, 295-308.
31.
Van der Linden, W. J. (1984). Some thoughts on the use of decision theory to set cutoff scores: Comment on de Gruijter and Hambleton. Applied Psychological Measurement, 8, 9-17.
32.
Wyer, R. S. and Srull, T. K. (1986). Human cognition in its social content. Psychological Review, 93, 322-359.
