Although the Angoff method is perhaps the most highly regarded and widely used judgmental method for setting criterion-referenced cutoff scores, little research has been conducted to determine the optimal number of raters to employ for obtaining stable cutoff scores. In the present study, Angoff ratings obtained from eight different occupational licensing examinations were evaluated via generalizability theory to estimate the optimal number of raters. Results indicated that approximately 10 to 15 raters is an optimal target range, although fewer raters may sometimes be sufficient. It is recommended that 10 to 15 raters be sampled as a general rule, followed by analyses to determine the optimal number of raters for examinations in a particular profession using a particular procedural variant of the Angoff method.
Get full access to this article
View all access options for this article.
References
1.
Berk, R. A. (1986). A consumer’s guide to setting performance standards on criterion-referenced tests. Review of Educational Research, 56, 137-172.
2.
Brennan, R. L. (1992). Elements of generalizability theory. Iowa City, IA: ACT.
3.
Brennan, R. L. , Gao, X., & Colton, D. A. (1995). Generalizability analyses of work keys listening and writing tests. Educational and Psychological Measurement, 55, 157-176.
4.
Brennan, R. L. , & Lockwood, R. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
5.
Busch, J. C. , & Jaeger, R. M. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the national teacher examinations. Journal of Educational Measurement, 27, 145-163.
6.
Cascio, W. F. , Alexander, R. A., & Barrett, G. V. (1988). Setting cutoff scores: Legal, psychometric, and professional issues and guidelines. Personnel Psychology, 41, 1-24.
7.
Cronbach, L. J. , Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley.
8.
Cross, L. H. , Impara, J. C., Frary, R. B., & Jaeger, R. M. (1984). A comparison of three methods for establishing minimum standards on the National Teacher Examinations. Journal of Educational Measurement, 21, 113-129.
9.
Fehrmann, M. L. , & Woehr, D. J. (1990, April). Standard setting: The impact of rater training. Paper presented at the annual convention of the Society for Industrial and Organizational Psychology, Miami Beach, FL.
10.
Fehrmann, M. L. , Woehr, D. J., & Arthur, W., Jr. (1991). The Angoff cutoff score method: The impact of frame-of-reference rater training. Educational and Psychological Measurement, 51, 857-872.
11.
Jarjoura, D. , & Brennan, R. L. (1983). Multivariate generalizability models for tests developed from tables of specifications. In L. J. Fyans (Ed.), Generalizability theory: Inferences and practical applications. San Francisco: Jossey-Bass.
12.
Kane, M. T. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
13.
Livingston, S. A. , & Zieky, M. J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.
14.
Maurer, T. J. , & Alexander, R. A. (1992). Methods of improving employment test critical scores derived by judging test content: A review and critique. Personnel Psychology, 45, 727-762.
15.
Maurer, T. J. , Alexander, R. A., Callahan, C. M., Bailey, J. J., & Dambrot, F. H. (1991). Methodological and psychometric issues in setting cutoff scores using the Angoff method. Personnel Psychology, 44, 235-262.
16.
Mehrens, W. A. , & Popham, W. J. (1992). How to evaluate the legal defensibility of high-stakes tests. Applied Measurement in Education, 5, 265-283.
17.
Norcini, J. J. , Lipner, R. S., Langdon, L. O., & Strecker, C. A. (1987). A comparison of three variations on a standard-setting method. Journal of Educational Measurement, 24, 56-64.
18.
Norcini, J. , Shea, J., & Grosso, L. (1991). The effect of numbers of experts and common items on cutting score equivalents based on expert judgment. Applied Psychological Measurement, 15, 241-246.
19.
Norcini, J. J. , Shea, J. A., & Kanya, D. T. (1988). The effect of various factors on standard setting. Journal of Educational Measurement, 25, 57-65.
20.
Plake, B. S. , Impara, J. C., & Potenza, M. T. (1994). Content specificity of expert judgments in a standard-setting study. Journal of Educational Measurement, 31, 339-347.
21.
Reilly, R. R. , Zink, D. L., & Israelski, E. W. (1984). Comparison of direct and indirect methods for setting minimum passing scores. Applied Psychological Measurement, 8, 421-429.
22.
Shavelson, R. J. , & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
23.
Shepard, L. (1980). Standard setting issues and methods. Applied Psychological Measurement, 4, 447-467.
24.
Van der Linden, W. J. (1982). A latent trait method for determining intrajudge inconsistency in the Angoff and Nedelsky techniques of standard setting. Journal of Educational Measurement, 19, 295-308.
