The Angoff method for setting cutoff scores on examinations has been researched extensively over the past 30 years, including investigations of several procedural modifications to the method. In the current study, the authors meta-analytically synthesized this research and evaluated the systematic effects of the most common modifications on the level of the resulting cutoff scores and the degree of consensus among judges. The authors found the following: (a) when judges use a common definition of minimally competent test-takers, this tends to increase judgment consensus; (b) when judges discuss their estimates, this tends to result in higher cutoff scores with increased consensus; and (c) when judges view normative data, this tends to systematically lower cutoff scores. In addition, the authors found an interaction effect which revealed that when judges use a common definition and later discuss their estimates, this tends to produce the highest standards on average, with the highest degree of consensus among judges. Theoretical and practical implications of these effects are discussed.
Get full access to this article
View all access options for this article.
References
1.
Alliger, G., & Coats, G. (1992, April). An investigation of judges’ability to estimate item performance of minimally competent performers. Poster session presented at the annual meeting of the Society for Industrial and Organizational Psychology, Montreal, Canada.
2.
Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council in Education.
3.
Baker, C. (1993). The Angoff method and rater analysis: Enhancing cutoff score reliability and accuracy. Masters Abstracts International, 32(3), 1075.
4.
Behuniak, P., Jr., Archambault, F. X., & Gable, R. K. (1982). Angoff and Nedelsky standard setting procedures: Implications for the validity of proficiency test score interpretation. Educational and Psychological Measurement, 42, 247-255.
5.
Behuniak, P., Jr., Gable, R. K., & Archambault, F. X. (1982). The validity of categorized proficiency test scores. Educational and Psychological Measurement, 42, 869-876.
6.
Berk, R. A. (1986). A consumer’s guide to setting performance standards on criterion referenced tests. Review of Educational Research, 56, 137-172.
7.
Biddle, R. E. (1993). How to set cutoff scores for knowledge tests used in promotion, training, certification, and licensing. Public Personnel Management, 22(1), 63-79.
8.
Brennan, R. L., & Lockwood, R. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
9.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
10.
Bryk, A. S., Raudenbush, S. W., & Congdon, R. T., Jr. (1996). HLM: Hierarchical linear and nonlinear modeling with the HLM/2L and HLM/3L programs. Chicago, IL: Scientific Software International.
11.
Busch, J., & Jaeger, R. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the National Teacher Examinations. Journal of Educational Measurement, 27, 145-163.
12.
Cascio, W. F., Alexander, R. A., & Barrett, G. V. (1988). Setting cutoff scores: Legal, psychometric, and professional issues and guidelines. Personnel Psychology, 41, 1-24.
13.
Chang, L., Dziuban, C. D., & Hynes, M. C. (1996). Does a standard reflect minimal competency of examinees or judge competency? Applied Measurement in Education, 9, 161-173.
14.
Cizek, G. J. (1993). Reconsidering standards and criteria. Journal of Educational Measurement, 30, 93-106.
15.
Cross, L., Impara, J., Frary, R., & Jaeger, R. (1985). A comparison of three methods for establishing minimum standards on the National Teacher Examinations. Journal of Educational Measurement, 21, 113-129.
16.
Cusimano, M. D. (1996). A study of standard setting methods in an objective structured clinical examination (medicine, surgery, competency, expertise). Dissertation Abstracts International, 58 (06A), 2174.
17.
Fehrmann, M. L., Woehr, D. J., & Arthur, W., Jr. (1991). The Angoff cutoff score method: The impact of frame-of-reference rater training. Educational and Psychological Measurement, 51, 857-872.
18.
Fitzpatrick, A. R. (1989). Social influences in standard setting: The effects of social interaction on group judgments. Review of Educational Research, 59, 315-328.
19.
Goodwin, L. D. (1999). Relations between observed item difficulty levels and Angoff minimum passing levels for a group of borderline examinees. Applied Measurement in Education, 12, 13-28.
20.
Halpin, G., & Halpin, G. (1987). An analysis of the reliability and validity of procedures for setting minimum competency standards. Educational and Psychological Measurement, 47, 977-983.
21.
Halpin, G., Sigmon, G., & Halpin, G. (1983). Minimum competency standards set by three divergent groups of raters using three judgmental procedures: Implications for validity. Educational and Psychological Measurement, 43, 185-196.
22.
Harasym, P. H. (1981). A comparison of the Nedelsky and modified Angoff standard-setting procedure on evaluation outcome. Educational and Psychological Measurement, 41, 725-734.
23.
Hudson, J. P., Jr., & Campion, J. E. (1994). Hindsight bias in an application of the Angoff method for setting cutoff scores. Journal of Applied Psychology, 79, 860-865.
24.
Hunter, D. (1997). The evidential bases of judges’ decisions in setting performance standards for the 1996 Saskatchewan learning assessment in reading and writing. Masters Abstracts International, 37(1), 0020.
25.
Hurtz, G. M., & Hertz, N. (1999). How many raters should be used for establishing cutoff scores with the Angoff method? A generalizability theory study. Educational and Psychological Measurement, 59, 885-897.
26.
Jaeger, R. M. (1982). An iterative structured judgment for establishing standards on competency tests: Theory and application. Educational Evaluation and Policy Analysis, 4(4), 461-475.
27.
Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
28.
Leary, M. R., & Kowalski, R. M. (1990). Impression management: A literature review and twocomponent model. Psychological Bulletin, 107(1), 34-47.
29.
Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.
30.
Maurer, T., & Alexander, R. A. (1992). Methods of improving employment test critical scores derived by judging test content: A review and critique. Personnel Psychology, 45, 727-762.
31.
Maurer, T., Alexander, R., Callahan, C., Bailey, J., & Dambrot, F. H. (1991). Methodological and psychometric issues in setting cutoff scores using the Angoff method. Personnel Psychology, 44, 235-262.
32.
McGinty, D. (1997). Judgmental standard setting using a cognitive components model. Dissertation Abstracts International, 57 (10-A), 4340.
33.
Mehrens, W. A., & Popham, W. J. (1992). How to evaluate the legal defensibility of high-stakes tests. Applied Measurement in Education, 5, 265-283.
34.
Myers, D. G., & Lamm, H. (1976). The group polarization phenomenon. Psychological Bulletin, 83(4), 602-627.
35.
Norcini, J., Lipner, R., Langdon, L., & Strecker, C. (1987). A comparison of three variations on a standard-setting method. Journal of Educational Measurement, 24, 56-64.
36.
Norcini, J., & Shea, J. (1992a). Equivalent estimates of borderline group performance in standard setting. Journal of Educational Measurement, 29, 19-24.
37.
Norcini, J., & Shea, J. (1992b). The reproducibility of standards over groups and occasions. Applied Measurement in Education, 5, 63-72.
38.
Norcini, J., Shea, J., & Grosso, L. (1991). The effect of numbers of experts and common items on cutting score equivalents based on expert judgment. Applied Psychological Measurement, 15, 241-246.
39.
Norcini, J., Shea, J., & Kanya, D. T. (1988). The effect of various factors on standard setting. Journal of Educational Measurement, 25, 57-65.
40.
Norcini, J., Shea, J., Lipner, R. (1994). The effect of anchor item characteristics on equivalent cutting scores. Applied Measurement in Education, 7, 187-194.
41.
Norcini, J., Shea, J., & Ping, J. C. (1988). A note on the application of multiple matrix sampling to standard setting. Journal of Educational Measurement, 25, 159-164.
42.
Plake, B. S. (1998). Setting performance standards for professional licensure and certification. Applied Measurement in Education, 11, 65-80.
43.
Plake, B. S., Impara, J., & Potenza. (1994). Content specificity of expert judgments in a standard-setting study. Journal of Educational Measurement, 31, 339-347.
44.
Raudenbush, S. W., & Bryk, A. S. (1987). Examining correlates of diversity. Journal of Educational Statistics, 12(3), 241-269.
45.
Reilly, R. R., Zink, D. L., & Israelski, E. W. (1984). Comparison of direct and indirect methods for setting minimum passing scores. Applied Psychological Measurement, 8, 421-429.
46.
Shepard, L. (1980). Standard setting issues and methods. Applied Psychological Measurement, 4, 447-467.
47.
Smith, R., & Smith, J. (1989). Differential use of item imformation by judges using Angoff and Nedelsky procedures. Journal of Educational Measurement, 25, 259-274.
48.
Truxillo, D. M., Donahue, L. M., & Sulzer, J. L. (1996). Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations. Human Performance, 9, 275-295.
49.
West, J. A. (1998). Do performance standards reflect conceptions of competence? The relationship between implicit theories of competence and standard setting judgments. Dissertation Abstracts International, 59 (3-B), 1402.
50.
Woehr, D. J., Arthur, W., Jr., & Fehrmann, M. L. (1991). An empirical comparison of cutoff score methods for content-related and criterion-related validity settings. Educational and Psychological Measurement, 51, 1029-1039.
