Convergence Between Cluster Analysis and the Angoff Method for Setting Minimum Passing Scores on Credentialing Examinations

Abstract

Cluster analysis can be a useful statistical technique for setting minimum passing scores on high-stakes examinations by grouping examinees into homogenous clusters based on their responses to test items. It has been most useful for supplementing data or validating minimum passing scores determined from expert judgment approaches, such as the Ebel and Nedelsky methods. However, there is no evidence supporting how well cluster analysis converges with the modified Angoff method, which is frequently used in medical credentialing. Therefore, the purpose of this study is to investigate the efficacy of cluster analysis for validating Angoff-derived minimum passing scores. Data are from 652 examinees who took a national credentialing examination based on a content-by-process test blueprint. Results indicate a high degree of consistency in minimum passing score estimates derived from the modified Angoff and cluster analysis methods. However, the stability of the estimates from cluster analysis across different samples was modest.

Keywords

standard setting Angoff method cluster analysis high-stakes examinations

Get full access to this article

View all access options for this article.

References

Anderberg, M.R. (1973). Cluster analysis for applications. New York: Academic Press.

Angoff, W.H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.

Chinn, R.N. , & Hertz, N.R. (2002). Alternative approaches to standard setting for licensure and certification examinations. Applied Measurement in Education, 15, 1-14.

Cizek, G.J. (2001). Conjectures on the rise and call of standard setting: An introduction to context and practice. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 1-17). Mahwah, NJ: Lawrence Erlbaum.

Cohen, J. (1960). A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 10, 37-46.

Downing, S.M. , Tekian, A. , & Yudkowsky, R. (2006). Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teaching and Learning in Medicine, 18, 50-57.

Ebel, R.L. (1979). Essentials of educational measurement (3rd. ed.). Englewood Cliffs, NJ: Prentice Hall.

Engelhard, G. , & Stone, G.E. (1998). Evaluating the quality of ratings obtained from standard-setting judges. Educational and Psychological Measurement , 58, 179-196.

Hurtz, G.M. , & Auerbach, M.A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584-601.

10.

Livingston, S.A. , & Zieky, M.J. (1989). A comparative study of standard-setting methods . Applied Measurement in Education, 2, 121-141.

11.

Meara, K.P. , Hambleton, R.K. , & Sireci, S.G. (2001). Setting and validating standards on professional licensure and certification exams: A survey of current practices. Clear Exam Review, 12, 17-23.

12.

Nedelsky, L. (1954). Absolute grading standards for objective tests . Educational and Psychological Measurement, 14, 3-19.

13.

Plake, B.S. , Hambleton, R.K. , & Jaeger, R.M. (1997). A new standard-setting method for performance assessments: The dominant profile judgment method and some field tests results . Educational and Psychological Measurement, 57, 400-411.

14.

Ramsey, P.G. , Carline, J.D. , Inui, T.S. , Larson, E.B. , LoGerfo, J.P. , Norcini, J.J. , et al. (1991). Changes over time in the knowledge base of practicing internists. Journal of the American Medical Association, 266, 1103-1107.

15.

Searle, J. (2000). Defining competency: The role of standard setting . Medical Education, 34, 363-366.

16.

Sireci, S.G. (2001). Standard setting using cluster analysis. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 339-354). Mahwah, NJ: Lawrence Erlbaum.

17.

Sireci, S.G. , Hambleton, R.K. , & Pitoniak, M.J. (2004). Setting passing scores on licensure examinations using direct consensus. Clear Exam Review, 15, 21-25.

18.

Sireci, S.G. , Robin, F. , & Patelis, T. (1999). Using cluster analysis to facilitate standard setting. Applied Measurement in Education, 12, 301-325.

19.

Truxillo, D.M. , Donahue, L.M. , & Sulzer, J.L. (1996). Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations. Human Performance , 9, 275-295.

20.

Violato, C. , Marini, A. , & Lee, C. (2003). A validity study of expert judgment procedures for setting cutoff scores on high-stakes credentialing examinations using cluster analysis. Evaluation and the Health Professions, 26, 59-72.

21.

Wang, N. (2003). Use of the Rasch IRT model in standard setting: An item-mapping method. Journal of Educational Measurement , 40, 231-253.