This study compares an expert judgment process—minimal performance levels (MPL) using the Nedelsky and Ebel procedures—for setting cutoff scores for pass/fail on licensure examinations with an empirical approach—cluster analysis. Data from all three components of the Canadian Standard Assessment in Optometry (CSAO) examinations (knowledge, clinical judgment, and clinical skills) from 243 candidates were obtained. Results indicate that for the written components of the exams employing the Nedelsky method of MPL setting, there was a mean agreement of pass/fail of 81% with the cluster analysis approach on pass/fail categorization. For the performance exams using the Ebel method, the mean agreement of pass/fail with the cluster analysis was 93%. Thus the subjective approaches to setting cutoff scores (i.e., expert judgment methods) converge with the objective method (i.e., cluster analysis) of classifying test takers in the same categories.
Anderberg, M.R.
(1973). Cluster analysis for application. New York: Academic Press.
2.
Behuniak, P.
, Archambault, R.G., & Gable, R.K. (1982). Angoff and Nedelsky standard-setting procedures: Implications for the validity of proficiency score interpretation. Educational and Psychological Measurement, 42, 247-255.
3.
Berk, R.
(1996). Standard setting: The next generation (where few psychometricians have gone before). Applied Measurement in Education, 9, 215-235.
4.
Blashfield, R.K.
, & Aldenderfer, M.S. (1978). The literature on cluster analysis. Multivariate Behavioral Research, 13, 271-295.
5.
Dwyer, C.
(1996). Cutoff scores and testing: Statistics, judgment, truth, and error. Psychological Assessment, 8, 360-362.
Glass, G.
(1978). Standards and criteria. Journal of Educational Measurement, 15, 237-261.
9.
Gore, P.A.
(2000). Cluster analysis. In H.E. Tinsley & S.D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling. New York: Academic Press.
10.
Hambleton, R.K.
(1995, August). Setting standards in licensure tests. Paper presented at the 103rd convention of the American Psychological Association, New York, NY.
11.
Jaeger, R.M.
(1980, April). Applying the Angoff and Nedelsky techniques to the national licensing examinations in landscape architecture. Paper presented at the annual meeting of the National Council of Measurement in Education, Boston.
12.
Jaeger, R.M.
(1995). Setting performance standards through two-stage judgmental policy capturing. Applied Measurement in Education, 8, 15-40.
13.
Kane, M.
(1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
14.
Kane, M.
(1995). Validating interpretive arguments for licensure and certification examinations. Evaluation & the Health Professions, 17, 133-159.
15.
Kaufman, L.
, & Rousseeuw, P.J. (1990). Finding groups in data: An introduction to cluster analysis. New York: John Wiley.
16.
Meskauskas, J.
(1976). Evaluation models for criterion-referenced testing: View regarding mastery and standard setting. Review of Educational Research, 45, 133-158.
17.
Nedelsky, L.
(1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 3-19.
18.
Raymond, M.
(1996). Establishing weights for test plans for licensure and certification examinations. Applied Measurement in Education, 9, 237-256.
19.
Sireci, S.G.
(1995, August). Using cluster analysis to solve the problem of standard setting. Paper presented at the annual meeting of the American Psychological Association, New York, NY.
20.
Sireci, S.G.
, Robin, F., & Patellis, T. (1997, April). Using cluster analysis to facilitate the standard-setting process. Paper presented at the 103rd convention of the National Council of Measurement in Education, Chicago.
21.
Swanson, D.B.
, Norman, G.R., & Linn, R.L. (1995, June/July). Performance-based assessment: Lessons from the health professions. Educational Researcher, 35, 5-11.
22.
van der Vleuten, C.P.
, & Swanson, D.B. (1990). Assessment of clinical skills with standardized patients: State of the art. Teaching and Learning in Medicine, 2, 58-76.
23.
Violato, C.
, Chou, R., McDowell, M., & Marini, A. (1999). A psychometric analysis of the Canadian Standard Assessment in Optometry. Canadian Journal of Optometry, 61, 144-149.
24.
Violato, C.
, McDougal, D., & Marini, A. (1992). Educational measurement and evaluation. Dubuque, IA: Kendall/Hunt.
25.
Vu, N.V.
, Marcy, M.L., Barnhart, A., Colliver, J.A., Henkle, J.Q., & Hodgson, K. (1994). Further evidence of construct validity of standardized patient-based performance examinations. Teaching and Learning in Medicine, 6, 255-259.
