Judgmental standard setting methods have been criticized for the cognitive complexity of the judgment task that panelists are asked to complete. This study compared two methods designed to reduce this complexity: the yes/no method and the single-passage bookmark method. Two mock standard setting panel meetings were convened, one for each method, using experienced third-grade teachers as panelists. The methods produced very similar cut scores and they seemed to succeed in reducing the cognitive complexity of the judgment task. There was also evidence, however, that the participants in both groups experienced a great deal of difficulty in estimating item performance for borderline students and that they may have used contextual information associated with the judgmental task to estimate their cut scores.
Angoff, W.H. ( 1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (pp. 508-600). Washington, DC: American Council on Education.
2.
Beretvas, S.N. ( 2004). Comparison of bookmark difficulty locations under different item response models. Applied Psychological Measurement, 28, 25-47.
3.
Brandon, P.R. ( 2004). Conclusions about frequently studied modified Angoff standard-setting topics. Applied Measurement in Education , 17, 59-88.
4.
Buckendahl, C.W., Smith, R.W., Impara, J.C., & Plake, B.S. ( 2002). A comparison of Angoff and bookmark standard setting methods. Journal of Educational Measurement, 39, 253-263.
5.
Cizek, G., & Bunch, M. ( 2006). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.
6.
Clauser, B.E., Mee, J., Baldwin, S.G., Margolis, M.J., & Dillon, G.F. ( 2009). Judges’ use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: An experimental study. Journal of Educational Measurement, 46, 390-407.
7.
Dawber, T., Lewis, D.M., & Rogers, W.T. ( 2002, April). The cognitive experience of bookmark standard setting participants. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
8.
Egan, K., & Green, D.R. ( 2003, April). Influences on judges’ decisions. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
9.
Ferdous, A.A., & Plake, B.S. ( 2005). Understanding the factors that influence decisions of panelists in a standard setting-study. Applied Measurement in Education, 18, 257-267.
10.
Giraud, G., Impara, J.C., & Plake, B.S. ( 2005). Teachers’ conceptions of the target examinee in Angoff standard setting. Applied Measurement in Education , 18, 223-232.
11.
Hurtz, G.M., & Auerbach, M.A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584-601.
12.
Impara, J.C., Giraud, G., & Plake, B.S. ( 2000, April). The influence of providing target group descriptors when setting a passing score (ERIC Document Reproduction Service No. ED445013) . Paper presented the annual meeting of the American Educational Research Association, New Orleans, LA.
13.
Impara, J.C., & Plake, B.S. ( 1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353-366.
14.
Jaeger, R.M. ( 1995). On the cognitive construction of standard-setting judgments: The case of configural scoring. In Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 57-69). Washington, DC: National Assessment Governing Board and National Center for Education Statistics .
15.
Lewis, D.M., & Green, D.R. ( 1997, June). The validity of performance level descriptors. Paper presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
16.
Loomis, S.C. ( 2000, April). Feedback in the NAEP achievement levels setting process. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
17.
McGinty, D. ( 2005). Illuminating the "black box" of standard setting: An exploratory qualitative study. Applied Measurement in Education , 18, 269-287.
18.
Mislevy, R.J., & Bock, R.J. ( 1989). BILOG3: Item analysis and test scoring with binary logistic models (1sted.). Mooresville, IN: Scientific Software.
19.
Mitzel, H.C., Lewis, D.M., Patz, R.J., & Green, D.R. ( 2001). The bookmark procedure: Psychological perspectives. In C. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 249-281). Mahwah, NJ: Erlbaum.
20.
No Child Left Behind Act (NCLB) of 2001, 20 U.S.C. §6311 et seq. (2002).
21.
Reckase, M.D. ( 2005, April). A theoretical evaluation of an item rating method and a bookmark method for setting standards. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada.
22.
Shepard, L.A. ( 1995). Implications for standard setting of the National Academy of Education evaluation of the National Assessment of Educational Progress achievement levels. In Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 143-160). Washington, DC: National Assessment Governing Board and National Center for Education Statistics .
23.
Shepard, L., Glaser, R., Linn, R.L., & Bohrnstedt, G. (1993). Setting performance standards for student achievement. A report of the National Academy of Education Panel on the evaluation of the NAEP trial state assessment: An evaluation of the 1992 achievement levels. Stanford, CA : Stanford University, National Academy of Education.
24.
Skaggs, G. ( 2007). Bookmark locations and item response model selection in the presence of local item dependence. Journal of Applied Measurement, 8, 65-83.
25.
Skaggs, G., Hein, S., & Awuor, R. ( 2007). Setting passing scores on passage-based tests: A comparison of traditional and single-passage bookmark methods. Applied Measurement in Education, 20, 405-426.
26.
Skaggs, G., & Tessema, A. ( 2001, April). Item disordinality with the bookmark standard setting procedure. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
27.
Skorupski, W.P., & Hambleton, R.K. (2005). What are panelists thinking when they participate in standard-setting studies? Applied Measurement in Education, 18, 233-256.
