Item Selection Strategy for Reducing the Number of Items Rated in an Angoff Standard Setting Study

Abstract

In an Angoff standard setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item in the test. In many cases, these item performance estimates are made twice, with information shared with the panelists between estimates. Especially for long tests, this estimation process can be time-consuming and fatiguing for the judges. This study extended an item selection strategy earlier proposed to form subsets of test items. The results of this study suggest that 40% to 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study when those items are selected to represent the full test in content, discrimination, and difficulty.

Keywords

standard setting minimum passing score Angoff method subsets of test items stratified item selection

Get full access to this article

View all access options for this article.

References

Angoff, W.H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-560). Washington, DC: American Council on Education.

Ferdous, A. , & Plake, B. (2005). The use of subsets of test questions in an Angoff standard setting method. Educational and Psychological Measurement , 65(2), 185-201.

Hambleton, R.K. (1998). Setting performance standards on achievement tests: Meeting the requirements of Title I. In L. N. Hansche (Ed.), Handbook for the development of performance standards: Meeting the requirements of Title I (pp. 97-114). Washington, DC: Council of Chief State School Officers.

Harvey, A.L. , & Way, W.D. (1999, April). A comparison of web-based standard setting and monitored standard setting. Paper presented at the annual meeting of the National Council on Measurement in Education , Montreal, Canada.

National Assessment Governing Board. (1994 ). Setting achievement levels on the 1994 National Assessment of Educational Progress in Geography and in US History and the 1996 National Assessment of Education Progress in Science. Final Version. Washington, DC: Author.

Plake, B.S. , & Impara, J.C. (1997, April). Consistency of Angoff-based predictions as a function of item difficulty. Paper presented at the annual meeting of the National Council on Measurement in Education , Chicago.

Plake, B.S. , & Impara, J.C. (2001). Ability of panelists to estimate item performance for a target group of candidates: An issue in judgmental standard setting . Educational Assessment, 7, 87-97.

Reckase, M.D. (2001). The evaluation of the NAEP achievement level setting process: A summary of the research and developmental efforts. Iowa City, IA: ACT, Inc.

Sireci, S.G. , Patelis, T. , Rizavi, S. , Dillingham, A.M. , & Rodriguez, G. (2000, April). Setting standards on a computerized-adaptive placement examination. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.

10.

U.S. Department of Education. (2002). No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002).