Cut-Score Operating Function Extensions: Penalty-Based Errors and Uncertainty in Standard Settings

Abstract

We model pass/fail examinations aiming to provide a systematic tool to minimize classification errors. We use the method of cut-score operating functions to generate specific cut-scores on the basis of minimizing several important misclassification measures. The goal of this research is to examine the combined effects of a known distribution of examinee abilities and uncertainty in the standard setting on the optimal choice of the cut-score. In addition, we describe an online application that allows others to utilize the cut-score operating function for their own standard settings.

Get full access to this article

View all access options for this article.

References

American Educational Research Association (2014). American Psychological Association, & National Council on Measurement in Education Standards for educational and psychological testing. American Educational Research Association.

Angoff

W. H.

(1971). Scales, norms, and equivalent scores. In Thorndike

R. L.

(Ed.), Educational measurement (pp. 508-600). Washington, DC: American Council on Education.

Attali

(2018). shinyjs: Easily Improve the User Experience of Your Shiny Apps in Seconds. R package version 1.0.

Buckendahl

C. W.

Smith

R. W.

Impara

J. C.

Plake

B. S.

(2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39(3), 253-263.

Chang

Borges Ribeiro

(2018). shinydashboard: Create Dashboards with ’Shiny’. R package version 0.7.1.

Chang

Cheng

Allaire

Xie

McPherson

(2018). shiny: Web Application Framework for R. R package version 1.2.0.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.

Grabovsky

Pace

Runyon

(2020). CutScore. https://cutscore.shinyapps.io/cutscore/.

Grabovsky

Wainer

(2017a). The cut-score operating function: A new tool to aid in standard setting. Journal of Educational and Behavioral Statistics, 42(3), 251-263.

10.

Grabovsky

Wainer

(2017b). A guide for setting the cut scores to minimize weighted classification errors in test batteries. Journal of Educational and Behavioral Statistics, 42(3), 264-281.

11.

Lee

W.-C.

(2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47(1), 1-17.

12.

Leventhal

B. C.

Grabovsky

(2019). Adding objectivity to standard setting: Evaluating consequence using the conscious and subconscious weight methods. submitted.

13.

Livingston

S. A.

Lewis

(1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32(2), 179-197.

14.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates, Inc.

15.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Addison-Wesley.

16.

Pitoniak

M. J.

Morgan

D. L.

(2012). Setting and validating cut scores for tests. In Handbook on measurement, assessment, and evaluation in higher education (pp. 362-385). Routledge.

17.

R Core Team . (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

18.

Yerushalmy

(1947). Statistical problems in assessing methods of medical diagnosis, with special reference to x-ray techniques. Public Health Reports, 62(40), 1432-1449.

19.

Youden

W. J.

(1950). Index for rating diagnostic tests. Cancer, 3(1), 32-35.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.51 MB