A Two-Stage Scoring Method to Enhance Accuracy of Performance Level Classification

Abstract

Many tests classify each examinee into one of multiple performance levels on the basis of a combination of multiple-choice (MC) and constructed-response (CR) items. This study introduces a two-stage scoring method that identifies examinees whose MC scores place them near a cut point, advising scorers on which examinees will be most affected by score discrepancies. The idea that additional attention be given to such examinees' CR responses is explored from both psychometric and practical perspectives. Because more precision is achieved for examinees near the cut points, the method has potential to enhance the percentage of examinees classified correctly. In an empirical investigation of two operational assessments, the two-stage scoring approach successfully predicted which examinees would ultimately attain scores near the border between two performance levels.

Keywords

classification accuracy interrater reliability large-scale assessment score resolution

Get full access to this article

View all access options for this article.

References

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 392-479). Reading, MA : Addison-Wesley.

Hambleton, R.K. , & Novick, M.R. (1973). Toward an integration of theory and method for criterion-references tests. Journal of Educational Measurement , 10, 159-170.

Hambleton, R.K. , & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer .

Hambleton, R.K. , & van der Linden, W.J. (1997). Handbook of modern item response theory. New York: Springer-Verlag.

Huynh, H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13, 725-736.

Huynh, H. (1980). A nonrandomized minimax solution for passing scores in the binomial error model. Psychometrika, 45, 167-182.

Johnson, R.L. , Penny, J. , Fisher, S. , & Kuhs, T. (2003). Score resolution: An investigation of the reliability and validity of resolved scores. Applied Measurement in Education , 16(4), 299-322.

Johnson, R.L. , Penny, J. , & Gordon, B. (2000). The relation between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric . Applied Measurement in Education, 13(2), 121-138.

Livingston, S.A. , & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-197.

10.

Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.

11.

Mislevy, R.J. , & Bock, R.D. (1990). BILOG-3 (2nd ed.): Item analysis and test scoring with binary logistic models. Mooresville, IN: Scientific Software.

12.

Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, Monograph Supplement No. 17.

13.

Schaeffer, G.A. , Henderson-Montero, D. , Julian, M. , & Bené, N.H. (2002). A comparison of three scoring methods for tests with selected-response and constructed-response items. Educational Assessment, 8, 317-340.

14.

Sykes, R.C. , & Ito, K. (1998). The effect of rounding aggregated item ratings for constructed response items in mixed-item format tests. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.

15.

van der Linden, W.J. , & Vos, H.J. (1996). A compensatory approach to optimal selection with mastery scores. Psychometrika, 61, 155-172.

16.

Yen, W.M. (1984). Obtaining maximum likelihood trait estimates from number-correct scores for the three-parameter logistic model. Journal of Educational Measurement, 21, 93-111.

17.

Yen, W.M. , & Candell, G.L. (1991). Increasing score reliability with item-pattern scoring: An empirical study in five score metrics. Applied Measurement in Education, 4, 209-228.