Comparison Between Dichotomous and Polytomous Scoring of Innovative Items in a Large-Scale Computerized Adaptive Test

Abstract

This study explored the impact of partial credit scoring of one type of innovative items (multiple-response items) in a computerized adaptive version of a large-scale licensure pretest and operational test settings. The impacts of partial credit scoring on the estimation of the ability parameters and classification decisions in operational test settings were explored in one real data analysis and two simulation studies when two different polytomous scoring algorithms, automated polytomous scoring and rater-generated polytomous scoring, were applied. For the real data analyses, the ability estimates from dichotomous and polytomous scoring were highly correlated; the classification consistency between different scoring algorithms was nearly perfect. Information distribution changed slightly in the operational item bank. In the two simulation studies comparing each polytomous scoring with dichotomous scoring, the ability estimates resulting from polytomous scoring had slightly higher measurement precision than those resulting from dichotomous scoring. The practical impact related to classification decision was minor because of the extremely small number of items that could be scored polytomously in this current study.

Keywords

polytomous scoring innovative items computerized adaptive test Rasch model partial credit model

Get full access to this article

View all access options for this article.

References

Bennett

R. E.

Goodman

Hessinger

Ligget

Marshall

Kahn

Zack

(1999). Using multimedia in large-scale computer-based testing programs. Computers in Human Behavior, 15, 283-294.

Chung

Baker

(1997). Year 1 technology studies: Implications for technology in assessment (CSE Technical Report No. 459). Los Angeles: Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, University of California.

Harmes

J. C.

(1999). Computer-based testing: Toward the design and use of innovative items. Tampa: University of South Florida.

Linacre

J. M.

(2005). WINSTEPS: Rasch measurement program. Retrieved from www.winsteps.com

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

Parshall

C. G.

Davey

Pashley

(2000). Innovative items for computerized testing. In van der Linden

W. J.

Glas

C. A. W.

(Eds.), Computerized adaptive testing: Theory and practice (pp. 129-148). Norwell, MA: Kluwer Academic.

Parshall

C. G.

Harmes

J. C.

(2008). The design of innovative item types: Targeting constructs, selecting innovations, and refining prototypes. Clear Exam Review, XIX(2), 18-25.

Parshall

C. G.

Spray

Kalohn

Davey

(2002). Issues in innovative item types. In Practical considerations in computer-based testing (pp. 70-91). New York, NY: Springer.

Pellegrino

J. W.

Chudowsky

Glaser

(Eds.). (2001). Knowing what students know: The science and design of educational assessment (National Research Council Center for Education Report). Washington, DC: National Academies Press.

10.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danmarks Paedogogische Institut.

11.

Scalise

(2007). Using innovative items in computer-based testing: Automated scoring and measurement models. Retrieved from http://www.lamark.com/downloads/Kathleen_Scalise.pdf

12.

Wendt

(2008). Investigation of the item characteristics of innovative item formats. Clear Exam Review, 19(1), 22-28.

13.

Wolfe

E. W.

McGill

M. T.

(2006). The comparability of item quality indices from sparse data matrices. Research report submitted to the NCSBN Joint Research Committee.