Sage Journals: Discover world-class research

Abstract

Test items scored as polytomous have the potential to display multidimensionality across rating scale score categories. This article uses a multidimensional nominal response model (MNRM) to examine the possibility that the proficiency dimension/dimensional composite best measured by a polytomously scored item may vary by score category, an issue not generally considered in multidimensional item response theory (MIRT). Some practical considerations in exploring rubric-related multidimensionality, including potential consequences of not attending to it, are illustrated through simulation examples. A real data application is applied in the study of item format effects using the 2007 administration of Trends in Mathematics and Science Study (TIMSS) among eighth graders in the United States.

Keywords

item response theory multidimensionality polytomous models

Get full access to this article

View all access options for this article.

References

Ackerman

T. A.

(1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7, 255-278.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.

Bolt

D. M.

Johnson

T. R.

(2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33, 335-352.

Falk

C. F.

Cai

(2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21, 328-347. doi:10.1037/met0000059

Hattie

(1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139-164.

Kelderman

Rijkes

C. P. M.

(1994). Loglinear multidimensional IRT models for polytomously scored items. Psychometrika, 59, 149-176.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

Meulman

J. J.

Van der Kooij

A. J.

Heiser

W. J.

(2004). Principal components analysis with nonlinear optimal scaling transformations for ordinal and nominal data. In Kaplan

(Ed.), Handbook of quantitative methodology for the social sciences (pp. 49-70). Newbury Park, CA: Sage.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.

10.

Muraki

Carlson

J. E.

(1993, April). Full-information factor analysis for polytomous item responses. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA.

11.

Reckase

M. D.

(1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401-412.

12.

Reckase

M. D.

(2009). Multidimensional item response theory (Vol. 150). New York, NY: Springer.

13.

Rodriguez

M. C.

(2003). Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations. Journal of Educational Measurement, 40, 163-184.

14.

Thissen

Cai

Bock

R. D.

(2010). The nominal categories item response model. In Nering

Ostini

(Eds.), Handbook of polytomous item response theory models (pp. 43-75). New York, NY: Routledge.

15.

Thissen

Steinberg

(1986). A taxonomy of item response models. Psychometrika, 51, 567-577.

16.

Thissen

Wainer

Wang

X.-B.

(1994). Are tests comprising both multiple-choice and free-response items necessarily less unidimensional than multiple-choice tests? An analysis of two tests. Journal of Educational Measurement, 31, 113-123.

17.

Thurstone

L. L.

(1947). Multiple factor analysis. Chicago, IL: The University of Chicago Press.

18.

Vermunt

J. K.

Magidson

(2008). LG-syntax user’s guide: Manual for Latent GOLD 4.5 syntax module Belomonte, MA: Statistical Innovations Inc.

19.

Wang

Drasgow

Liu

(2016). Classification accuracy of mixed format tests: A bifactor item response theory approach. Frontiers in Psychology, 7, 270. doi: 10.3389/fpsyg.2016.00270

20.

Adams

(2006). Modelling mathematics problem solving item responses using a multidimensional IRT model. Mathematics Education Research Journal, 18, 93-113.

21.

Yao

Schwarz

R. D.

(2006). A multidimensional partial credit model with associated item and test statistics: An application to mixed-format tests. Applied Psychological Measurement, 30, 469-492.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.56 MB

Exploring Rubric-Related Multidimensionality in Polytomously Scored Test Items

Abstract

Keywords

Get full access to this article

References

Supplementary Material