Sage Journals: Discover world-class research

Abstract

Dimensionality assessment is an important component of test validation. While the vast majority of statewide standardized tests contain both dichotomous and polytomous items, much of the work in dimensionality assessment has focused on the case of dichotomous item exams. Poly-NEWDIM is a polytomous version of DIMTEST that is a nonparametric hypothesis testing procedure for dimensionality assessments based on comparison of an assessment subtest (AT) with a partitioning subtest (PT). A good AT and PT selection is vital to the performance of Poly-NEWDIM. The proposed two new AT-selection procedures, HCA/CCPROX-PolyDIMTEST (HCP) and HCA/CCPROX-PolyNEWDIM (HCN), perform better than the previous AT-selection method (HCD) in terms of power. Moreover, HCN shows less sensitivity than HCP to sample size, correlation, and structural complexity. The results also indicate that 70% of data for AT selection is appropriate for all three kinds of tests (dichotomous only, polytomous only, and mixed) with large sample sizes and dichotomous tests with small sample sizes, while 50% is good for polytomous tests and mixed tests with small sample sizes.

Keywords

dimensionality DIMTEST polytomous items

Get full access to this article

View all access options for this article.

References

Ackerman

T. A.

(1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurements, 13, 113-127.

Ackerman

T. A.

(1996). Graphical representation of multidimensional item response theory. Applied Psychological Measurement, 20, 311-329.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.

Froelich

Habing

(2008). Conditional covariance-based subtest selection for DIMTEST. Applied Psychological Measurement, 32, 138-155.

Habing

(2001). Nonparametric regression and the parametric bootstrap for local dependence assessment. Applied Psychological Measurement, 25, 221-233.

Hattie

J. A.

(1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139-164.

Hattie

J. A.

Krakowski

Rogers

H. J.

Swaminathan

(1996). An assessment of Stout’s index of essential unidimensionality. Applied Psychological Measurement, 20, 1-14.

Kim

H. R.

(1994). New techniques for the dimensionality assessment of standardized test data (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign, Department of Statistics, Champaign.

Kirisci

Hsu

(1995, April). The robustness of BILOG to violations of the assumptions of unidimensionality of test items and normality of ability distribution. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.

10.

H.-H.

Stout

W. F.

(1995, April). Assessment of unidimensionality for mixed polytomous and dichotomous item data: Refinements of Poly-DIMTEST. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Francisco, CA.

11.

(2011). Assessing the dimensionality of polytomous item exams (Doctoral dissertation). Retrieved from http://scholarcommons.sc.edu/etd/2672

12.

Habing

Roussos

(2010, May). Conditional covariance-based subtest selection for polytomous item DIMTEST. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO

13.

Habing

Roussos

(2011, April). Improved conditional covariance-based subtest selection for polytomous DIMTEST. Paper presented at the annual meeting of the American Educational Research Association (AERA), New Orleans, LA.

14.

Nandakumar

Stout

(1998). Assessing unidimensionality of polytomous data. Applied Psychological Measurement, 22, 99-115.

15.

Reckase

M. D.

(1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401-412.

16.

Reckase

M. D.

(1997). A linear logistic multidimensional model for dichotomous item response data. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 271-286). New York, NY: Springer.

17.

Reckase

M. D.

(2009). Multidimensional Item Response Theory (Statistics for social and behavioral sciences). New York, NY: Springer.

18.

Reckase

M. D.

McKinley

R. L.

(1991). The discriminating power of items that measure more than one dimension. Applied Psychological Measurement, 15, 361-373.

19.

Roussos

L. A.

Stout

W. F.

Marden

J. I.

(1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35, 1-30.

20.

Samejima

(1997). Graded response model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 85-100). New York, NY: Springer.

21.

Seo

Roussos

L. A.

(2009, April). Evaluation of DIMTEST effect-size measure and its application. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Diego, CA.

22.

Seo

Roussos

L. A.

(2010). Formulation of a DIMTEST effect size measure (DESM) and evaluation of the DESM estimator bias. Journal of Educational Measurement, 47, 413-431.

23.

Seraphine

A. E.

(2000). The performance of DIMTEST when latent trait and item difficulty distributions differ. Applied Psychological Measurement, 42, 82-94.

24.

Stout

W. F.

(1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.

25.

Stout

W. F.

Froelich

A. G.

Gao

(2001). Using resampling methods to produce an improved DIMTEST procedure. In Boomsma

van Duijn

M. A. J.

Snijders

T. A. B.

(Eds.), Essays on item response theory (pp. 357-375). New York, NY: Springer.

26.

Stout

W. F.

Habing

Douglas

Kim

H. R.

Roussos

Zhang

(1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 19, 331-354.

27.

Tate

(2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203.

28.

Zhang

(2007). Conditional covariance theory and DETECT for polytomous items. Psychometrika, 72, 69-91.

29.

Zhang

Stout

W. F.

(1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152.

30.

Zhang

Stout

W. F.

(1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-219.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB

Conditional Covariance-Based Subtest Selection for Polytomous DIMTEST

Abstract

Keywords

Get full access to this article

References

Supplementary Material