A Comparison of Selected Empirical Methods for Assessing the Structure of Responses to Test Items

Abstract

Selected methods of empirically assessing the structure of tests with dichotomous items were compared. The methods included both exploratory and confirmatory procedures from two different families, those based on parametric models and nonparametric methods based on conditional item covariances. The analysis conditions considered were typical of large-scale assessments, for example, the tests were composed of a relatively large number of items, and it was assumed that a relatively large sample size would be available for analysis. Comparisons of the methods were conducted for real data from a 62-item test of reading ability and for computer-generated data for multiple unidimensional and multidimensional cases. For the most part, all methods performed reasonably well over a relatively wide range of conditions. The several exceptions to this outcome occurred when the test data departed appreciably from the assumptions or inherent limitations associated with a method, for example, when guessing was present but not allowed for in the analysis or when the multidimensional test structure was nonsimple but the goal of the method was to estimate the amount of multidimensional simple structure. Index terms: test structure, test dimensionality, local item dependencies, test factors.