If data exhibit multidimensionality, key conditional independence assumptions of unidimensional models do not hold. The current work pursues posterior predictive model checking, a flexible family of model-checking procedures, as a tool for criticizing models due to unaccounted for dimensions in the context of item response theory. Factors hypothesized to influence dimensionality and dimensionality assessment are couched in conditional covariance theory and conveyed via geometric representations of multidimensionality. A simulation study investigates the performance of the model-checking tools for dichotomous observables. Key findings include support for the hypothesized effects of the manipulated factors with regard to their influence on dimensionality assessment and the superiority of certain discrepancy measures for conducting posterior predictive model checking for dimensionality assessment.
Agresti, A. ( 2002). Categorical data analysis (2nd ed.). New York: Wiley.
2.
Berkhof, J., van Mechelen, I., & Gelman, A. ( 2004). Enhancing the performance of a posterior predictive check (IAP Statistics Network Tech. Rep. 0350). Louvain-la-Neuve , Belgium: Interuniversity Attraction Pole, Katholieke Universiteit Leuven.
3.
Bolt, D., & Stout, W. ( 1996). Differential item functioning: Its multidimensional model and resulting SIBT-EST detection procedure. Behaviormetrika , 23, 67-95.
4.
Bradlow, E.T., Wainer, H., & Wang, X. ( 1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168.
5.
Chen, W., & Thissen, D. ( 1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.
6.
Donoghue, J.R., & Hombo, C.M. ( 2003, April). A corrected asymptotic distribution of an IRT fit measure that accounts for the effects of item parameter estimation. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
7.
Fu, J., Bolt, D.M., & Li, Y. ( 2005, April). Evaluating item fit for a polytomous fusion model using posterior predictive checks. Paper presented at the annual meeting of the National Council on Measurement in Education , Montreal, Quebec, Canada.
8.
Gelman, A., Meng, X.L., & Stern, H. ( 1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6, 733-807.
9.
Habing, B., & Roussos, L.A. ( 2003). On the need for local item dependence. Psychometrika, 68, 435-451.
10.
Hoijtink, H. ( 2001). Conditional independence and differential item functioning in the two-parameter logistic model. In A. Boomsma, M. A. J. van Duijn, & T. A. B. Snijders (Eds.), Essays in item response theory (pp. 109-130). New York: Springer-Verlag.
11.
Janssen, R., Tuerlinckx, F., Meulders, M., & De Boeck, P. ( 2000). A hierarchical IRT model for criterion-referenced measurement . Journal of Educational and Behavioral Statistics, 25, 285-306.
12.
Levy, R., Mislevy, R.J., & Sinharay, S. ( 2006, April). Posterior predictive model checking for multidimensionality in item response theory. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.
13.
McDonald, R.P., & Mok, M.M.C. (1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30, 23-40.
Murdoch, D.J., & Chow, E.D. ( 1996). A graphical display of large correlation matrices. American Statistician, 50, 178-180.
16.
Patz, R.J., & Junker B.W. ( 1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146-178.
17.
Reckase, M.D. ( 1997). A linear logistic multidimensional model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 271-286). New York: Springer-Verlag .
18.
Reckase, M.D., & McKinley, R.L. (1991). The discriminating power of test items that measure more than one ability. Applied Psychological Measurement, 15, 361-373.
19.
Robins, J.M., van der Vaart, A., & Ventura, V. ( 2000). The asymptotic distribution of P values in composite null models. Journal of the American Statistical Association , 95, 1143-1172.
20.
Rubin, D.B. ( 1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. Annals of Statistics, 12, 1151-1172.
Sinharay, S. ( 2005). Assessing fit of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement , 42, 375-394.
23.
Sinharay, S. ( 2006). Bayesian item fit analysis for unidimensional item response theory models. British Journal of Mathematical and Statistical Psychology, 59, 429-449.
24.
Sinharay, S., & Johnson, M. ( 2003, October). Simulation studies applying posterior predictive model checking for assessing fit of the common item response theory models (ETS RR-03-28). Princeton, NJ: Educational Testing Service.
25.
Sinharay, S., Johnson, M., & Stern, H.S. ( 2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30, 298-321.
26.
Sinharay, S., & Stern, H.S. ( 2003). Posterior predictive model checking in hierarchical models . Journal of Statistical Planning and Inference, 111, 209-221.
27.
Stern, H.S. ( 2000). Comment. Journal of the American Statistical Association, 95, 1157-1159.
28.
Stout, W., Habing, B., Douglas, J., Kim, H.R., Roussos, L., & Zhang, J. ( 1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331-354.
29.
Yen, W. ( 1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement , 30, 187-213.
30.
Zhang, J., & Stout, W. ( 1999). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152.
