Second-order item response theory models have been used for assessments consisting of several domains, such as content areas. We extend the second-order model to a third-order model for assessments that include subdomains nested in domains. Using a graphical model framework, it is shown how the model does not suffer from the curse of multidimensionality. We apply unidimensional, second-order, and third-order item response models to the 2007 Trends in International Mathematics and Science Study. Our findings suggest that deviations from unidimensionality are more pronounced at the content domain level than at the cognitive domain level and that deviations from unidimensionality at the content domain level become negligible after taking into account topic areas.
BentlerP. M. (1976). Multistructure statistical model applied to factor analysis. Multivariate Behavioral Research, 11, 3–25.
2.
BickleyP. G.KeithK. Z.WolfeL. M (1995). The three-stratum theory of cognitive abilities: Test of the structure of intelligence across the life span. Intelligence, 20, 309–328.
3.
BishopC. M. (2006). Pattern recognition and machine learning. New York, NY: Springer-Verlag.
4.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: An application of the EM algorithm. Psychometrika, 46, 443–459.
5.
BradlowE. T.WainerH.WangX. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153–168.
6.
BreslowN. E.ClaytonD. G. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88, 9–25.
7.
CarrollJ. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge, England: Cambridge University Press.
8.
CowellR. G.DawidA. P.LauritzenS. L.SpiegelhalterD. J. (1999). Probabilistic networks and expert systems. New York, NY: Springer.
9.
de la TorreJ.SongH. (2009). Simultaneous estimation of overall and domain abilities: A higher order IRT model approach. Applied Psychological Measurement, 33, 620–639.
10.
FoxJ.-P.GlasC. A. W. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 271–288.
11.
FoyP.OlsonA. M. (2009). TIMSS 2007 international database and user guide. Chestnut Hill, MA: Boston College, TIMSS & PIRLS International Study Center.
GoldsteinH.RasbashJ. (1996). Improved approximations for multilevel models with binary responses. Journal of the Royal Statistical Society, Series A, 159, 505–513.
14.
HabermanS. J. (2008). When can subscores have value?Journal of Educational and Behavioral Statistics, 33, 204–229.
15.
HabermanS. J.SinharayS. (2010). Reporting of subscores using multidimensional item response theory. Psychometrika, 75, 209–227.
16.
JeonM.RijmenF.Rabe-HeskethS. (2013). Modeling differential item functioning using the multiple-group bifactor model. Journal of Educational and Behavioral Statistics, 38, 32–60.
17.
JöreskogK. G. (1970). A general method for analysis of covariance structures. Biometrika, 57, 239–251.
18.
JöreskogK. G. (1994). On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59, 381–389.
19.
LauritzenS. L. (1995). The EM algorithm for graphical association models with missing data. Computational Statistics and Data Analysis, 19, 191–201.
20.
LiY.BoltD. M.FuJ. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30, 3–21.
21.
MastersG. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.
22.
McLachlanG.KrishnanT. (1997). The EM algorithm and extensions. New York, NY: Wiley.
23.
MeilijsonI. (1989). A fast improvement to the EM algorithm on its own terms. Journal of the Royal Statistical Society, B, 51, 127–138.
24.
MislevyR. J. (1985). Recent developments in the factor analysis of categorical variables(ETS Research Rep. No. RR-85-24). Princeton, NJ: ETS.
25.
MuthénB. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49, 115–132.
26.
PinheiroP. C.BatesD. M. (1995). Approximations to the log-likelihood function in the nonlinear mixed-effects model. Journal of Computational and Graphical Statistics, 4, 12–35.
27.
Rabe-HeskethS.SkrondalA.PicklesA. (2005). Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects. Journal of Econometrics, 128, 301–323.
28.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.
29.
RaudenbushS. W.YangM.-L.YosefM. (2000). Maximum likelihood for generalized linear models with nested random effects via high-order, multivariate Laplace approximation. Journal of Computational and Graphical Statistics, 9, 141–157.
30.
RijmenF. (2006). BNL: A Matlab toolbox for Bayesian networks with logistic regression nodes(Technical Report). Amsterdam, the Netherlands: VU University Medical Center.
31.
RijmenF. (2010). Formal relations and an empirical comparison between the bi-factor, the testlet, and a second order multidimensional IRT model. Journal of Educational Measurement, 47, 361–372.
32.
RijmenF. (2011). The latent class model as a measurement model for situational judgment tests. Psychologica Belgica, 51, 197–212.
33.
RijmenF.TuerlinckxF.De BoeckP.KuppensP. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185–205.
34.
RijmenF.VansteelandtK.De BoeckP. (2008). Latent class models for diary method data: Parameter estimation by local computations. Psychometrika, 73, 167–182.
35.
RutkowskiL.GonzalezE.JoncasM.von DavierM. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39, 142–151.
36.
SchmidJ.LeimanJ. M. (1957). The development of hierarchical factor solutions. Psychometrika, 22, 53–61.
37.
TaubG. E. (2001). A confirmatory analysis of the Wechsler adult intelligence scale-third edition: Is the verbal/performance discrepancy justified?Practical Assessment, Research & Evaluation, 7. Retrieved fromhttp://PAREonline.net/getvn.asp?v=7&n=22
38.
ThurstoneL. L. (1944). Second-order factors. Psychometrika, 9, 71–100.
39.
von DavierM.SinharayS. (2007). An importance sampling EM algorithm for latent regression models. Journal of Educational and Behavioral Statistics, 32, 233–251.
40.
von DavierM.SinharayS. (2010). Stochastic approximation for latent regression item response models. Journal of Educational and Behavioral Statistics, 35, 174–193.
41.
WainerH.VeveaJ. L.CamachoF.ReeveB. B.RosaK.NelsonL. (2001). Augmented scores—“Borrowing strength” to compute scores based on small numbers of items. In ThissenD.WainerH. (Eds.), Test scoring (pp. 343–387). Hillsdale, MI: Erlbaum.
42.
YungY.-F.ThissenD.McLeodL. D. (1999). On the relationship between the higher order factor model and the hierarchical factor model. Psychometrika, 64, 113–128.
