Item response theory (IRT) models are the main psychometric approach for the development, evaluation, and refinement of multi-item instruments and scaling of latent traits, whereas multilevel models are the primary statistical method when considering the dependence between person responses when primary units (e.g., students) are nested within clusters (e.g., classes). This article introduces multilevel IRT (MLIRT) modeling, and provides the basic information to conduct, interpret, and report results based on an analysis using MLIRT modeling. The procedures are demonstrated using a sample data set based on the National Institute for the Evaluation of School System survey completed in Italy by fifth-grade students nested in classrooms to assess math achievement. The data and command files (Stata, Mplus, flexMIRT) needed to reproduce all analyses and plots in this article are available as supplemental online materials at http://jea.sagepub.com/supplemental.
*AdamsR. J.WilsonM.WuM. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22, 47-76.
2.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: MIT Press.
3.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459.
4.
BrowneW. J.Golalizadeh LahiM.ParkerR. M. A. (2009). A guide to sample size calculations for random effect models via simulation and the MLPowSim software package. Bristol, UK: University of Bristol.
5.
CaiL. (2009). High-dimensional exploratory item factor analysis by A Metropolis–Hastings Robbins–Monro algorithm. Psychometrika, 75, 33-57.
6.
CaiL. (2010). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307-335.
7.
CaiL. (2015). flexMIRT®: Flexible Multilevel Multidimensional Item Analysis and Test Scoring (Version 3.03) [Computer software]. Chapel Hill, NC. Vector Psychometric Group.
8.
ChoS.-J.SuhY. (2012). Bayesian analysis of item response models using WinBUGS 1.4.3. Applied Psychological Measurement, 36, 147-148.
9.
De AyalaR. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
10.
De AyalaR. J. (2013). Factor analysis with categorical indicators: Item response theory. In PetcscherY.SchatschneiderC.ComptonD. L. (Eds.), Applied quantitative analysis in education and the social sciences (pp. 208-242). New York, NY: Routledge.
11.
*De BoeckP.WilsonM. (Eds.). (2004). Item response models: A generalized linear and nonlinear approach (Statistics for social and behavioral sciences). New York, NY: Springer.
12.
**DraperD.GittoesM. (2004). Statistical analysis of performance indicators in UK higher education. Journal of the Royal Statistical Society, Series A, 167, 449-474.
13.
EdelenM. O.ReeveB. B. (2007). Applying item theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research, 16, 5-18. doi:10.1007/s11136-007-9198-0
14.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
15.
FisherG. H.MolenaarI. W. (1995). Rasch models, foundations, recent developments, and applications. New York, NY: Springer-Verlag.
16.
ForerB.ZumboB. D. (2011). Validation of multilevel constructs: Validation methods and empirical findings for the EDI. Social Indicators Research, 103, 231-265.
17.
*FoxJ.-P. (2007). Multilevel IRT modeling in practice with the package mlirt. Journal of Statistical Software, 20(5), 1-16.
18.
FoxJ.-P.GlasC. A. W. (2003). Bayesian modeling of measurement error in predictor variables using item response theory. Psychometrika, 68, 169-191.
19.
GlaserD.HastingsR. H. (2011). An introduction to multilevel modeling for anesthesiologists. Statistical Ground Rounds, 113, 877-887.
20.
GoldsteinH.HealyM. J. R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society A, 158, 175-177.
21.
**GoldsteinH.SpiegelhalterD. J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. Journal of Royal Statistical Society A, 159, 385-443.
22.
**GoldsteinH.ThomasS. (1996). Using examination results as indicators of school and college performance. Journal of the Royal Statistical Society A, 159, 97-114.
23.
GreenB. F.BockR. D.HumphreysL. G.LinnR. L.RechaseM. D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 32, 347-360.
24.
HayesA. F. (2006). A primer on multilevel modeling. Human Communication Research, 32, 385-410. doi:10.1111/j.1468-2958.2006.00281.x
25.
HoxJ. J. (2010). Multilevel analysis methods: Techniques and applications. New York, NY: Routledge.
26.
JakS.OortF.DolanC. V. (2014). Measurement bias in multilevel data. Structural Equation Modeling: A Multidisciplinary Journal, 21, 31-39.
27.
JeonM.Rabe-HeskethS. (2012). Profile likelihood approach for estimating generalized linear mixed models with factor structures. Journal of Educational and Behavioral Statistics, 37, 4, 518-542.
28.
JeonM.RijmenF. (2015). A modular approach for item response theory modeling with the R package flirt. Behavioral Research Methods. doi:10.3758/s13428-015-0606-z.
29.
**JiaoH.KamataA.WangS.JinY. (2012). A multilevel testlet model for dual local dependence. Journal of Educational Measurement, 49, 82-100.
30.
**KamataA. (2001). Item analysis by hierarchical linear model. Journal of Educational Measurement, 38, 79-93.
LinnR. L. (2009). The concept of validity in the context of NCLB. In LissitzR.W. (Ed.), The concept of validity: Revisions, new directions and applications (pp. 195-212). Charlotte, NC: IAP-Information Age Publishing.
MuthénB. O.SatorraA. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316.
38.
MuthénL. K.MuthénB. O. (1998-2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.
39.
**PastorD. A. (2003). The use of multilevel item response theory modeling in applied research: An illustration. Applied measurement in education, 16, 223-243.
40.
PeughJ. L. (2014). Conducting three-level cross-sectional analyses. The Journal of Early Adolescence, 34, 7-37.
Rabe-HeskethS.SkrondalA.PicklesA. (2004b). GLAMM manual (U.C. Berkeley Division of Biostatistics Working Paper Series, 160). Berkeley: University of California.
43.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Nielsen and Lydicke.
44.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: SAGE.
45.
RavandH. (2015). Item response theory using hierarchical generalized linear models. Practical Assessment Research & Evaluation, 20, 1-17.
46.
ReiseS. P.VenturaJ.NuechterleinK. H.KimK. H. (2005). An illustration of multilevel factor analysis. Journal of personality assessment, 84, 126-136.
47.
RijmenF.BriggsD. (2004). Multiple person dimensions and latent item predictors. In De BoeckP.WilsonM. (Eds), Explanatory item response models: A generalized linear and nonlinear approach (pp. 247-265). New York, NY: Springer.
48.
*RijmenF.TuerlinckxF.De BoeckP.KuppensP. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185-205.
49.
SingerJ. D. (1998). Using SAS Proc Mixed to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 23, 323-355. doi:10.3102/10769986023004323
*SkrondalA.Rabe-HeskethS. (2009). Prediction in multilevel generalized linear models. Journal of the Royal Statistical Society Series A, 172, 659-687.
52.
SnijdersT. A. B. (2005). Power and sample Size in multilevel linear models. In EverittB. S.HowellD. C. (Eds.), Encyclopedia of statistics in behavioral science (Vol. 3, pp. 1570-1573). Chichester, UK: Wiley.
53.
SnijdersT. A. B.BoskerR. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Thousand Oaks, CA: SAGE.
54.
StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: StataCorp LP.
TolandM. D. (2014). Practical guide to conducting an item response theory analysis. The Journal of Early Adolescence, 34, 120-151. doi:10.1177/0272431613511332
YenW. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.
59.
ZumboB. D.ForerB. (2011). Testing and measurement from a multilevel view: Psychometrics and validation. In BovairdJ.GeisingerK.BuckendahlC. (Eds.) High stakes testing in education—Science and practice in K-12 settings (Festschrift to Barbara Plake) (pp. 177-190). Washington, DC: American Psychological Press.
60.
ZwindermanA. H. (1991). A generalized rasch model for manifest predictors. Psychometrika, 56, 589-600.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
