This study explores the performance of the item response tree (IRTree) approach in modeling missing data, comparing its performance to the expectation–maximization (EM) algorithm and multiple imputation (MI) methods. Both simulation and empirical data were used to evaluate these methods across different missing data mechanisms, test lengths, sample sizes, and missing data proportions. Expected a posteriori was used for ability estimation, and bias and root mean square error (RMSE) were calculated. The findings indicate that IRTree provides more accurate ability estimates with lower RMSE than both EM and MI methods. Its overall performance was particularly strong under missing completely at random and missing not at random, especially with longer tests and lower proportions of missing data. However, IRTree was most effective with moderate levels of omitted responses and medium-ability test takers, though its accuracy decreased in cases of extreme omissions and abilities. The study highlights that IRTree is particularly well suited for low-stakes tests and has strong potential for providing deeper insights into the underlying missing data mechanisms within a data set.
AlagözÖ. E. C.MeiserT. (2024). Investigating heterogeneity in response strategies: A mixture multidimensional IRTree approach. Educational and Psychological Measurement, 84(5), 957–993. https://doi.org/10.1177/00131644231206765
3.
AlarconG. M.LeeM. A.JohnsonD. (2023). A Monte Carlo study of IRTree models’ ability to recover item parameters. Frontiers in Psychology, 14, Article 1003756. https://doi.org/10.3389/fpsyg.2023.1003756
4.
BakerF. B. (2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation.
5.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397–460). Addison-Wesley.
6.
BöckenholtU. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17, 665–678. https://doi.org/10.1037/a0028111
7.
Cetin-BerberD. D.SariH. I.Huggins-ManleyA. C. (2019). Imputation methods to deal with missing responses in computerized adaptive multistage testing. Educational and Psychological Measurement, 79(3), 495–511.
8.
ChalmersR. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
9.
CulbertsonM. (2011, April). Is it wrong? Handling missing responses in IRT [Speech presentation]. Annual Meeting of the National Council on Measurement in Education, New Orleans, LA, United States.
10.
CummingsP. (2013). Missing data and multiple imputation. JAMA Pediatrics, 167(7), 656–661. https://doi:10.1001/jamapediatrics.2013.1329
11.
De AyalaR. J.PlakeB. S.ImparaJ. C. (2001). The impact of omitted responses on the accuracy of ability estimation in item response theory. Journal of Educational Measurement, 38(3), 213–234. https://doi.org/10.1111/j.1745-3984.2001.tb01124.x
12.
DebeerD.JanssenR.De BoeckP. (2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54(3), 333–363. https://doi.org/10.1111/jedm.12147
13.
De BoeckP.PartchevI. (2012). IRTrees: Tree based item response models of the GLMM family. Journal of Statistical Software, 48, 1–28. https://doi.org/10.18637/jss.v048.c01
DibekM. I. (2019). Examination of the extreme response style of students using IRTree: The case of TIMMS 2015. International Journal of Assessment Tools in Education, 6, 300–313. https://doi.org/10.21449/ijate.534118
16.
EdwardsJ. M.FinchH. (2018). Recursive partitioning methods for data imputation in the context of item response theory: A Monte Carlo simulation. Psicológica Journal, 39, 88–117. https://doi.org/10.2478/psicolj-2018-0005
17.
EndersC. K. (2004). The impact of missing data on sample reliability estimates: Implications for reliability reporting practices. Educational and Psychological Measurement, 64(3), 419–436. https://doi.org/10.1177/0013164403261050
18.
EndersC. K. (2013). Dealing with missing data in developmental research. Child Development Perspectives, 7(1), 27–31. https://doi.org/10.1111/cdep.12008
19.
FeinbergR. A.RubrightJ. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36–49. https://doi.org/10.1111/emip.12111
20.
FinchH. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225–245. https://doi.org/10.1111/j.1745-3984.2008.00062.x
21.
GlasC. A. W.PimentelJ. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 48(6), 907–922. https://doi.org/10.1177/0013164408315262
22.
GlasC. A. W.PimentelJ. L.LamersS. M. A. (2015). Nonignorable data in IRT models: Polytomous models with covariates. Psychological Test and Assessment Modeling, 57(4), 523–541
23.
GrahamJ. W. (2012). Missing data analysis and design. Springer.
24.
HambletonR. K.SwaminathanH.RogersH. J. (1991). Fundamentals of item response theory. Sage.
25.
HolmanR.GlasC. A. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58, 1–17. https://doi.org/10.1111/j.2044-8317.2005.tb00312.x
26.
HuangH. Y. (2020). A mixture IRTree model for performance decline and nonignorable missing data. Educational and Psychological Measurement, 80(6), 1168–1195. https://doi.org/10.1177/0013164420914711
27.
JeonM.De BoeckP. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48, 1070–1085. https://doi.org/10.3758/s13428-015-0631-y
28.
JeonM.De BoeckP.van der LindenW. (2017). Modeling answer change behavior: An application of a generalized item response tree model. Journal of Educational and Behavioral Statistics, 42(4), 467–490. https://doi.org/10.3102/1076998616688015
29.
JeonM.RijmenF.Rabe-HeskethS. (2014). Flexible item response theory modeling with FLIRT. Applied Psychological Measurement, 38, 404–405. https://doi.org/10.1177/0146621614524982
30.
KhorramdelL.von DavierM. (2014). Measuring response styles across the big five: A multiscale extension of an approach using multinomial processing trees. Multivariate Behavioral Research, 49(2), 161–177. https://doi.org/10.1080/00273171.2013.866536
31.
KimN.BoltD. M. (2021). A mixture IRTree model for extreme response style: Accounting for response process uncertainty. Educational and Psychological Measurement, 81(1), 131–154. https://doi.org/10.1177/0013164420913915
32.
KöhlerC.PohlS.CarstensenC. (2017). Dealing with item nonresponse in large-scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54, 397–419. https://doi.org/10.1111/jedm.12154
33.
LeeJ. H.HuberJ. C., Jr. (2021). Evaluation of multiple imputation with large proportions of missing data: How much is too much?Iranian Journal of Public Health, 50(7), 1372–1380. https://doi.org/10.18502/ijph.v50i7.6626
34.
LeventhalB. C. (2019). Extreme response style: A simulation study comparison of three multidimensional item response models. Applied Psychological Measurement, 43(4), 322–335. https://doi.org/10.1177/0146621618789392
35.
LittleR. J. A.RubinD. B. (1987). Statistical analysis with missing data. John Wiley & Sons.
36.
LittleR. J. A.RubinD. B. (2002). Statistical analysis with missing data. John Wiley & Sons.
37.
LittleT. D.LangK. M.WuW.RhemtullaM. (2016). Developmental psychopathology. In CicchettiD. (Ed.), Missing data (pp. 760–797). John Wiley & Sons.
38.
McKnightP. E.McKnightK. M.SidaniS.FigueredoA. J. (2007). Missing data: A gentle introduction. Guilford Press.
39.
MeiserT.ReiberF. (2023). Item-specific factors in IRTree models: When they matter and when they don’t. Psychometrika, 88, 739–744. https://doi.org/10.1177/00131644231206765
40.
Organisation for Economic Co-operation and Development. (2019). PISA 2018 results (volume I): What students know and can do. https://doi.org/10.1787/5f07c754-en
41.
ParkM.WuA. D. (2019). Item response tree models to investigate acquiescence and extreme response styles in Likert-type rating scales. Educational and Psychological Measurement, 79(5), 911–930. https://doi.org/10.1177/0013164419829855
42.
PeughJ. L.EndersC. K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74(4), 525–556. https://doi.org/10.3102/00346543074004525
43.
PlieningerH. (2021). Developing and applying IR-Tree models: Guidelines, caveats, and an extension to multiple groups. Organizational Research Methods, 24(3), 654–670. https://doi.org/10.1177/1094428120911096
44.
PohlS.BeckerB. (2020). Performance of missing data approaches under nonignorable missing data conditions. Methodology, 16(2), 147–165. https://doi.org/10.5964/meth.2805
45.
PohlS.GräfeL.RoseN. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74(3), 423–452. https://doi.org/10.1177/0013164413504926
RoseN.von DavierM.XuX. (2010). Modeling nonignorable missing data with item response theory [IRT] (ETS Research Rep. no. RR-10-11). Educational Testing Service.
RubinD. B. (1987). Multiple imputation for nonresponse in surveys. John Wiley & Sons.
52.
SchaferJ. L. (1997). Analysis of incomplete multivariate data. CRC Press.
53.
SoğuksuY. B. (2024). The efficacy of the IRTree framework for detecting missing data mechanisms in educational assessments. Journal of Measurement and Evaluation in Education and Psychology, 15(3), 209–220. https://doi.org/10.21031/epod.1514741
54.
TabachnickB. G.FidellL. S. (2007). Using multivariate statistics. Allyn and Bacon.
55.
ThissenD.SteinbergL.GerrardM. (1986). Beyond group-mean differences: The concept of item bias. Psychological Bulletin, 99(1), 118–128. https://doi.org/10.1037/0033-2909.99.1.118
56.
Van BuurenS.Groothuis-OudshoornK. (2011). Mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1–67. https://doi.org/10.18637/jss.v045.i03
57.
XiaoJ.BulutO. (2020). Evaluating the performances of missing data handling methods in ability estimation from sparse data. Educational and Psychological Measurement, 80(5), 932–954. https://doi.org/10.1177/0013164420911136
58.
YenW. M. (1984). Effects of local item dependence on the fit and equating performance of the three parameter logistic model. Applied Psychological Measurement, 8, 125–145. https://doi.org/10.1177/014662168400800201
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
