So far, modeling approaches for not-reached items have considered one single underlying process. However, missing values at the end of a test can occur for a variety of reasons. On the one hand, examinees may not reach the end of a test due to time limits and lack of working speed. On the other hand, examinees may not attempt all items and quit responding due to, for example, fatigue or lack of motivation. We use response times retrieved from computerized testing to distinguish missing data due to lack of speed from missingness due to quitting. On the basis of this information, we present a new model that allows to disentangle and simultaneously model different missing data mechanisms underlying not-reached items. The model (a) supports a more fine-grained understanding of the processes underlying not-reached items and (b) allows to disentangle different sources describing test performance. In a simulation study, we evaluate estimation of the proposed model. In an empirical study, we show what insights can be gained regarding test-taking behavior using this model.
AlvarezI.NiemiJ.SimpsonM. (2014). Bayesian inference for a covariance matrix. Retrieved from https://arxiv.org/abs/1408.4050
2.
BakerS. G.FitzmauriceG. M.FreedmanL. S.KramerB. S. (2005). Simple adjustments for randomized trials with nonrandomly missing or censored outcomes arising from informative covariates. Biostatistics, 7, 29-40. doi:10.1093/biostatistics/kxi038
3.
ChenH. H.von DavierM.YamamotoK.KongN. (2015). Comparing data treatments on item-level nonresponse and their effects on data analysis of large-scale assessments: 2009 PISA study (ETS Research Report No. RR-15-12). Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12059
4.
CosgroveJ. (2011, December). Does student engagement explain performance on PISA? Comparisons of response patterns on the PISA tests across time. Dublin, Ireland: Educational Research Centre. Retrieved from http://www.erc.ie/documents/engagement_and_performance_over_time.pdf
5.
DanielsM. J.KassR. E. (1999). Nonconjugate Bayesian estimation of covariance matrices and its use in hierarchical models. Journal of the American Statistical Association, 94, 1254-1263. doi:10.1080/01621459.1999.10473878
6.
DebeerD.JanssenR.BoeckP. (2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54, 333-363. doi:10.1111/jedm.12147
7.
DoeblerA.HollingH. (2016). A processing speed test based on rule-based item generation: An analysis with the Rasch Poisson counts model. Learning and Individual Differences, 52, 121-128. doi:10.1016/j.lindif.2015.01.013
8.
FoxJ.-P. (2010). Bayesian item response modeling: Theory and applications. New York, NY: Springer.
9.
FoxJ.-P.MariantiS. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51, 540-553. doi:10.1080/00273171.2016.1171128
FoyP. (2018). PIRLS 2016 user guide for the international database. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and International Association for the Evaluation of Educational Achievement (IEA). Retrieved from https://timssandpirls.bc.edu/pirls2016/international-database/downloads/P16_UserGuide.pdf
12.
GelmanA.HillJ. (2007). Data analysis using regression and multilevel hierarchical models. Cambridge, England: Cambridge University Press.
13.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457-472. doi:10.1214/ss/1177011136
14.
GelmanA.ShirleyK. (2011). Inference from simulations and monitoring convergence. In BrooksS.GelmanA.JonesG.MengX.-L. (Eds.), Handbook of Markov chain Monte Carlo (pp. 163-174). Boca Raton, FL: Chapman & Hall.
15.
GlasC. A.PimentelJ. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68, 907-922. doi:10.1177/0013164408315262
16.
GonzalezE.RutkowskiL. (2010). Principles of multiple matrix booklet designs and parameter recovery in large-scale assessments. In von DavierM.HastedtD. (Eds.), IERI monograph series: Issues and methodologies in large-scale assessments (Vol. 3, pp. 125-156). Hamburg, Germany: IEA-ETS Research Institute.
17.
JansenM. G. H. (1994). Parameters of the latent distribution in Rasch’s Poisson counts model. In FischerC.LamingD. (Eds.), Contributions to mathematical psychology, psychometrics, and methodology (pp. 319-326). New York, NY: Springer.
18.
JansenM. G. H. (1995). The Rasch Poisson counts model for incomplete data: An application of the EM algorithm. Applied Psychological Measurement, 19, 291-302. doi:10.1177/014662169501900307
19.
Klein-EntinkR. H.KuhnJ.-T.HornkeL. F.FoxJ.-P. (2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14, 54-75. doi:10.1037/a0014877
20.
Klein-EntinkR.van der LindenW.FoxJ.-P. (2009). A Box-Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62, 621-640. doi:10.1348/000711008X374126
21.
KöhlerC.PohlS.CarstensenC. H. (2017). Dealing with item nonresponse in large-scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54, 397-419. doi:10.1111/jedm.12154
22.
LawrenceI. M. (1993). The effect of test speededness on subgroup performance (ETS Research Report No. RR-93-49). Princeton, NJ: Educational Testing Service. doi:10.1002/j.2333-8504.1993.tb01560.x
23.
ListM. K.KöllerO.NagyG. (2017). A semiparametric approach for modeling not-reached items. Educational and Psychological Measurement. Advance online publication. doi:10.1177/0013164417749679
24.
LiuO. L.RiosJ. A.BordenV. (2015). The effects of motivational instruction on college students’ performance on low-stakes assessment. Educational Assessment, 20, 79-94. doi:10.1080/10627197.2015.1028618
25.
LordF. M. (1983). Maximum likelihood estimation of item response parameters when some responses are omitted. Psychometrika, 48, 477-482. doi:10.1007/BF02293689
26.
MandinachE. B.BridgemanB.Cahalan-LaitusisC.TrapaniC. (2005). The impact of extended time on SAT®test performance (Research Report No. 2005-8). New York, NY: College Board. Retrieved from https://files.eric.ed.gov/fulltext/ED563027.pdf
27.
MislevyR. J.WuP.-K. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing (ETS Research Report No. RR-96-30-ONR). Princeton, NJ: Educational Testing Service. doi:10.1002/j.2333-8504.1996.tb01708.x
28.
MolenaarD.TuerlinckxF.van der MaasH. L. (2015). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68, 197-219. doi:10.1111/bmsp.12042
29.
MuthénL. K.MuthénB. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling: A Multidisciplinary Journal, 9, 599-620. doi:10.1207/S15328007SEM0904\_8
30.
O’MuircheartaighC.MoustakiI. (1999). Symmetric pattern models: A latent variable approach to item non-response in attitude scales. Journal of the Royal Statistical Society: Series A (Statistics in Society), 162, 177-194. doi:10.1111/1467-985X.00129
PlummerM. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In HornikK.LeischF.ZeileisA. (Eds.), Proceedings of the Third International Workshop on Distributed Statistical Computing (pp. 1-10). Vienna, Austria: Austrian Association for Statistical Computing.
PohlS.CarstensenC. H. (2012). NEPS technical report-Scaling the data of the competence tests (NEPS Working Paper No. 14). Bamberg, Germany: Otto-Friedrich-Universität, Nationales Bildungspanel.
37.
PohlS.GräfeL.RoseN. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74, 423-452. doi:10.1177/0013164413504926
38.
PohlS.UlitzschE.von DavierM. (2019). Using response time models to account for not-reached items. Psychometrika, 84(3), 892-920. doi:10.1007/s11336-019-09669-2
39.
R Development Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
40.
RoseN. (2013). Item nonresponses in educational and psychological measurement (Unpublished doctoral dissertation). Friedrich-Schiller-Universität Jena, Jena, Thuringia, Germany. Retrieved from https://d-nb.info/1036873145/34
41.
RoseN.von DavierM.NagengastB. (2017). Modeling omitted and not-reached items in IRT models. Psychometrika, 82, 795-819. doi:10.1007/s11336-016-9544-7
42.
RoseN.von DavierM.XuX. (2010). Modeling nonignorable missing data with item response theory (IRT) (ETS Research Report No. RR-10-11). Princeton, NJ: Educational Testing Service. doi:10.1002/j.2333-8504.2010.tb02218.x
43.
RubinD. B. (1976). Inference and missing data. Biometrika, 63, 581-592. doi:10.1093/biomet/63.3.581
44.
SchuurmanN.GrasmanR.HamakerE. (2016). A comparison of inverse-Wishart prior specifications for covariance matrices in multilevel autoregressive models. Multivariate Behavioral Research, 51, 185-206. doi:10.1080/00273171.2015.1065398
45.
UlitzschE.von DavierM.PohlS. (2019). Using response times for joint modeling of response and omission behavior. Multivariate Behavioral Research, 1-29. doi:10.1080/00273171.2019.1643699
46.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181-204. doi:10.3102/10769986031002181
47.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287-308. doi:10.1007/s11336-006-1478-z
48.
van ErpS.MulderJ.OberskiD. L. (2017). Prior sensitivity analysis in default Bayesian structural equation modeling. Psychological Methods, 23, 363-388. doi:10.1037/met0000162
49.
VenablesW. N.RipleyB. D. (2002). Modern applied statistics with S (4th ed.). New York, NY: Springer. Retrieved from http://www.stats.ox.ac.uk/pub/MASS4
50.
WildC.DursoR. (1979, June). Effect of increased test-taking time on test scores by ethnic groups, age, and sex. Princeton, NJ: Educational Testing Service. Retrieved from https://www.ets.org/Media/Research/pdf/GREB-76-06R.pdf
51.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1-17. doi:10.1207/s15326977ea1001\_1
52.
YamamotoK.EversonH. (1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model. In RostJ. (Ed.), Applications of latent trait and latent class models in the social sciences (pp. 89-98). Münster, Germany: Waxmann.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
