A Multiprocess Item Response Model for Not-Reached Items due to Time Limits and Quitting

Abstract

So far, modeling approaches for not-reached items have considered one single underlying process. However, missing values at the end of a test can occur for a variety of reasons. On the one hand, examinees may not reach the end of a test due to time limits and lack of working speed. On the other hand, examinees may not attempt all items and quit responding due to, for example, fatigue or lack of motivation. We use response times retrieved from computerized testing to distinguish missing data due to lack of speed from missingness due to quitting. On the basis of this information, we present a new model that allows to disentangle and simultaneously model different missing data mechanisms underlying not-reached items. The model (a) supports a more fine-grained understanding of the processes underlying not-reached items and (b) allows to disentangle different sources describing test performance. In a simulation study, we evaluate estimation of the proposed model. In an empirical study, we show what insights can be gained regarding test-taking behavior using this model.

Keywords

response times not-reached items missing data item response theory survival

Get full access to this article

View all access options for this article.

References

Alvarez

Niemi

Simpson

(2014). Bayesian inference for a covariance matrix. Retrieved from https://arxiv.org/abs/1408.4050

Baker

S. G.

Fitzmaurice

G. M.

Freedman

L. S.

Kramer

B. S.

(2005). Simple adjustments for randomized trials with nonrandomly missing or censored outcomes arising from informative covariates. Biostatistics, 7, 29-40. doi:10.1093/biostatistics/kxi038

Chen

H. H.

von Davier

Yamamoto

Kong

(2015). Comparing data treatments on item-level nonresponse and their effects on data analysis of large-scale assessments: 2009 PISA study (ETS Research Report No. RR-15-12). Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12059

Cosgrove

(2011, December). Does student engagement explain performance on PISA? Comparisons of response patterns on the PISA tests across time. Dublin, Ireland: Educational Research Centre. Retrieved from http://www.erc.ie/documents/engagement_and_performance_over_time.pdf

Daniels

M. J.

Kass

R. E.

(1999). Nonconjugate Bayesian estimation of covariance matrices and its use in hierarchical models. Journal of the American Statistical Association, 94, 1254-1263. doi:10.1080/01621459.1999.10473878

Debeer

Janssen

Boeck

(2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54, 333-363. doi:10.1111/jedm.12147

Doebler

Holling

(2016). A processing speed test based on rule-based item generation: An analysis with the Rasch Poisson counts model. Learning and Individual Differences, 52, 121-128. doi:10.1016/j.lindif.2015.01.013

Fox

J.-P.

(2010). Bayesian item response modeling: Theory and applications. New York, NY: Springer.

Fox

J.-P.

Marianti

(2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51, 540-553. doi:10.1080/00273171.2016.1171128

10.

Foy

(2017). TIMSS 2015 user guide for the international database. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and International Association for the Evaluation of Educational Achievement. Retrieved from https://timssandpirls.bc.edu/timss2015/international-database/downloads/T15_UserGuide.pdf

11.

Foy

(2018). PIRLS 2016 user guide for the international database. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and International Association for the Evaluation of Educational Achievement (IEA). Retrieved from https://timssandpirls.bc.edu/pirls2016/international-database/downloads/P16_UserGuide.pdf

12.

Gelman

Hill

(2007). Data analysis using regression and multilevel hierarchical models. Cambridge, England: Cambridge University Press.

13.

Gelman

Rubin

D. B.

(1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457-472. doi:10.1214/ss/1177011136

14.

Gelman

Shirley

(2011). Inference from simulations and monitoring convergence. In Brooks

Gelman

Jones

Meng

X.-L.

(Eds.), Handbook of Markov chain Monte Carlo (pp. 163-174). Boca Raton, FL: Chapman & Hall.

15.

Glas

C. A.

Pimentel

J. L.

(2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68, 907-922. doi:10.1177/0013164408315262

16.

Gonzalez

Rutkowski

(2010). Principles of multiple matrix booklet designs and parameter recovery in large-scale assessments. In von Davier

Hastedt

(Eds.), IERI monograph series: Issues and methodologies in large-scale assessments (Vol. 3, pp. 125-156). Hamburg, Germany: IEA-ETS Research Institute.

17.

Jansen

M. G. H.

(1994). Parameters of the latent distribution in Rasch’s Poisson counts model. In Fischer

Laming

(Eds.), Contributions to mathematical psychology, psychometrics, and methodology (pp. 319-326). New York, NY: Springer.

18.

Jansen

M. G. H.

(1995). The Rasch Poisson counts model for incomplete data: An application of the EM algorithm. Applied Psychological Measurement, 19, 291-302. doi:10.1177/014662169501900307

19.

Klein-Entink

R. H.

Kuhn

J.-T.

Hornke

L. F.

Fox

J.-P.

(2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14, 54-75. doi:10.1037/a0014877

20.

Klein-Entink

van der Linden

Fox

J.-P.

(2009). A Box-Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62, 621-640. doi:10.1348/000711008X374126

21.

Köhler

Pohl

Carstensen

C. H.

(2017). Dealing with item nonresponse in large-scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54, 397-419. doi:10.1111/jedm.12154

22.

Lawrence

I. M.

(1993). The effect of test speededness on subgroup performance (ETS Research Report No. RR-93-49). Princeton, NJ: Educational Testing Service. doi:10.1002/j.2333-8504.1993.tb01560.x

23.

List

M. K.

Köller

Nagy

(2017). A semiparametric approach for modeling not-reached items. Educational and Psychological Measurement. Advance online publication. doi:10.1177/0013164417749679

24.

Liu

O. L.

Rios

J. A.

Borden

(2015). The effects of motivational instruction on college students’ performance on low-stakes assessment. Educational Assessment, 20, 79-94. doi:10.1080/10627197.2015.1028618

25.

Lord

F. M.

(1983). Maximum likelihood estimation of item response parameters when some responses are omitted. Psychometrika, 48, 477-482. doi:10.1007/BF02293689

26.

Mandinach

E. B.

Bridgeman

Cahalan-Laitusis

Trapani

(2005). The impact of extended time on SAT®test performance (Research Report No. 2005-8). New York, NY: College Board. Retrieved from https://files.eric.ed.gov/fulltext/ED563027.pdf

27.

Mislevy

R. J.

P.-K.

(1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing (ETS Research Report No. RR-96-30-ONR). Princeton, NJ: Educational Testing Service. doi:10.1002/j.2333-8504.1996.tb01708.x

28.

Molenaar

Tuerlinckx

van der Maas

H. L.

(2015). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68, 197-219. doi:10.1111/bmsp.12042

29.

Muthén

L. K.

Muthén

B. O.

(2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling: A Multidisciplinary Journal, 9, 599-620. doi:10.1207/S15328007SEM0904\_8

30.

O’Muircheartaigh

Moustaki

(1999). Symmetric pattern models: A latent variable approach to item non-response in attitude scales. Journal of the Royal Statistical Society: Series A (Statistics in Society), 162, 177-194. doi:10.1111/1467-985X.00129

31.

Organisation for Economic Co-operation and Development. (2013). Technical report of the survey of adult skills (PIAAC). Paris, France: Author. Retrieved from https://www.oecd.org/skills/piaac/_Technical%20Report_17OCT13.pdf

32.

Organisation for Economic Co-operation and Development. (2014). PISA 2012 technical report. Paris, France: Author. Retrieved from https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf

33.

Organisation for Economic Co-operation and Development. (2017). PISA 2015 technical report. Paris, France: Author. Retrieved from https://www.oecd.org/pisa/sitedocument/PISA-2015-technical-report-final.pdf

34.

Plummer

(2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Hornik

Leisch

Zeileis

(Eds.), Proceedings of the Third International Workshop on Distributed Statistical Computing (pp. 1-10). Vienna, Austria: Austrian Association for Statistical Computing.

35.

Plummer

(2016). rjags: Bayesian graphical models using MCMC (R package Version 4-6). Retrieved from https://CRAN.R-project.org/package=rjags

36.

Pohl

Carstensen

C. H.

(2012). NEPS technical report-Scaling the data of the competence tests (NEPS Working Paper No. 14). Bamberg, Germany: Otto-Friedrich-Universität, Nationales Bildungspanel.

37.

Pohl

Gräfe

Rose

(2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74, 423-452. doi:10.1177/0013164413504926

38.

Pohl

Ulitzsch

von Davier

(2019). Using response time models to account for not-reached items. Psychometrika, 84(3), 892-920. doi:10.1007/s11336-019-09669-2

39.

R Development Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org

40.

Rose

(2013). Item nonresponses in educational and psychological measurement (Unpublished doctoral dissertation). Friedrich-Schiller-Universität Jena, Jena, Thuringia, Germany. Retrieved from https://d-nb.info/1036873145/34

41.

Rose

von Davier

Nagengast

(2017). Modeling omitted and not-reached items in IRT models. Psychometrika, 82, 795-819. doi:10.1007/s11336-016-9544-7

42.

Rose

von Davier

(2010). Modeling nonignorable missing data with item response theory (IRT) (ETS Research Report No. RR-10-11). Princeton, NJ: Educational Testing Service. doi:10.1002/j.2333-8504.2010.tb02218.x

43.

Rubin

D. B.

(1976). Inference and missing data. Biometrika, 63, 581-592. doi:10.1093/biomet/63.3.581

44.

Schuurman

Grasman

Hamaker

(2016). A comparison of inverse-Wishart prior specifications for covariance matrices in multilevel autoregressive models. Multivariate Behavioral Research, 51, 185-206. doi:10.1080/00273171.2015.1065398

45.

Ulitzsch

von Davier

Pohl

(2019). Using response times for joint modeling of response and omission behavior. Multivariate Behavioral Research, 1-29. doi:10.1080/00273171.2019.1643699

46.

van der Linden

W. J

. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181-204. doi:10.3102/10769986031002181

47.

van der Linden

W. J

. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287-308. doi:10.1007/s11336-006-1478-z

48.

van Erp

Mulder

Oberski

D. L

. (2017). Prior sensitivity analysis in default Bayesian structural equation modeling. Psychological Methods, 23, 363-388. doi:10.1037/met0000162

49.

Venables

W. N.

Ripley

B. D.

(2002). Modern applied statistics with S (4th ed.). New York, NY: Springer. Retrieved from http://www.stats.ox.ac.uk/pub/MASS4

50.

Wild

Durso

(1979, June). Effect of increased test-taking time on test scores by ethnic groups, age, and sex. Princeton, NJ: Educational Testing Service. Retrieved from https://www.ets.org/Media/Research/pdf/GREB-76-06R.pdf

51.

Wise

S. L.

DeMars

C. E.

(2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1-17. doi:10.1207/s15326977ea1001\_1

52.

Yamamoto

Everson

(1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model. In Rost

(Ed.), Applications of latent trait and latent class models in the social sciences (pp. 89-98). Münster, Germany: Waxmann.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB