Recent studies show increasing interest in using process data (e.g., response time, response actions) to enhance measurement accuracy for respondents’ latent traits. Yet, few have explored the possibility of incorporating process information into cognitive diagnostic models (CDMs). This study proposes a novel CDM approach that utilizes a four-component joint modeling approach with response action sequences (i.e., similarity and efficiency), response time, and item responses. We employed the Markov Chain Monte Carlo method for parameter estimation and evaluated the performance of the proposed model using both an empirical study and two simulation studies. The results suggest that the process data can improve respondents’ classification accuracy under varied conditions and support the interpretation of the association between process and response data.
AlmondR. G.MislevyR. J.SteinbergL. S.YanD.WilliamsonD. M.AlmondR. G.MislevyR. J.SteinbergL. S.YanD.WilliamsonD. M. (2015). The future of Bayesian networks in educational assessment. In FienbergS. E. (Eds.), Bayesian networks in educational assessment (pp. 583–599). Springer. https://doi.org/10.1007/978-1-4939-2125-6_16
2.
BalotaD. A.AschenbrennerA. J.YapM. J. (2013). Additive effects of word frequency and stimulus quality: The influence of trial history and data transformations. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(5), 1563–1571. https://doi.org/10.1037/a0032186
3.
BarnardJ.McCullochR.MengX.-L. (2000). Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage. Statistica Sinica, 10, 1281–1311.
4.
BergnerY.von DavierA. A. (2019). Process data in NAEP: Past, present, and future. Journal of Educational and Behavioral Statistics, 44(6), 706–732. https://doi.org/10.3102/1076998618784700
BolsinovaM.de BoeckP.TijmstraJ. (2017). Modelling conditional dependence between response time and accuracy. Psychometrika, 82(4), 1126–1148. https://doi.org/10.1007/s11336-016-9537-6
7.
BolsinovaM.TijmstraJ.MolenaarD.De BoeckP. (2017). Conditional dependence between response time and accuracy: An overview of its possible sources and directions for distinguishing between them. Frontiers in Psychology, 8, 202. https://doi.org/10.3389/fpsyg.2017.00202
8.
ChenJ.de la TorreJ. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37(6), 419–437. https://doi.org/10.1177/0146621613479818
9.
de BoeckP.JeonM. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102. https://doi.org/10.3389/fpsyg.2019.00102
de La TorreJ.DouglasJ. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.
12.
de ValpineP.TurekD.Anderson-BergmanC.Temple LangD.BodikR. (2017). Programming with models: Writing statistical algorithms for general model structures with NIMBLE. Journal of Computational and Graphical Statistics, 26, 403–413. https://doi.org/10.1080/10618600.2016.1172487
13.
DomingueB. W.KanopkaK.StenhaugB.SulikM. J.BeverlyT.BrinkhuisM.CirciR.FaulJ.LiaoD.McCandlissB.ObradovićJ.PiechC.PorterT.ConsortiumP. i.SolandJ.WeeksJ.WiseS. L.YeatmanJ. (2022). Speed–Accuracy trade-Off? Not so fast: Marginal changes in speed have inconsistent relationships with accuracy in real-world settings. Journal of Educational and Behavioral Statistics, 47(5), 576–602. https://doi.org/10.3102/10769986221099906
14.
FoxJ. P.MariantiS. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51(4), 540–553. https://doi.org/10.1080/00273171.2016.1171128
15.
FoxJ. P.MariantiS. (2017). Person-fit statistics for joint models for accuracy and speed. Journal of Educational Measurement, 54(2), 243–262. https://doi.org/https://doi.org/10.1111/jedm.12143
GelmanA. (2006). Prior distributions for variance parameters in hierarchical models. Bayesian Analysis, 1(3), 515–533.
18.
GelmanA.CarlinJ. B.SternH. S.RubinD. B. (2014). Bayesian data analysis. Chapman and Hall/CRC.
19.
GoldhammerF.NaumannJ.KeßelY. (2013). Assessing individual differences in basic computer skills. European Journal of Psychological Assessment, 44(6), 263–275. https://doi.org/10.3102/1076998619881789
20.
GuoH.DeaneP. D.van RijnP. W.ZhangM.BennettR. E. (2018). Modeling basic writing processes from keystroke logs. Journal of Educational Measurement, 55(2), 194–216. https://doi.org/10.1111/jedm.12172
21.
HeQ.BorgonoviF.PaccagnellaM. (2019). Using process data to understand adults’ problem-solving behaviour in the Programme for the International Assessment of Adult Competencies (PIAAC): Identifying generalised patterns across multiple tasks with sequence mining (OECD Education Working Paper No. 205). https://doi.org/10.1787/650918f2-en
22.
HeQ.BorgonoviF.PaccagnellaM. (2021). Leveraging process data to assess adults’ problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166, 104170. https://doi.org/10.1016/j.compedu.2021.104170
23.
HeQ.LiaoD.JiaoH. (2019). Clustering behavioral patterns using process data in piaac problem-solving items. In VeldkampB. P.SluijterC. (Eds.), Theoretical and practical advances in computer-based educational measurement (pp. 189–212). Springer. https://doi.org/10.1007/978-3-030-18480-3
24.
HeQ.ShiQ.TigheE. L. (2023). Predicting problem-solving proficiency with multiclass hierarchical classification on process data: A machine learning approach. Psychological Test and Assessment Modeling, 65(1), 145–177.
25.
HeQ.von DavierM. (2015). Identifying feature sequences from process data in problem-solving items with n-grams. Quantitative Psychology Research. The 79th Annual Meeting of the Psychometric Society, Madison, WI, United States. https://link.springer.com/chapter/10.1007/978-3-319-19977-1_13
26.
HeQ.von DavierM. (2016). Analyzing process data from problem-solving items with n-grams: Insights from a computer-based large-scale assessment. In RosenY.FerraraS.MosharrafM. (Eds.), Handbook of research on technology tools for real-world skill development (pp. 750–777). IGI Global. https://doi.org/10.4018/978-1-4666-9441-5.ch029
27.
KayaY.LeiteW. L. (2017). Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and psychological measurement, 77(3), 369–388. https://doi.org/10.1177/003164416659314
28.
KleinE. R.FoxJ.-P.van der LindenW. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74(1), 21–48. https://doi.org/10.1007/s11336-008-9075-y
29.
Klein EntinkR. H.KuhnJ.-T.HornkeL. F.FoxJ.-P. (2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14(1), 54.
LeeY.-H.ChenH. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53(3), 359–379.
32.
LewandowskiD.KurowickaD.JoeH. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9), 1989–2001. https://doi.org/10.1016/j.jmva.2009.04.008
33.
LiangK.TuD.CaiY. (2023). Using process data to improve classification accuracy of cognitive diagnosis model. Multivariate Behavioral Research, 58(5), 969–987. https://doi.org/10.1080/00273171.2022.2157788
34.
LiaoD.HeQ.JiaoH. (2019). Mapping background variables with sequential patterns in problem-solving environments: An investigation of United States adults’ employment status in PIAAC. Frontiers in Psychology, 10, 646. https://doi.org/10.3389/fpsyg.2019.00646
35.
LiaoM.JiaoH. (2020). Modelling multiple problem-solving strategies and strategy shift in cognitive diagnosis for growth. British Journal of Mathematical and Statistical Psychology, 76, 20–51. https://doi.org/10.1111/bmsp.12280
36.
MaW.GuoW. (2019). Cognitive diagnosis models for multiple strategies. British Journal of Mathematical and Statistical Psychology, 72(2), 370–392. https://doi.org/10.1111/bmsp.12155
37.
ManK.HarringJ. R. (2021). Assessing preknowledge cheating via innovative measures: A multiple-group analysis of jointly modeling item responses, response times, and visual fixation counts. Educational and Psychological Measurement, 81(3), 441–465. https://doi.org/10.1177/0013164420968630
38.
ManK.HarringJ. R.JiaoH.ZhanP. (2019). Joint modeling of compensatory multidimensional item responses and response times. Applied Psychological Measurement, 43(8), 639–654. https://doi.org/10.1177/0146621618824853
MengX. B.TaoJ.ChangH. H. (2015). A conditional joint modeling approach for locally dependent item responses and response times. Journal of Educational Measurement, 52(1), 1–27. https://doi.org/10.1111/jedm.12060.
41.
MinchenN. D.de la TorreJ.LiuY. (2017). A cognitive diagnosis model for continuous response. Journal of Educational and Behavioral Statistics, 42(6), 651–677. https://doi.org/10.3102/1076998617703060
OECD. (2012). Literacy, numeracy and problem solving in technology-rich environments: Framework for the OECD survey of adult skills. OECD Publishing. https://doi.org/10.1787/9789264128859-en
QiaoX.JiaoH.HeQ. (2023). Multiple-group joint modeling of item responses, response times, and action counts with the Conway-Maxwell-Poisson Distribution. Journal of Educational Measurement, 60(2), 255–281. https://doi.org/10.1111/jedm.12349
46.
R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
47.
RezaeeA.GanjaliM.SamaniE. B. (2023). Bayesian inference in a sample selection model with multiple selection rules. Communications in Statistics-Theory and Methods, 53(12), 4290–4310. https://doi.org/10.1080/03610926.2023.2178260
48.
RobertsR. D.StankovL. (1999). Individual differences in speed of mental processing and human cognitive abilities: Toward a taxonomic model. Learning and Individual Differences, 11(1), 1–120.
49.
RuppA. A.TemplinJ. L. (2007). Unique characteristics of cognitive diagnosis models. Annual Meeting of the National Council on Measurement in Education, Chicago, IL, United States.
50.
ShrinerD.YiN. (2009). Deviance information criterion (DIC) in Bayesian multiple QTL mapping. Computational Statistics & Data Analysis, 53(5), 1850–1860. https://doi.org/10.1016/j.csda.2008.01.016
51.
SorensenT.VasishthS. (2015). Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists. arXiv preprint arXiv: 1506.06201. https://browse.arxiv.org/pdf/1506.06201.pdf
52.
SpiegelhalterD. J.BestN. G.CarlinB. P.Van Der LindeA. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583–639.
53.
SuY.-S.YajimaM. (2015). R2jags: Using R to run “JAGS.” R package version 0.5-7, 34.
54.
TangX.WangZ.LiuJ.YingZ. (2021). An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology, 74(1), 1–33. https://doi.org/10.1111/bmsp.12203
55.
UlitzschE.von DavierM.PohlS. (2020). Using response times for joint modeling of response and omission behavior. Multivariate Behavioral Research, 55(3), 425–453. https://doi.org/10.1080/00273171.2019.1643699
56.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308.
57.
van der LindenW. J. (2009). A bivariate lognormal response-time model for the detection of collusion between test takers. Journal of Educational and Behavioral Statistics, 34(3), 378–394.
58.
van der LindenW. J.FoxJ.-P. (2016). Joint hierarchical modeling of responses and response times. In van der LindenW. J. (Eds.), Handbook of item response theory, volume one (pp. 509–528). Chapman and Hall/CRC. https://doi.org/10.1007/S11336-007-9046-8
59.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384.
60.
van ZandtT. (2002). Analysis of response time distributions. In PashlerH. (Eds.), Stevens’ handbook of experimental psychology (Vol. 4, pp. 461–516).
61.
von DavierM. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307.
ZhanP.JiaoH.LiaoD. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286. https://doi.org/10.1111/bmsp.12114
64.
ZhanP.JiaoH.ManK.WangL. (2019). Using JAGS for Bayesian cognitive diagnosis modeling: A tutorial. Journal of Educational and Behavioral Statistics, 44(4), 473–503. https://doi.org/10.3102/1076998619826040
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
