Respondents’ problem-solving behaviors comprise behaviors that represent complicated cognitive processes that are frequently systematically tied to one another. Biometric data, such as visual fixation counts (FCs), which are an important eye-tracking indicator, can be combined with other types of variables that reflect different aspects of problem-solving behavior to quantify variability in problem-solving behavior. To provide comprehensive feedback and accurate diagnosis when using such multimodal data, the present study proposes a multimodal joint cognitive diagnosis model that accounts for latent attributes, latent ability, processing speed, and visual engagement by simultaneously modeling response accuracy (RA), response times, and FCs. We used two simulation studies to test the feasibility of the proposed model. Findings mainly suggest that the parameters of the proposed model can be well recovered and that modeling FCs, in addition to RA and response times, could increase the comprehensiveness of feedback on problem-solving-related cognitive characteristics as well as the accuracy of knowledge structure diagnosis. An empirical example is used to demonstrate the applicability and benefits of the proposed model. We discuss the implications of our findings as they relate to research and practice.
AgardC.von DavierA. (2018). The virtual world and reality of testing: Building virtual assessments. In JiaoH.LissitzR. (Eds.), Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective (pp. 1–30). Information Age Publishing.
2.
AnL.WangY.SunY. (2017). Reading words or pictures: Eye movement patterns in adults and children differ by age group and receptive language ability. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.00791
BergnerY.von DavierA. (2018). Process data in NEAP: Past, present, and future. Journal of Educational and Behavioral Statistics. http://doi.org/10.3102/1076998618784700
5.
BezirhanU.von DavierM.GrabovskyI. (2021). Modeling item revisit behavior: The hierarchical speed-accuracy-revisits model. Educational and Psychological Measurement, 81(2), 363–387.
6.
BolsinovaM.MarisG. (2016). A test for conditional independence between response time and accuracy. British Journal of Mathematical and Statistical Psychology, 69, 62–79.
7.
BolsinovaM.TijmstraJ. (2016). Posterior predictive checks for conditional independence between response time and accuracy. Journal of Educational and Behavioral Statistics, 41, 123–145.
8.
BolsinovaM.TijmstraJ. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71, 13–38.
9.
BrooksS. P.GelmanA. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434–455. https://doi.org/10.2307/1390675
ChiuC.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37, 598–618. https://doi.org/10.1177/0146621613488436
12.
CorcoranP. M.NanuF.PetrescuS.BigioiP. (2012). Real-time eye gaze tracking for gaming design and consumer electronics systems. IEEE Transactions on Consumer Electronics, 58(2), 347–355. https://doi.org/10.1109/tce.2012.6227433
13.
CulpepperS. A. (2015). Bayesian estimation of the DINA model with Gibbs Sampling. Journal of Educational and Behavioral Statistics, 40, 454–476.
14.
De BoeckP.JeonM. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102.
15.
de la TorreJ. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.
16.
de la TorreJ.ChiuC.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 252–273.
17.
de la TorreJ.DouglasJ. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353. https://doi.org/10.1007/BF02295640
18.
DunnJ. C.NewellB. R.KalishM. L. (2012). The effect of feedback delay and feedback type on perceptual category learning: The limits of multiple systems. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38(4), 840–859. https://doi.org/10.1037/a0027867
19.
FoxJ.-P.MariantiS. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51, 540–553. https://doi.org/10.1080/00273171.2016.1171128
20.
GardnerR. W.HolzmanP. S.KleinG. S.LintonH. B.SpenceD. (1959). Cognitive control: A study of individual consistencies in cognitive behavior. Psychological Issues, 1, Monograph 4.
21.
GelmanA.CarlinJ. B.SternH. S.DunsonD. B.VehtariA.RubinD. B. (2014). Bayesian data analysis. CRC Press.
22.
GorinJ. S. (2006). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25(4), 21–35.
23.
GrafS.KinshukD. (2007). Providing adaptive courses in learning management systems with respect to learning styles. In RichardsG. (Eds.), Proceedings of the World Conference on E-Learning in Corporate, Government, Health Care, and Higher Education (e-Learn) (pp. 2576–2583). AACE Press.
24.
GuY.XuG. (2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31, 449–472.
25.
HeQ.Von DavierM. (2016). Analyzing process data from problem-solving items with n-grams: Insights from a computer-based large-scale assessment. In RosenY.FerraraS.MosharrafM. (Eds.), Handbook of research on technology tools for real-world skill development (pp. 749–776). Chapman and Hall/CRC Press. https://doi.org/10.4018/978-1-4666-9441-5.ch029
26.
HensonR.TemplinJ.WillseJ. (2009). Defining a family of cognitive diagnosis models using loglinear models with latent variables. Psychometrika, 74, 191–210. https://doi.org/10.1007/s11336-008-9089-5
27.
HitchG. J.McAuleyE. (1991). Working memory in children with specific arithmetical learning difficulties. British Journal of Psychology, 82, 375–386.
28.
JacobR. J. K. (1995). Eye tracking in advanced interface design. In BarfieldW.FurnessT. A. (Eds.), Virtual environments and advanced interface design (pp. 258–288). Oxford University Press.
29.
JeonM.De BoeckP.LuoJ.LiX.LuZ.-L. (2021). Modeling within-item dependencies in parallel data on test responses and brain activation. Psychometrika. Advanced Online. https://doi.org/10.1007/s11336-020-09741-2
30.
JiangZ.MaW. (2018). Integrating differential evolution optimization to cognitive diagnostic model estimation. Frontiers in Psychology, 9, 2142. https://doi.org/10.3389/fpsyg.2018.02142
31.
JiaoH.LissitzR. W. (2018). Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective. Information Age Publishing.
32.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.
33.
JusticeM.LankfordC. (2002). Pilot findings. Communication Disorders Quarterly, 1, 11–21.
34.
KaganJ. (1965). Reflection-impulsivity and reading ability in primary grade children. Child Development, 36, 609–628.
35.
KangI.De BoeckP.RatcliffR (2022). Modeling conditional dependence of response accuracy and response time with the diffusion item response theory model. Psychometrika. https://doi.org/10.1007/s11336-021-09819-5
36.
LahatD.AdahT.JuttenC. (2015). Multimodal data fusion: An overview of methods, challenges and prospects. Proceedings of the IEEE, 103(9), 1449–1477.
37.
LeightonJ. P.GierlM. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.
38.
LevyR.MislevyR. J. (2016). Bayesian psychometric modeling. CRC Press.
39.
LiuH.LiuY.LiM. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372.
40.
MaW.de la TorreJ. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69, 253–275.
41.
MaW.de la TorreJ. (2020). Choosing between CDM and unidimensional IRT: The proportional reasoning test case. Measurement Interdisciplinary Research and Perspectives, 18(2), 87–96.
42.
MaddoxW. T.LoveB. C.GlassB. D.FiloteoJ. V. (2008). When more is less: Feedback effects in perceptual category learning. Cognition, 108(2), 578–589. https://doi.org/10.1016/j.cognition.2008.03.010
43.
ManK.HarringJ. R. (2019). Negative binomial models for visual fixation counts on test items. Educational and Psychological Measurement, 79(4), 617–635. http://doi.org/0.1177/0013164418824148
44.
ManK.HarringJ. R. (2021). Assessing pre-knowledge cheating via innovative measures: A multiple-group analysis of jointly modeling item responses, response times, and visual fixation counts. Educational and Psychological Measurement, 81(3), 441–465. https://doi.org/10.1177/0013164420968630
45.
ManK.HarringJ. R.ZhanP. (2022). Bridging models of biometric and psychometric assessment: A three-way joint modeling approach of item responses, response times, and gaze fixation counts. Applied Psychological Measurement, 46(5), 361–381. https://doi.org/10.1177/01466216221089344
46.
MayesS. D.CalhounS. L. (2007). Learning, attention, writing, and processing speed in typical children and children with ADHD, autism, anxiety, depression, and oppositional-defiant disorder. Child Neuropsychology, 13(6), 469–493. https://doi.org/10.1080/09297040601112773
47.
MengX.-B.TaoJ.ChangH.-H. (2015). A conditional joint modeling approach for locally dependent item responses and response times. Journal of Educational Measurement, 52, 1–27. https://doi.org/10.1111/jedm.12060
MolenaarD.TuerlinckxF.van der MaasH. L. (2015). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68(2), 197–219. https://doi.org/10.1111/bmsp.12042
50.
PassolunghiM. C.SiegelL. S. (2001). Short-term memory, working memory and inhibitory control in children with difficulties in arithmetic problem solving. Journal of Experimental Child Psychology, 80, 44–51.
RangerJ. (2013). A note on the hierarchical model for responses and response times in tests of van der Linden (2007). Psychometrika, 78, 3, 538–544.
53.
RidingR.RaynerS. (2012). Cognitive styles and learning strategies: Understanding style differences in learning and behavior. Routledge.
54.
RuppA.TemplinJ.HensonR. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
55.
ShuZ.BergnerY.ZhuM.HaoJ.von DavierA. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109–131.
56.
ShuteV. J.VentureM. (2013). Measuring and supporting learning in games: Stealth assessment. The MIT Press.
57.
TangF.ZhanP. (2021). Does diagnostic feedback promote learning? Evidence from a longitudinal cognitive diagnostic assessment. ResearchGate Preprint. https://doi.org/10.13140/RG.2.2.23511.19365
58.
TatsuokaK. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x
59.
UnkelbachC. (2006). The learned interpretation of cognitive fluency. Psychological Science, 17(4), 339–345.
60.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.
61.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308.
62.
von DavierA.HalpinP. F. (2013). Collaborative problem solving and the assessment of cognitive skills: Psychometric considerations. (RR-13-41). ETS research reports.
63.
von DavierM.LeeY.-S. (2019). Handbook of diagnostic classification models: Models and model extensions, applications, software packages. Springer.
64.
WangC.ChangH.DouglasJ. (2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66, 144–168. http://dx.doi.org/10.1111/j.2044-8317.2012.02045.x
65.
WangC.XuG. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68, 456–477. http://dx.doi.org/10.1111/bmsp.12054
66.
WangL.TangF.ZhanP. (2020). Effect analysis of individualized remedial teaching based on cognitive diagnostic assessment: Taking “linear equation with one unknown” as an example. Journal of Psychological Science, 43(6), 1490–1497.
67.
WangS.ChenY. (2020). Using response times and response accuracy to measure fluency within cognitive diagnosis models. Psychometrika, 85, 600–629.
68.
WangS.ZhangS.DouglasJ.CulpepperS. (2018). Using response times to assess learning progress: A joint model for responses and response times. Measurement: Interdisciplinary Research and Perspectives, 16(1), 45–58.
ZhanP.JiaoH.LiaoD. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71, 262–286.
71.
ZhanP.JiaoH.ManKWangL. (2019). Using JAGS for Bayesian cognitive diagnosis modeling: A tutorial. Journal of Educational and Behavioral Statistics, 44, 473–503.
72.
ZhanP.JiaoH.ManK.WangW.-C.HeK. (2021). Variable speed across dimensions of ability in the joint model for responses and response times. Frontiers in Psychology, 12, 469196.
73.
ZhanP.LiaoM.BianY. (2018). Joint testlet cognitive diagnosis modeling for paired local item dependence in response times and response accuracy. Frontiers in Psychology, 9, 607.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
