A Multistrategy Cognitive Diagnosis Model Incorporating Item Response Times Based on Strategy Selection Theories

Abstract

Response times (RTs) facilitate the quantification of underlying cognitive processes in problem-solving behavior. To provide more comprehensive diagnostic feedback on strategy selection and attribute profiles with multistrategy cognitive diagnosis model (CDM) and utilize additional information for item RTs, this study develops a multistrategy cognitive diagnosis modeling framework combined with RTs. The proposed model integrates individual response accuracy and RT into a unified framework to define strategy selection and make it closer to the individual’s strategy selection process. Simulation studies demonstrated that the proposed model had reasonable parameter recovery and attribute classification accuracy and outperformed the existing multistrategy CDMs and single-strategy CDMs in terms of performance. Empirical results further illustrated the practical application and the advantages of the proposed model.

Keywords

cognitive diagnosis model multistrategy response times strategy selection

Get full access to this article

View all access options for this article.

References

Bolsinova

de Boeck

Tijmstra

(2017). Modelling conditional dependence between response time and accuracy. Psychometrika, 82(4), 1126–1148. https://doi.org/10.1007/s11336-016-9537-6

Brooks

S. P.

Gelman

(1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434–455. https://doi.org/10.1080/10618600.1998.10474787

Carpenter

P. A.

Just

M. A.

Shell

(1990). What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test. Psychological Review, 97(3), 404–431. https://doi.org/10.1037/0033-295X.97.3.404

Chiu

C.-Y.

Douglas

J. A.

(2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665. https://doi.org/10.1007/s11336-009-9125-0

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353. https://doi.org/10.1007/bf02295640

de la Torre

Douglas

J. A.

(2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595–624. https://doi.org/10.1007/s11336-008-9063-2

DeShon

R. P.

Chan

Weissbein

D. A

. (1995). Verbal overshadowing effects on Raven’s advanced progressive matrices: Evidence for multidimensional performance determinants. Intelligence, 21(2), 135–155. https://doi.org/10.1016/0160-2896(95)90023-3

DiBello

L. V.

Stout

W. F.

Roussos

L. A.

(1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In Nichols

P. D.

Chipman

S. F.

Brennan

R. L.

(Eds.), Cognitively diagnostic assessment (pp. 361–389). Hillsdale, NJ: Erlbaum.

10.

Duverne

Lemaire

(2005). Arithmetic split effects reflect strategy selection: An adult age comparative study in addition comparison and verification tasks. Journal of Experimental Psychology, 59, 262–278. https://doi.org/10.1037/h0087479

11.

Gilks

W. R.

Richardson

Spiegelhalter

D. J.

(1996). Markov Chain Monte Carlo in Practice. Boca Raton, FL: Chapman and Hall/CRC.

12.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality [Unpublished doctoral dissertation]. University of Illinois at Urbana-Champaign.

13.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210. https://doi.org/10.1007/s11336-008-9089-5

14.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

15.

Klein Entink

R. H.

Fox

J.-P.

van der Linden

W. J.

(2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21–48. https://doi.org/10.1007/S11336-008-9075-Y

16.

Köhn

H.-F.

Chiu

C.-Y.

(2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests. Psychometrika, 82, 112–132. https://doi.org/10.1007/s11336-016-9536-7

17.

Liu

Zhao

Dai

(2016). Multiple-strategy diagnosis of rules for solving figure reasoning tasks. Chinese Journal of Clinical Psychology, 24(3), 459–463. https://doi.org/10.16128/j.cnki.1005-3611.2016.03.017

18.

Logan

Medford

Hughes

(2011). The importance of intrinsic motivation for high and low ability readers’ reading comprehension performance. Learning and Individual Differences, 21, 124–128. https://doi.org/10.1016/j.lindif.2010.09.011

19.

Lovett

M. C.

Anderson

J. R.

(1996). History of success and current context in problem solving: Combined influences on operator selection. Cognitive Psychology, 31(2), 168–217. https://doi.org/10.1006/cogp.1996.0016

20.

(2018). A diagnostic tree model for polytomous responses with multiple strategies. British Journal of Mathematical and Statistical Psychology, 72(1), 61–82. https://doi.org/10.1111/bmsp.12137

21.

Guo

(2019). Cognitive diagnosis models for multiple strategies. British Journal of Mathematical and Statistical Psychology, 72(2), 370–392. https://doi.org/10.1111/bmsp.12155

22.

Marewski

J. N.

Schooler

L. J.

(2011). Cognitive niches: An ecological model of strategy selection. Psychological Review, 118(3), 393–437. https://doi.org/10.1037/a0024143

23.

Mislevy

R. J.

Verhelst

(1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55(2), 195–215. https://doi.org/10.1007/bf02295283

24.

Mislevy

R. J.

Wingersky

M. S.

Irvine

S. H.

Dann

P. L.

(1991). Resolving mixtures of strategies in spatial visualization tasks. British Journal of Mathematical and Statistical Psychology, 44(2), 265–288. https://doi.org/10.1111/j.2044-8317.1991.tb00961.x

25.

Peng

Cai

Wang

Luo

(2021). A generalized diagnostic classification modeling framework integrating differential speediness: Advantages and illustrations in psychological and educational testing. Multivariate Behavioral Research, 57(6), 940–959. https://doi.org/10.1080/00273171.2021.1928474

26.

Plummer

(2015). JAGS Version 4.0.0 user manual. Lyon, France.

27.

Qian

Staniewska

Reckase

Woo

(2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35, 38–47. https://doi.org/10.1111/emip.12102

28.

Rieskamp

Otto

P. E.

(2006). SSL: A theory of how people learn to select strategies. Journal of Experimental Psychology: General, 135(2), 207–236. https://doi.org/10.1037/0096-3445.135.2.207

29.

Rittle-Johnson

Siegler

R. S.

(1999). Learning to spell: Variability, choice, and change in children’s strategy use. Child Development, 70(2), 332–348. https://doi.org/10.2307/1132092

30.

Rost

(1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14, 271–282. https://doi.org/10.1177/014662169001400305

31.

Siegler

R. S.

(1987). The perils of averaging data over strategies: An example from children’s addition. Journal of Experimental Psychology: General, 116, 250–264. https://doi.org/10.1037/0096-3445.116.3.250

32.

Siegler

R. S.

(1989). Hazards of mental chronometry: An example from children’s subtraction. Journal of Educational Psychology, 81(4), 497–506. https://doi.org/10.1037/0022-0663.81.4.497

33.

Siegler

R. S.

Lemaire

(1997). Older and younger adults’ strategy choices in multiplication: Testing predictions of ASCM using the choice/no-choice method. Journal of Experimental Psychology: General, 126(1), 71–92. https://doi.org/10.1037/0096-3445.126.1.71

34.

Y. S.

Yajima

(2015). R2jags: Using R to run “JAGS” (R package version 0.5-7).

35.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x

36.

Templin

Bradshaw

(2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251–275. https://doi.org/10.1007/s00357-013-9129-4

37.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnostic models. Psychological Methods, 11(3), 287–305. https://doi.org/10.1037/1082-989X.11.3.287

38.

van der Linden

W. J.

(2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. https://doi.org/10.3102/10769986031002181

39.

van der Linden

W. J.

(2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z

40.

von Davier

(2010). Mixture distribution item response theory, latent class analysis, and diagnostic mixture models. In Embretson

S. E.

(Ed.), Measuring psychological constructs: Advances in model-based approaches (pp. 11–34). American Psychological Association. https://doi.org/10.1037/12074-000

41.

Wang

Zhang

Shen

(2020). A joint modeling framework of responses and response times to assess learning outcomes. Multivariate Behavioral Research, 55(1), 49–68. https://doi.org/10.1080/00273171.2019.1607238

42.

Wang

T. Y.

Zhang

J. W.

(2006). Optimal partitioning of testing time: Theoretical properties and practical implications. Psychometrika, 71, 105–120. https://doi.org/10.1007/s11336-002-1013-x

43.

Xiang

Y. H.

Shang

R. H.

Chao

Ren

T. T.

Zheng

L. L.

(2018). Influence of the large–small split effect on strategy choice in complex subtraction. International Journal of Psychology, 53(2), 92–96. https://doi.org/10.1002/ijop.12268

44.

Zhan

Jiao

Liao

(2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286. https://doi.org/10.1111/bmsp.12114