Recent developments in psychometric modeling underscore the need to incorporate learning behavior into intelligent tutoring systems (ITS) assessments. To determine mastery of target skills in ITSs, a detection procedure that captures the change in the response behavior is necessary. Leveraging items characterized by their properties, a two-stage detection rule can promote learning while mitigating the risk of false detections. A detection scheme is designed in two stages, employing cumulative sum (CUSUM) statistics to delineate progress within each stage. The procedure partitions each training process into learning and mastery detection stages. Item pools are stratified based on item properties, enabling tailored allocation of items to enhance learning and mastery detection. CUSUM thresholds, which incorporate learning rates, ensure a predetermined false detection rate. The two-stage procedure offers a nuanced approach that accelerates learning with highly pedagogical items in the first stage and efficiently detects mastery in the second stage. Simulation results demonstrate the efficiency and effectiveness of the two-stage procedure. Compared to an ordinary single-stage approach, the procedure consistently yields smaller average change points, indicating quicker mastery detection.
AkbayL.de la TorreJ. (2020). Estimation approaches in cognitive diagnosis modeling when attributes are hierarchically structured. Psicothema, 32, 122–129.
2.
ChangH.-H.Van der LindenW. J. (2003). Optimal stratification of item pools in α-stratified computerized adaptive testing. Applied Psychological Measurement, 27(4), 262–274.
ChangH.-H.QianJ.YingZ. (2001). A-Stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement, 25(4), 333–341.
5.
ChenY.CulpepperS. A.WangS.DouglasJ. (2018). A hidden Markov model for learning trajectories in cognitive diagnosis with application to spatial rotation skills. Applied Psychological Measurement, 42(1), 5–23.
6.
ChenY.LiX.LiuJ.YingZ. (2018). Recommendation system for adaptive learning. Applied Psychological Measurement, 42(1), 24–41.
7.
De La TorreJ. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.
8.
DeonovicB.ChopadeP.YudelsonM.de la TorreJ.von DavierA. A. (2019). Application of cognitive diagnostic models to learning and assessment systems. In Handbook of diagnostic classification models: Models and model extensions, applications, software packages (pp. 437–460). Springer International Publishing.
9.
ElosuaP.AguadoD.Fonseca-PedreroE.AbadF. J.SantamaríaP. (2023). New trends in digital technology-based psychological and educational assessment. Psicothema, 35(1), 50–57.
10.
HanR.ChenK.TanC. (2020). Curiosity-driven recommendation strategy for adaptive learning via deep reinforcement learning. British Journal of Mathematical and Statistical Psychology, 73(3), 522–540.
11.
KaplanM.de la TorreJ.BarradaJ. R. (2015). New item selection methods for cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 39(3), 167–188.
12.
KimN. J.BellandB. R.WalkerA. E. (2018). Effectiveness of computer-based scaffolding in the context of problem-based learning for stem education: Bayesian meta-analysis. Educational Psychology Review, 30, 397–429.
13.
KulikJ. A.FletcherJ. D. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 86(1), 42–78.
14.
LeightonJ. P.GierlM. J.HunkaS. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3), 205–237.
15.
LiX.XuH.ZhangJ.ChangH.-H. (2023). Deep reinforcement learning for adaptive learning systems. Journal of Educational and Behavioral Statistics, 48(2), 220–243.
16.
LiuY.CulpepperS. A.ChenY. (2023). Identifiability of hidden Markov models for learning trajectories in cognitive diagnosis. Psychometrika, 88(2), 361–386.
17.
LordenG. (1971). Procedures for reacting to a change in distribution. The Annals of Mathematical Statistics, 42, 1897–1908.
18.
MoustakidesG. V. (1986). Optimal stopping times for detecting changes in distributions. The Annals of Statistics, 14(4), 1379–1387.
19.
PageE. S. (1954). Continuous inspection schemes. Biometrika, 41(1/2), 100–115.
20.
ShiryaevA. N. (1963). On optimum methods in quickest detection problems. Theory of Probability & Its Applications, 8(1), 22–46.
21.
ShiryaevA. N. (2007). Optimal stopping rules (Vol. 8). Springer Science & Business Media.
22.
Steenbergen-HuS.CooperH. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331–347.
23.
TanC.HanR.YeR.ChenK. (2020). Adaptive learning recommendation strategy based on deep q-learning. Applied Psychological Measurement, 44(4), 251–266.
24.
TangX.ChenY.LiX.LiuJ.YingZ. (2019). A reinforcement learning approach to personalized learning recommendation systems. British Journal of Mathematical and Statistical Psychology, 72(1), 108–135.
25.
VanLehnK. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.
26.
VanLehnK. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221.
27.
WangC. (2021). On interim cognitive diagnostic computerized adaptive testing in learning context. Applied Psychological Measurement, 45(4), 235–252.
28.
WangC.ChangH.-H.DouglasJ. (2012). Combining cat with cognitive diagnosis: A weighted item selection approach. Behavior Research Methods, 44, 95–109.
29.
WangS.YangY.CulpepperS. A.DouglasJ. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87.
30.
WoodD.BrunerJ. S.RossG. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89–100.
YeS.FellourisG.CulpepperS.DouglasJ. (2016). Sequential detection of learning in cognitive diagnosis. British Journal of Mathematical and Statistical Psychology, 69(2), 139–158.
33.
YigitH. D.DouglasJ. A. (2021). First-order learning models with the GDINA: Estimation with the EM algorithm and applications. Applied Psychological Measurement, 45(3), 143–158.
34.
YuA.DouglasJ. A. (2023). IRT models for learning with item-specific learning parameters. Journal of Educational and Behavioral Statistics, 48(6), 866–888.
35.
ZhangS.ChangH.-H. (2016). From smart testing to smart learning: How testing technology can assist the new generation of education. International Journal of Smart Technology and Learning, 1(1), 67–92.
