A tree model for diagnostic educational testing is described along with Monte Carlo simulations designed to evaluate measurement accuracy based on the model. The model is implemented in an assessment of inferential reading comprehension, the Multiple-Choice Online Causal Comprehension Assessment (MOCCA), through a sequential, multidimensional, computerized adaptive testing (CAT) strategy. Assessment of the first dimension, reading comprehension (RC), is based on the three-parameter logistic model. For diagnostic and intervention purposes, the second dimension, called process propensity (PP), is used to classify struggling students based on their pattern of incorrect responses. In the simulation studies, CAT item selection rules and stopping rules were varied to evaluate their effect on measurement accuracy along dimension RC and classification accuracy along dimension PP. For dimension RC, methods that improved accuracy tended to increase test length. For dimension PP, however, item selection and stopping rules increased classification accuracy without materially increasing test length. A small live-testing pilot study confirmed some of the findings of the simulation studies. Development of the assessment has been guided by psychometric theory, Monte Carlo simulation results, and a theory of instruction and diagnosis.
AdamsR. J.WilsonM.WangW. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1–23.
2.
BiancarosaG.KennedyP. C.CarlsonS. E.YoonH.-J.SeipelB.LiuB.DavisonM. L. (2019). Constructing subscores that add validity: A case study identifying students at risk. Educational and Psychological Measurement, 79(1), 65–84. https://doi.org/10.1177/0013164418763255
3.
BradshawL.TemplinJ. (2014). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79, 403–425.
4.
CainK.OakhillJ. (2006), Profiles of children with specific reading comprehension difficulties. British Journal of Educational Psychology, 76(4), 683–696. https://doi.org/10.1348/000709905X67610
5.
CarlsonS. E.SeipelB.McMasterK. (2014). Development of a new reading comprehension assessment: Identifying comprehension differences among readers. Learning and Individual Differences, 32, 40–53.
6.
CattsH. W.ComptonDTomblinJ.B.BridgesM.S. (2012). Prevalence and nature of late-emerging poor readers. Journal of Educational Psychology, 104(1). https://doi.org/10.1037/a0025323
7.
ChenJ.de la TorreJ. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37, 419–437. http://doi.org/10.1177/0146621613479818
8.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
9.
CurrieN. K.CainK (2015). Children's inference generation: The role of vocabulary and working memory Journal of Child Experimental Psychology, 137, 57–75.
10.
DavisonM. L.BiancarosaG.CarlsonS. E.SeipelB.LiuB. (2018). Preliminary findings on the computer administered Multiple-choice Online Causal Comprehension Assessment (MOCCA), a diagnostic reading comprehension test. Assessment for Effective Intervention, 43(3), 169–181. https://doi.org/10.1177/1534508417728685
11.
DavisonM. L.BiancarosaG.SeipelB.CarlsonS. E.LiuB.KennedyP. C. (2019). Administration, interpretation, and technical manual 2019: Multiple-Choice Online Comprehension Assessment (MOCCA Technical Report MTR-2019-1). University of Minnesota.
12.
DavisonM. L.LiuB.SeipelB.CarlsonS. E.BiancarosaG. (2017). Continuing efforts in selecting an item response theory: Year 2 MOCCA. Paper presented to the American Educational Research Association.
13.
De BoeckP.ChenH.DavisonM. L. (2017). Spontaneous and imposed speed of cognitive test responses. British Journal of Mathematical and Statistical Psychology, 70(2), 225–237.
14.
De BoeckP.PartchevI. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 28, 2–28.
15.
DelmasR.GarfieldJ.OomsA.ChanceB. (2007). Assessing students’ conceptual understanding after a first course in statistics. Statistics Education Research Journal, 6(2), 28–58.
16.
GuyerR.ThompsonN. A. (2014). User’s manual for Xcalibre item response theory calibration software, version 4.2.2 and later. Assessment Systems Corporation.
17.
Hermann-AbellC. F.DeBoearG. E. (2011). Using distractor-driven standards-based multiple-choice assessments and Rasch modeling to investigate hierarchies of chemistry misconceptions and detect structural problems with individual items. Chemistry Education Research and Practice, 12(2), 184–192. https://doi.org/10.1039/C1RP90023D
18.
HestenesD.WellsM.SwackhamerG. (1992). Force concept inventory. The Physics Teacher, 30(3), 141–158. https://doi.org/10.1119/1.2343497
19.
HongH.WangC.LimY. S.DouglasJ. (2015). Efficient models for cognitive diagnosis with continuous and mixed-type latent variables. Applied Psychological Measurement, 39(1), 31–43. https://doi.org/10.1177/0146621614524981
20.
JohnsonT. R.BoltD. M. (2010). On the use of factor-analytic multinomial logit item response models to account for individual differences in response style. Journal of Educational and Behavioral Statistics, 35(1), 92–114. https://doi.org/103102/1076998609340529
21.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064
22.
KimN. (2022). A mixture IRTree model for understanding response process in rating scales. Paper presented to the CANAM Online symposium.
23.
KimN.BoltD. M. (2021). A mixture IRTree model for extreme responses style: Accounting for response process uncertainty. Educational and Psychological Measurement, 81(1), 131–154. https://doi.org/10.1177/0013164420913915
24.
LiuB.KennedyP.SeipelB.CarlsonS. E.BiancarosaG.DavisonM. L. (2019). Can we learn from student mistakes in a reading comprehension assessment?Journal of Educational Measurement, 56, 815–835. https://doi.org/10.1111/jedm.12238
25.
McMasterK. L.EspinC. A.van den BroekP. (2014). Making connections: Linking cognitive psychology and intervention research to improve comprehension of struggling readers. Learning Disabilities Research & Practice, 29(1), 17–24.
26.
McMasterK. L.van den BroekP.EspinC. A.WhiteM. J.KendeouP.RappD. N.Bohn-GettlerK.CarlsonS. E. (2012). Making the right connections: Differential effects of reading intervention for subgroups of comprehenders. Learning and Individual Differences, 22, 100–111.
27.
PartchevI.BoeckP. D. (2012). Can fast and slow intelligence be differentiated. Intelligence, 40, 23–32.
PimpertonH.NationK. (2010). Suppressing irrelevant information from working memory: Evidence for domain-specific deficits in poor comprehenders. Journal of Memory and Language, 4, 380–391.
30.
RappD. N.van den BroekP.McMasterK. L.KendeouP.EspinC. A. (2007). Higher-order comprehension processes in struggling readers: A perspective for research and intervention. Scientific Studies of Reading, 11, 289–312. https://doi.org/10.1080/10888430701530417
31.
SadlerP. M. (1998). Psychometric models of student misconceptions in science: Reconciling qualitative studies and distractor-driven assessment instruments. Journal of Research in Science Teaching, 35(3), 165–396.
SpencerM.WagnerR. K.PetscherY. (2019) The reading comprehension and vocabulary knowledge of children with poor reading comprehension despite adequate decoding: Evidence from a regression-based matching approach. Journal of Educational Psychology, 111(1), 1–14. https://psycnet.apa.org/doi/10.1037/edu0000274
34.
SuS.DavisonM. L. (2019). Improving the predictive validity of reading comprehension using response times of correct responses. Applied Measurement in Education, 32(2), 166–182. https://doi.org/10.1080/08957347.2019.1577247
35.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287. https://doi.org/10.1037/1082-989X.11.3.287
36.
ThompsonN.YonH.BerhadM. (2012). Multiple cutscore classification testing with the generalized likelihood ratio test. Paper presented to the International Association for Computerized Adaptive Testing.
37.
Von DavierM. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307.
38.
WangC.ChenP.HuebnerA. (2021). Stopping rules for multi-category computerized classification testing. British Journal of Mathematical and Statistical Psychology, 74(2), 184–202. https://doi.org/10.1111/BMSP.12202
39.
WarmT. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), 427–450. http://doi.org/10.1007/bf02294627
40.
WeissD. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473–492.
41.
YuX.ChengY.ChangH.-H. (2019). Recent developments in cognitive diagnostic computerized adaptive testing (CD-CAT): A comprehensive review. In von DavierM.LeeY.-S. (Eds.), Handbook of diagnostic classification models: Models and model extensions, applications, software packages (pp. 307–331). Springer International Publishing. https://doi.org/10.1007/978-3-030-05584-4_15
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
