Sage Journals: Discover world-class research

Abstract

In this article, a testlet hierarchical diagnostic classification model (TH-DCM) was introduced to take both attribute hierarchies and item bundles into account. The expectation-maximization algorithm with an analytic dimension reduction technique was used for parameter estimation. A simulation study was conducted to assess the parameter recovery of the proposed model under varied conditions, and to compare TH-DCM with testlet higher-order CDM (THO-DCM; Hansen, M. (2013). Hierarchical item response models for cognitive diagnosis (Unpublished doctoral dissertation). UCLA; Zhan, P., Li, X., Wang, W.-C., Bian, Y., & Wang, L. (2015). The multidimensional testlet-effect cognitive diagnostic models. Acta Psychologica Sinica, 47(5), 689. https://doi.org/10.3724/SP.J.1041.2015.00689). Results showed that (1) ignoring large testlet effects worsened parameter recovery, (2) DCMs assuming equal testlet effects within each testlet performed as well as the testlet model assuming unequal testlet effects under most conditions, (3) misspecifications in joint attribute distribution had an differential impact on parameter recovery, and (4) THO-DCM seems to be a robust alternative to TH-DCM under some hierarchical structures. A set of real data was also analyzed for illustration.

Keywords

cognitive diagnosis diagnostic classification higher order hierarchical model testlet

Get full access to this article

View all access options for this article.

References

Bradshaw

Templin

(2014). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403–425. https://doi.org/10.1007/s11336-013-9350-4

Bransford

J. D.

Brown

A. L.

Cocking

R. R.

(2000). How people learn: Brain, mind, experience, and school. National Academy Press. https://doi.org/10.17226/9853

Cai

Yang

J. S.

Hansen

(2011). Generalized full-information item bifactor analysis. Psychological Methods, 16(3), 221–248. https://doi.org/10.1037/a0023350

Chen

de la Torre

Zhang

(2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123–140. https://doi.org/10.1111/j.1745-3984.2012.00185.x

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353. https://doi.org/10.1007/bf02295640

delMas

(2014). Trends in students’ conceptual understanding of statistics. In Makar

de Sousa

Gould

(Eds.), Proceedings of the ninth international conference on teaching statistics (icots9), Flagstaff, Arizona, USA, 13–18 July 2014.

delMas

Garfield

Ooms

Chance

(2007). Assessing students’ conceptual understanding after a first course in statistics. Statistics Education Research Journal, 6(2), 28–58.

Dempster

A. P.

Laird

N. M.

Rubin

D. B.

(1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1–22. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x

10.

Garfield

delMas

(2010). A web site that provides resources for assessing students’ statistical literacy, reasoning and thinking. Teaching Statistics, 32(1), 2–7. https://doi.org/10.1111/j.1467-9639.2009.00373.x

11.

Gibbons

R. D.

Hedeker

D. R.

(1992). Full-information item bi-factor analysis. Psychometrika, 57(3), 423–436. https://doi.org/10.1007/bf02295430

12.

(2019). Identifiability of hierarchical latent attribute models. https://arxiv.org/pdf/1906.07869

13.

Hansen

(2013). Hierarchical item response models for cognitive diagnosis (Unpublished doctoral dissertation). UCLA.

14.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. https://doi.org/10.1007/s11336-008-9089-5

15.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

16.

Leighton

J. P.

Gierl

M. J.

(2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.

17.

Leighton

J. P.

Gierl

M. J.

Hunka

S. M.

(2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3), 205–237. https://doi.org/10.1111/j.1745-3984.2004.tb01163.x

18.

Ouyang

(2023). Learning latent and hierarchical structures in cognitive diagnosis models. Psychometrika, 88(1), 175–207. https://doi.org/10.1007/s11336-022-09867-5

19.

Chen

Jiang

(2022). Attribute continuity in cognitive diagnosis models: Impact on parameter estimation and its detection. Behaviormetrika, 50(1), 217, 240. https://doi.org/10.1007/s41237-022-00174-y

20.

de la Torre

(2020). GDINA: The generalized dina model framework: R package. https://CRAN.R-project.org/package=GDINA

21.

Macready

G. B.

Dayton

C. M.

(1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2(2), 99–120. https://doi.org/10.3102/10769986002002099

22.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187–212. https://doi.org/10.1007/bf02294535

23.

Molenaar

I. W.

(2001). Thirty years of nonparametric item response theory. Applied Psychological Measurement, 25(3), 295–299. https://doi.org/10.1177/01466210122032091

24.

Nocedal

Wright

S. J.

(2006). Numerical optimization (2nd ed.). Springer.

25.

Rijmen

(2009). Efficient full information maximum likelihood estimation for multidimensional irt models. ETS Research Report Series, 2009(1), 31. https://doi.org/10.1002/j.2333-8504.2009.tb02160.x

26.

Sha

(2016). Nonparametric diagnostic classification analysis for testlet based tests (Unpublished doctoral dissertation). The University of North Carolina at Greensboro.

27.

Simon

M. A.

Tzur

(2004). Explicating the role of mathematical tasks in conceptual learning: An elaboration of the hypothetical learning trajectory. Mathematical Thinking and Learning, 6(2), 91–104. https://doi.org/10.1207/s15327833mtl0602_2

28.

Tan

de La Torre

Huh

Larimer

M. E.

Mun

E.-Y.

(2022). A tutorial on cognitive diagnosis modeling for characterizing mental health symptom profiles using existing item responses. Prevention Science : The Official Journal of the Society for Prevention Research. https://doi.org/10.1007/s11121-022-01346-8

29.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x

30.

Templin

Bradshaw

(2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2), 317–339. https://doi.org/10.1007/s11336-013-9362-0

31.

Tjoe

de la Torre

(2014). On recognizing proportionality: Does the ability to solve missing value proportional problems presuppose the conception of proportional reasoning? The Journal of Mathematical Behavior, 33, 1–7. https://doi.org/10.1016/j.jmathb.2013.09.002

32.

von Davier

(2008). A general diagnostic model applied to language testing data. The British Journal of Mathematical and Statistical Psychology, 61(Pt 2), 287–307. https://doi.org/10.1348/000711007X193957

33.

von Davier

Lee

Y.-S.

(Eds.). (2019). Handbook of diagnostic classification models. Springer International Publishing. https://doi.org/10.1007/978-3-030-05584-4

34.

Wainer

Kiely

G. L.

(1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24(3), 185–201. https://doi.org/10.1111/j.1745-3984.1987.tb00274.x

35.

Wainer

Wang

(2001). Using a new statistical model for testlets to score toefl. ETS Research Report Series, 2001(1), 23. https://doi.org/10.1002/j.2333-8504.2001.tb01851.x

36.

Wang

Gierl

M. J.

(2011). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in critical reading. Journal of Educational Measurement, 48(2), 165–187. https://doi.org/10.1111/j.1745-3984.2011.00142.x

37.

Wang

(2021). Learning attribute hierarchies from data: Two exploratory approaches. Journal of Educational and Behavioral Statistics, 46(1), 58–84. https://doi.org/10.3102/1076998620931094

38.

Wang

Song

Chen

Meng

Ding

(2015). Attribute–level and pattern–level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457–476. https://doi.org/10.1111/jedm.12096

39.

Wiggins

G. P.

(1998). Educative assessment: Designing assessments to inform and improve student performance. Jossey-Bass.

40.

Yen

W. M.

(1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(3), 187–213. https://doi.org/10.1111/j.1745-3984.1993.tb00423.x

41.

Zhai

Haudek

K. C.

(2022). Assessing argumentation using machine learning and cognitive diagnostic modeling. Research in Science Education, 53(2), 405, 424. https://doi.org/10.1007/s11165-022-10062-w

42.

Zhan

Wang

W.-C.

Bian

Wang

(2015). The multidimensional testlet-effect cognitive diagnostic models. Acta Psychologica Sinica, 47(5), 689. https://doi.org/10.3724/SP.J.1041.2015.00689

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.48 MB

A Testlet Diagnostic Classification Model with Attribute Hierarchies

Abstract

Keywords

Get full access to this article

References

Supplementary Material