Cognitive diagnostic assessment (CDA) has been applied to language assessment in a number of studies in which a diagnostic classification model (DCM) was retrofitted to the results of a non-diagnostic assessment. However, the need to apply CDA through utilization of an inductive rather than a retrofitted approach has been a recurrent theme in these studies. Thus, this study aimed to develop a diagnostic L2 reading comprehension test in English to investigate adult examinees’ reading performances in an EFL (English as a Foreign Language) academic setting. The test was based on a cognitive model of L2 reading comprehension and was administered to a sample of 1058 examinees across Turkey. The results were analyzed using log-linear cognitive diagnosis modeling (LCDM), which is one of the general DCM families subsuming other core DCMs. The findings of the study indicated that obtaining fine-grained diagnostic information about examinees’ performances in a given domain would be possible by coupling an adequate understanding of the construct with a CDA framework.
AldersonJ. C. (1984). Reading in a foreign language: A reading problem or a language problem? In AldersonJ.UrquhartA. (Eds.), Reading in a foreign language (pp. 1–27). Longman.
2.
AldersonJ. C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. Continuum.
3.
AldersonJ. C. (2010). “Cognitive diagnosis and Q-matrices in language assessment”: A commentary. Language Assessment Quarterly, 7(1), 96–103. https://doi.org/10.1080/15434300903426748
4.
AldersonJ. C.HaapakangasE. L.HuhtaA.NieminenL.UllakonojaR. (2014). The diagnosis of reading in a second or foreign language. Routledge.
5.
AldersonJ. C.BrunfautT.HardingL. (2015). Towards a theory of diagnosis in second and foreign language assessment: Insights from professional practice across diverse fields. Applied Linguistics, 36(2), 236–260. https://doi.org/10.1093/applin/amt046
6.
AlmondR. G.DiBelloL. V.MoulderB.Zapata-RiveraJ. D. (2007). Modeling diagnostic assessments with Bayesian networks. Journal of Educational Measurement, 44(4), 341–359. https://doi.org/10.1111/j.1745-3984.2007.00043.x
7.
AryadoustV. (2012). Using cognitive diagnostic assessment to model the underlying structure of a listening test: A sub-skill-based approach. Asian EFL Journal, 14(4), 81–106.
8.
BachmanL. F.PalmerA. S. (1996). Language testing in practice. Oxford University Press.
9.
BernhardtE. B.KamilM. L. (1995). Interpreting relationships between L1 and L2 reading: Consolidating the linguistic threshold and the linguistic interdependence hypotheses. Applied Linguistics, 16, 15–34. https://doi.org/10.1093/applin/16.1.15
10.
BernhardtE. B. (1993). Reading development in a second language. Ablex.
11.
BradshawL.IzsakA.TemplinJ.JacobsonE. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33, 2–14. https://doi.org/10.1111/emip.12020
12.
BuckG.TatsuokaK.KostinI. (1997). The subskills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension. Language Learning, 47(3), 423–466. https://doi.org/10.1111/0023-8333.00016
13.
CarverR. P. (1977). Toward a theory of reading comprehension and rauding. Reading Research Quarterly, 13, 8–63. https://doi.org/10.2307/747588
GaoL.RogersT. (2011). Use of tree-based regression in the analyses of L2 reading test items. Language Testing, 28(1), 77–104. https://doi.org/10.1177/0265532210364380
17.
GierlM. J.CuiY. (2008). Defining characteristics of diagnostic classification models and the problem of retrofitting in cognitive diagnostic assessment. Measurement: Interdisciplinary Research & Perspective, 6(4), 263–268. https://doi.org/10.1080/15366360802497762
GraesserA. C.McNamaraD. S.LouwerseM. M.CaiZ. (2004). Coh-metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, 36(2), 193–202. https://doi.org/10.3758/BF03195564
22.
GrayW. S. (1960). The major aspects of reading. In RobinsonJ. (Ed.), Sequential development of reading abilities (pp. 8–24). Chicago University Press.
23.
HardingL.AldersonJ. C.BrunfautT. (2015). Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles. Language Testing, 32(3), 317–336. https://doi.org/10.1177/0265532214564505
24.
HartzS. M. (2002). A Bayesian guide for the unified model for assessing cognitive abilities: Blending theory with practicality [Unpublished doctoral dissertation]. University of Illinois, Urbana-Champaign.
25.
HensonR.TemplinJ.WillseJ. (2009). Defining a family of cognitive diagnosis models using log linear models with latent variables. Psychometrika, 74, 191–210. https://doi.org/10.1007/s11336-008-9089-5
26.
JangE. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL [Unpublished doctoral dissertation]. University of Illinois, Urbana-Champaign.
27.
JangE. E. (2008). A framework for cognitive diagnostic assessment. In ChapelleC. A.ChungY. R.XuJ. (Eds.), Towards adaptive CALL: Natural language processing for diagnostic language assessment (pp. 117–131). Iowa State University.
28.
JangE. E. (2009a). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment. Language Testing, 26(1), 031–073. https://doi.org/10.1177/0265532208097336
29.
JangE. E. (2009b). Demystifying a Q-matrix for making diagnostic inferences about L2 reading skills. Language Assessment Quarterly, 6(3), 210–238. https://doi.org/10.1080/15434300903071817
30.
JangE. E.DunlopM.WagnerM.KimY.GuZ. (2013). Elementary school ELLs’ reading skill profiles using cognitive diagnosis modeling: Roles of length of residence and home language environment. Language Learning, 63(3), 400–436. https://doi.org/10.1111/lang.12016
31.
JunkerB.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. https://doi.org/10.1177/01466210122032064
32.
JurichD. P.BradshawL. P. (2014). An illustration of diagnostic classification modeling in student learning outcomes assessment. International Journal of Testing, 14(1), 37–41. https://doi.org/10.1080/15305058.2013.835728
33.
KasaiM. (1997). Application of the rule-space model to the reading comprehension section of the Test of English as a Foreign Language (TOEFL) [Unpublished doctoral dissertation]. University of Illinois, Urbana-Champaign.
34.
KimA. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing, 3(2), 227–258. https://doi.org/10.1177/0265532214558457
35.
KintschW. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95, 163–182. https://doi.org/10.1037/0033-295X.95.2.163
36.
KlineR. (2011). Principles and practices of structural equation modeling (3rd ed.). Guilford Press.
LeeY. W.SawakiY. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239–263. https://doi.org/10.1080/15434300903079562
39.
LeightonJ. P.GierlM. J. (2007). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 26, 3–16. https://doi.org/10.1111/j.1745-3992.2007.00090.x
40.
LiH.SuenH. K. (2013). Constructing and validating a Q-Matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1–25. https://doi.org/10.1080/10627197.2013.761522
41.
LiH.HunterC. V.LeiP. W. (2016). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing, 33(3), 391–409. https://doi.org/10.1177/0265532215590848
42.
MadisonM. J.BradshawL. P. (2015). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model. Educational and Psychological Measurement, 75(3), 491–511. https://doi.org/10.1177/0013164414539162
43.
MislevyR. J.SteinbergL. S.AlmondR. G. (2003). On the structure of educational assessment. Measurement: Interdisciplinary Research and Perspective, 1(1), 3–62. https://doi.org/10.1207/S15366359MEA0101_02
NassajiH. (2003). Higher–level and lower–level text processing skills in advanced ESL reading comprehension. The Modern Language Journal, 87(2), 261–276. https://doi.org/10.1111/1540-4781.00189
46.
NicholsP. D. (1994). A framework for developing cognitively diagnostic assessments. Review of Educational Research, 64(4), 575–603. https://doi.org/10.3102/00346543064004575
47.
O’ReillyT.SheehanK. M. (2009). Cognitively based assessment of, for, and as learning: A framework for assessing reading competency. (TOEFL Research Reports No. RR-09–26). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2009.tb02183.x
48.
RumelhartD. E. (1977). Toward and interactive model of reading. In DornicS. (Ed.), Attention and performance. Lawrence Erlbaum.
49.
RuppA. A.TemplinJ. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6(4), 219–262. https://doi.org/10.1080/15366360802490866
50.
RuppA.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
51.
SamuelsJ.KamilM. (1984). Models of the reading process. In PearsonP. D.BarrR.KamilM. L.MosenthalP. (Eds.), The handbook of reading research (pp. 185–224). Longman.
52.
SawakiY.KimH. J.GentileC. (2009). Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190–209. https://doi.org/10.1080/15434300902801917
53.
SmithF. (2004). Understanding reading: A psycholinguistic analysis of reading and learning to read. Lawrence Erlbaum.
54.
StanovichK. E. (1980). Towards an interactive compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly, 16, 32–71. https://doi.org/10.2307/747348
55.
TatsuokaK. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x
56.
TemplinJ.BradshawL. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30, 251–275. https://doi.org/10.1007/s00357-013-9129-4
57.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305. https://doi.org/10.1037/1082-989X.11.3.287
58.
ToprakT. E.AryadoustV.GohC. (2019). The log-linear cognitive diagnosis modeling (LCDM) in second language listening assessment. In AryadoustV.GohC. (Eds.), Quantitative analysis for language assessment: Advanced methods (pp. 56–78). Routledge.
59.
UrquhartA. H.WeirC. J. (1998). Reading in a second language: Process, product and practice. Addison Wesley Longman.
60.
van DijkT. A.KintschW. (1983). Strategies of discourse comprehension. Academic Press.
61.
WeirC. J.PorterD. (1994). The multi-divisible or unitary nature of reading: The language tester between Scylla and Charybdis. Reading in a Foreign Language, 10, 1–19.
62.
WiddowsonH. G. (1984). Reading and communication. In AldersonC.UrquhartA. (Eds.), Reading in a foreign language (pp. 213–227). Longman.
