With recent statistical advances in cognitive diagnostic assessment (CDA), the CDA approach has been increasingly applied to non-diagnostic tests partly to meet accountability demands for student achievement. The study aimed to evaluate critically the validity of the CDA application to an existing non-diagnostic L2 reading comprehension test and to provide information about challenges and conditions for the CDA approach. Based on Jang's study (2005), this paper focuses on the dependability of the Fusion Model's skill profiling, the characteristics of resulting L2 skill profiles, and the diagnostic capacity of LanguEdge™ test items. In addition, the paper examines the validity arguments from the users' perspective by focusing on the usefulness of the diagnostic feedback. The results suggest that the CDA approach can provide more fine-grained diagnostic information about the level of competency in reading skills than traditional aggregated-test scoring can. While various empirical evidence supported the dependability of the skill profiling process, the results also raised some concerns about the application of the CDA approach to a test developed for non-diagnostic purposes, most significantly, a lack of diagnostic capacity of some of the test items with extremely easy or difficult levels. The results offer useful information about the potential challenges and conditions for future application of cognitive diagnostic assessment.
Afflerbach, P., & Johnston, P. (1984). Research methodology: On the use of verbal reports in reading research. Journal of Reading Behavior, 16(4), 307-322.
2.
Alderson, J.C. (2000). Assessing reading. Cambridge : Cambridge University Press.
3.
Alderson, J.C. (2005). Diagnosing foreign language proficiency: the interface between learning and assessment. London: Continuum.
4.
Alderson, J.C., Clapham, C., & Wall, D. (1995). Language testing construction and evaluation. Cambridge: Cambridge University Press.
5.
Alderson, J.C., & Huhta, A. (2005). The development of a suite of computer-based diagnostic tests based on the CommonEuropean Framework. Language Testing , 22, 301-320.
6.
Alderson, J.C., & Lukmani, Y. (1989). Cognition and reading: Cognitive levels as embodied in test questions. Reading in a Foreign Language, 5(2), 253-270.
7.
Anderson, N.J., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8, 41-66.
8.
Bachman, L.F., Davidson, F., & Milanovic, M. (1996). The use of test method characteristics in the content analysis and design of EFL proficiency tests. Language Testing, 13(2), 125-150.
9.
Bailey, K.M. (1999). Washback in language testing. (TOEFL Monograph Series Report. No. 15). Princeton, NJ: Educational Testing Service.
10.
Berkoff, N.A. (1979). Reading skills in extended discourse in English as a Foreign Language. Journal of Research in Reading, 2(2), 95-107.
11.
Black, P.J., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5, 7-74.
12.
Brown, J.D., & Hudson, T. (2002). Criterion-referenced language testing. Cambridge: Cambridge University Press.
13.
Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15(2), 119-157.
14.
Carver, R.P. (1992). What do standardized tests of reading comprehension measure in terms of efficiency, accuracy, and rate?Reading Research Quarterly, 27(4), 347-359.
15.
Cohen, A.D. (1987). Studying language learning strategies: How do we get the information? In A. L. Wenden & J. Rubin (Eds.), Learner strategies in language learning (pp. 31-40). Englewood Cliffs, NJ: Prentice-Hall. Cohen, A.D., & Hosenfeld, C. (1981). Some uses of mentalistic data in second-language research. Language Learning, 31(2), 285-313.
16.
Cohen, A.D., & Upton, T.A. (2007). `I want to go back to the text': Response strategies on the reading subtest of the new TOEFL. Language Testing , 24, 209-50.
17.
Collins, A. (1990). Reformulating testing to measure learning and thinking. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 75-88). Hillsdale, NJ: Lawrence Erlbaum.
18.
Cordon, L.A., & Day, J.D. (1996). Strategy use on standardized reading comprehension tests. Journal of Educational Psychology, 88(2), 288-295.
19.
Davidson, F., & Lynch, B.K. (2002). Testcraft: A teacher's guide to writing and using language test specifications. New Haven and London: Yale University Press.
20.
DiBello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 361-390). Hillsdale, NJ: Lawrence Erlbaum.
21.
DiBello, L.V., Roussos, L.A., & Stout, W. (2007). Review of cognitive diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 45-79). Elsevier Science B.V.: The Netherlands.
22.
Embretson, S. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 37, 359-374.
23.
Embretson, S. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3(3), 380-396.
24.
Faerch, C., & Kasper, G. (Eds.). (1987). Introspection in second language research. Philadelphia, PA: Multilingual Matters.
25.
Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27(3), 209-226.
26.
Fischer, G.H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologia, 37, 359-374.
27.
Frederiksen, N., Glaser, R., Lesgold, A., & Shafto, M. (Eds.). (1990). Diagnostic monitoring of skill and knowledge acquisition. Hillsdale, NJ: Lawrence Erlbaum.
28.
Gorin, J.S. (2007). Test construction and diagnostic testing. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 173-201). New York: Cambridge University Press.
29.
Grabe, W. (1991). Current developments in second language reading research. TESOL Quarterly, 25(3), 375-406.
30.
Grabe, W. (Ed.). (2000). Developments in reading research and their implications for computer-adaptive tests of reading. Cambridge: Cambridge University Press.
31.
Greene, J.C., & Caracelli, V.J. (1997). Defining and describing the paradigm issue in mixed method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed methods evaluation: The challenges and benefits of integrating diverse paradigms. New directions for evaluation No. 74(pp. 5-17). San Francisco, CA: Jossey-Bass.
32.
Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign .
33.
Hartz, S.M., & Roussos, L.A. (2005). The Fusion Model for skills diagnosis: Blending theory with practice. ETS Research Report, Educational Testing Service, Princeton, NJ.
34.
Jang, E.E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign .
35.
Jang, E.E. (2008). A framework for cognitive diagnostic assessment . In C. A. Chapelle, Y.-R. Chung, & J. Xu (Eds.), Towards adaptive CALL: Natural Language Processing for Diagnostic Language Assessment (pp. 117-131). Amer, IA: Iowa Scaner University.
36.
Kasai, M. (1997). Application of the rule space model to the reading comprehension section of the test of English as a foreign language (TOEFL) . Unpublished doctoral dissertation. University of Illinois, Urbana Champaign.
37.
Kunnan, A.J., & Jang, E.E. (forthcoming). Diagnostic feedback in language assessment. In M. Log & C. Doughty (Eds.), Handbook of second and foreign language teaching. Walden, MA: Wiley-Blackwell .
38.
Li, W. (1992). What is a test testing? An investigation of the agreement between students' test-taking processes and test constructors' presumption. Unpublished MA thesis. Lancaster University.
39.
Lumley, T. (1993). The notion of subskills in reading comprehension tests: An EAP example. Language Testing, 10(3), 211-234.
40.
Mislevy, R.J. (1995). Probability-based inference in cognitive diagnosis . In P. Nichols, S. Chipman, & R. J. Brennan (1995), Cognitively diagnostic assessment (pp. 43-71). Hillsdale, NJ: Lawrence Erlbaum.
41.
Mislevy, R.J., Steinberg, L.S., & Almond, R.G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-67.
42.
Mislevy, R.J., Almond, R.G., & Lukas, J.F. (2004). A brief introduction to Evidence-Centered Design . CSE Technical Report 632, The National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Center for the Study of Evaluation (CSE). LA, CA: University of California, Los Angeles.
43.
Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Learning, 6, 199-215.
44.
Nichols, P.D. (1994). A framework for developing cognitively diagnostic assessments. Review of Educational Research, 64(4), 575-603.
45.
Nicols, P. D., Chipman, S. F., & Brennan, R. L. (Eds.). (1995). Cognitively diagnostic assessment. Hillsdale, NJ: Lawrence Erlbaum.
46.
Patz, R.J. & Junker, B.W. (1999). A straightforward approach to Markov Chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146-178.
47.
Pellegrino, J.W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
48.
Rost, D. (1993). Assessing the different components of reading comprehension: Fact or fiction? Language Testing, 10(1), 79-82.
49.
Roussos, L., DiBello, L., Henson, R., Jang, E.E., & Templin, J. (in press). Skills diagnosis for education and psychology with IRT-based parametric latent class models. In S. E. Embretson & J. Roberts (Eds.), New directions in psychological measurement with model-based approaches. Washington, DC: American Psychological Association .
50.
Sheehan, K.M. (1997). A tree-based approach to proficiency scaling and diagnostic assessment. Journal of Educational Measurement , 34(4), 333-352.
51.
Shohamy, E. (1992). Beyond performance testing: A diagnostic feedback testing model for assessing foreign language learning. Modern Language Journal, 76(4), 513-521.
52.
Snow, R.E., & Lohman, D.F. (1989). Implications of cognitive psychology for educational measurement. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 263-332). New York: Macmillan.
53.
Spolsky, B. (1990). Social aspects of individual assessment. In J. de Jong & D. K. Stevenson (Eds.), Individualizing the assessment of language abilities (pp. 3-15). Avon: Multilingual Matters.
54.
Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage Publications.
55.
Tatsuoka, K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement , 20(4), 345-354.
56.
Tatsuoka, K. (1990). Toward an integration of Item-Response Theory and cognitive error diagnosis. In N. Frederiksen , R. Glaser, A. Lesgold & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Lawrence Erlbaum.
57.
Weir, C.J., & Porter, D. (1996). The multi-divisible or unitary nature of reading: The language tester between Scylla and Charybdis. Reading in a Foreign Language, 10, 1-19.
58.
Weir, C.J., Huizhong, Y., & Yan, J. (2000). An empirical investigation of the componentiality of L2 reading in English for academic purposes. Cambridge : Cambridge University Press.
