In this study, we present the development of individualized feedback for a large-scale listening assessment by combining standard setting and cognitive diagnostic assessment (CDA) approaches. We used the performance data from 3,358 students’ item-level responses to a field test of a national EFL test primarily intended for tertiary-level EFL learners. The results showed that proficiency classifications and subskill mastery classifications were generally of acceptable reliability, and the two kinds of classifications were in alignment with each other at individual and group levels. The outcome of the study is a set of descriptors that describe each test taker’s ability to understand certain level of oral texts and his or her cognitive performance. The current study, by illustrating the feasibility of combining standard setting and CDA approaches to produce individualized feedback, contributes to the enhancement of score reporting and addresses the long-standing criticism that large-scale language assessments fail to provide individualized feedback to link assessment with instruction.
AldersonC. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. Continuum.
3.
AldersonC. (2007). The challenge of (diagnostic) testing: Do we know what we are measuring? In FoxJ.WescheM.BaylissD.ChengL.TurnerC.DoeC. (Eds.), Language testing reconsidered (pp. 21–39). University of Ottawa Press.
4.
AldersonC.HuhtaA. (2005). The development of a suite of computer-based diagnostic tests based on the Common European Framework. Language Testing, 22(3), 301–320. http://doi/org/10.1191/0265532205lt310oa
5.
AryadoustV. (2018). A cognitive diagnostic assessment study of the listening test of the Singapore–Cambridge General Certificate of Education O-Level: Application of DINA, DINO, G-DINA, HO-DINA, and RRUM. International Journal of Listening, 35(1), 1–24. http://doi.org/10.1080/10904018.2018.1500915
6.
BradshawL. (2015, June24). PARCC diagnostic assessments for mathematics comprehension: A diagnostic classification model approach [Conference paper presentation]. Council of Chief State School Officers (CC-SSO) 2015 National Conference on Student Assessment (NCSA), San Diego, CA, United States. https://ccsso.confex.com/ccsso/2015/webprogram/Presentation/Session4269/Bradshaw_CCSSO_2015.pdf
7.
BradshawL.LevyR. (2019). Interpreting probabilistic classifications from diagnostic psychometric models. Educational Measurement: Issues and Practice, 38(2), 79–88. http://doi.org/10.1111/emip.12247
8.
BrennanR. (2004). Manual for BB-CLASS: A computer program that uses the Beta-Binomial Model for Classification Consistency and Accuracy Version 1.1 [Computer software]. Center for Advanced Studies in Measurement and Assessment.
9.
BuckG.TatsuokaK. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15(2), 119–157. http://doi.org/10.1177/026553229801500201
10.
CarrollJ. B. (1972). Defining language comprehension. In FreedleR. O.CarrollJ. B. (Eds.), Language comprehension and the acquisition of knowledge. John Wiley.
11.
Center for Applied Linguistics. (2019). WIDA Annual technical report for ACCESS for ELLs ® 2.0. Online English Language Proficiency Test Series 402, 2017–2018 administration. WIDA.
12.
ChoiI.PapageorgiouS. (2020). Evaluating subscore uses across multiple levels: A case of reading and listening subscores for young EFL learners. Language Testing, 37(2), 254–279. http://doi.org/10.1177/0265532219879654
13.
CizekG. J. (2012). An introduction to contemporary standard setting: Concepts, characteristics and contexts. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 3–14). Routledge.
14.
CizekG. J.BunchM. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. SAGE Publications.
15.
Council of Europe (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press.
DiBelloL.RoussosL.StoutW. (2007). Cognitive diagnosis Part I. In RaoC. R.SinharayS. (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 979–1030). Elsevier.
20.
EmonsW. H. M.SijtsmaK.MeijerR. R. (2007). On the consistency of individual classification using short scales. Psychological Methods, 12(1), 105–120. http://doi.org/10.1037/1082-989X.12.1.105
21.
FieldJ. (2013). Cognitive validity. In TaylorL.GeranpayehA. (Eds.), Examining listening (pp. 77–151). Cambridge University Press.
22.
FleissJ. L. (1981). Statistical methods for rates and proportions (2nd ed.). John Wiley.
23.
GomezP. G.NoahA.SchedlM.WrightC.YolkutA. (2007). Proficiency descriptors based on a scale-anchoring study of the new TOEFL iBT reading test. Language Testing, 24(3), 417–444. http://doi.org/10.1177/0265532207077209
24.
GreenA. (2018). Linking tests of English for academic purposes to the CEFR: The score user’s perspective. Language Assessment Quarterly, 15(1), 59–74, http://doi.org/10.1080/15434303.2017.1350685
25.
HabermanS.von DavierM. (2006). Some notes on models for cognitively based skills diagnosis. In RaoC.SinharayS. (Ed.), Handbook of statistics (pp. 1031–1038). Elsevier.
26.
HansonB. A.BrennanR. L. (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27(4), 345–359. http://doi.org/10.1111/j.1745-3984.1990.tb00753.x
27.
HardingL.AldersonC.BrunfautT. (2015). Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles. Language Testing, 32(3), 317–336. http://doi.org/10.1177/0265532214564505
28.
HeL.ChenD. (2017). Developing common listening ability scales for Chinese learners of English. Language Testing in Asia, 7(4), 1–12. https://doi.org/10.1186/s40468-017-0033-4
29.
HyattD.BrooksG. (2009). Investigating stakeholders' perceptions of IELTS as an entry requirement for higher education in the UK (IELTS Research Reports, Vol. 10). IELTS Australia and London: British Council. https://www.ielts.org/research/research-reports/volume-10-report-1
JangE. E.DunlopM.ParkG.van der BoomE. H. (2015). How do young students with different profiles of reading skill mastery, perceived ability, and goal orientation respond to holistic diagnostic feedback?Language Testing, 32(3), 359–383. http://doi.org/10.1177/0265532215570924
32.
JangE. E.KimH.VincettM.BarronC.RussellB. (2019). Improving IELTS reading test score interpretations and utilisation through cognitive diagnosis model-based skill profiling (IELTS Research Reports Online Series No. 2). British Council, Cambridge Assessment English and IDP: IELTS Australia. https://www.ielts.org/research/research-reports/online-series-2019-2
33.
JavidanmehrZ.SarabA. M. R. (2019). Retrofitting non-diagnostic reading comprehension assessment: Application of the G-DINA model to a high stakes reading comprehension test. Language Assessment Quarterly, 16(3), 294–311. http://doi.org/10.1080/15434303.2019.1654479
34.
JohnsonM. S.SinharayS. (2018). Measures of agreement assess attribute-level classification accuracy and consistency for cognitive diagnostic assessments. Journal of Educational Measurement, 55(4), 635–664. http://doi.org/10.1111/jedm.12196
35.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with non-parametric item response theory. Applied Psychological Measurement, 25(3), 258–272. http://doi.org/10.1177/01466210122032064
36.
KaftandjievaF. (2010). Methods for setting cut scores in criterion-referenced achievement tests: A comparative analysis of six recent methods with an application to tests of reading in EFL. EALTA.
37.
KaneM. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. http://doi.org/10.1111/jedm.12000
38.
KenyonD. M. (2012). Using Bachman’s Assessment Use Argument as a tool in conceptualizing the issues surrounding linking ACTFL and CEFR. In TschirnerE. (Ed.), Aligning frameworks of reference in language testing: The ACTFL Proficiency Guidelines and the Common European Framework of Reference for Languages (pp. 23–34). Stauffenburg Verlag.
39.
KenyonD. M.RömhildA. (2013). Standard setting in language testing. In KunnanA. J. (Ed.), The companion to language assessment (pp. 944–961). John Wiley & Sons.
40.
KimY. H. (2011). Diagnosing EAP writing ability using the reduced reparameterized unified model. Language Testing, 28(4), 509–541. http://doi.org/10.1177/0265532211400860
41.
KimA.-Y. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing, 32(2), 227–258. http://doi.org/10.1177/0265532214558457
42.
KunnanA. J. (2008). Large scale language assessments. In ShohamyE.HornbergerN. H. (Eds.), Encyclopedia of language and education, Vol. 7: Language testing and assessment (2nd ed., pp. 135–155). Springer.
43.
KunnanA. J.JangE. E. (2009). Diagnostic feedback in language assessment. In LongM. H.DoughtyC. J. (Eds.), The handbook of language teaching (pp. 610–627). Wiley-Blackwell.
44.
LeeY.-W.SawakiY. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239–263. http://doi.org/10.1080/15434300903079562
45.
LeightonJ. P. (2019). Students’ interpretation of formative assessment feedback: Three claims for why we know so little about something so important. Journal of Educational Measurement, 56(4), 793–814. http://doi.org/10.1111/jedm.12237
46.
LeightonJ. P.GierlM. J.HunkaS. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3), 205–237. http://doi.org/10.1111/j.1745-3984.2004.tb01163.x
47.
LiH.HunterC. V.LeiP.-W. (2016). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing, 33(3), 391–409. http://doi.org/10.1177/0265532215590848
LimG. S.GeranpayehA.KhalifaH.BuckendahlC. W. (2013). Standard setting to an International Reference Framework: Implications for theory and practice. International Journal of Testing, 13(1), 32–49. http://doi.org/10.1080/15305058.2012.678526
LiuR.QianH.LuoX.WooA. (2018). Relative diagnostic profile: A subscore reporting framework. Educational and Psychological Measurement, 78(6), 1072–1088. http://doi.org/10.1177/0013164417740170
52.
LivingstonS. A.LewisC. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32(2), 179–197. http://doi.org/10.1111/j.1745-3984.1995.tb00462.x
MartinyukW. (2010). Aligning Tests with the CEFR: Reflections on using the Council of Europe’s draft Manual. Cambridge University Press.
55.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305–328. http://doi.org/10.1080/00273171.2014.911075
56.
MirzaeiA.VinchehM. H.HashemianM. (2020). Retrofitting the IELTS reading section with a general cognitive diagnostic model in an Iranian EAP context. Studies in Educational Evaluation, 64, 1–10. http://doi.org/10.1016/j.stueduc.2019.100817
57.
NeymanJ.PearsonE. S. (1992). On the problem of the most efficient tests of statistical hypotheses. In KotzS.JohnsonN. L. (Eds.), Breakthroughs in statistics (pp. 73–108). Springer.
58.
NicolD. J.Macfarlane-DickD. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. http://doi.org/10.1080/03075070600572090
59.
NorthB. (2000). The development of a common framework scale of language proficiency. Peter Lang.
60.
O’LearyT. M.HattieJ. A.GriffinP. (2017). Actual interpretations and use of scores as aspects of validity. Educational Measurement: Issues and Practice, 36(2), 16–23. http://doi.org/10.1111/emip.12141
61.
O’SullivanB. O.WuS.LiuJ.DunleaJ. (2020). Linking the Aptis test to China’s standards of English language ability. In DamerowR. M.BaileyK. M. (Eds.), Chinese-speaking learners of English: Research, theory, and practice. Routledge.
62.
PapageorgiouS. (2010). Investigating the decision-making process of standard setting participants. Language Testing, 27(2), 261–282. http://doi.org/10.1177/0265532209349472
63.
PapageorgiouS.ChoiI. (2018). Adding value to second-language listening and reading sub-scores: Using a score augmentation approach. International Journal of Testing, 18(3), 207–230. http://doi.org/10.1080/15305058.2017.1407766
64.
PapageorgiouS.TannenbaumR. J. (2016). Situating standard setting within argument-based validity. Language Assessment Quarterly, 13(2), 109–123. http://doi.org/10.1080/15434303.2016.1149857
65.
PapageorgiouS.TannenbaumR. J.BridgemanB.ChoY. (2015). The Association Between TOEFL iBT® Test Scores and the Common European Framework of Reference (CEFR) Levels (Research Memorandum No. RM-15-06). Educational Testing Service. https://www.ets.org/Media/Research/pdf/RM-15-06.pdf
66.
PapageorgiouS.XiX.MorganR.SoY. (2015). Developing and validating band levels and descriptors for reporting overall examinee performance. Language Assessment Quarterly, 12(2), 153–177. http://doi.org/10.1080/15434303.2015.1008480
67.
PowersD.SchedlM.PapageorgiouS. (2017). Facilitating the interpretation of English language proficiency scores: Combining scale anchoring and test score mapping methodologies. Language Testing, 34(2), 175–195. http://doi.org/10.1177/0265532215623582
68.
ReckaseM. D. (2009). Standard setting theory and practice: Issues and difficulties. In FiguerasN.NoijonsJ. (Eds.), Linking to the CEFR levels: Research perspectives (pp. 13–21). EALTA.
69.
RobertsM.GierlM. J. (2010). Developing score reports for cognitive diagnostic assessments. Educational Measurement: Issues and Practice, 29(3), 25–38. http://doi.org/10.1111/j.1745-3992.2010.00181.x
70.
RuppA. A.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
71.
SawakiY.KimH. J.GentileC. (2009). Q-Matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190–209. http://doi.org/10.1080/15434300902801917
72.
SawakiY.KoizumiR. (2017). Providing test performance feedback that bridges assessment and instruction: The case of two standardized English language tests in Japan. Language Assessment Quarterly, 14(3), 234–256. http://doi.org/10.1080/15434303.2017.1348504
SinharayS. (2010). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement, 47(2), 150–174. http://doi.org/10.1111/j.1745-3984.2010.00106.x
75.
SinharayS.JohnsonM. S. (2019). Measures of agreement: Reliability, classification accuracy, and classification consistency. In von DavierM.LeeY.-S. (Eds.), Handbook of diagnostic classification models (pp. 359–377). Springer.
SinharayS.PuhanG.HabermanS. J.HambletonR. K. (2019). Subscores when to communicate them, what are their alternatives, and some recommendations. In Zapata-RiveraD. (Ed.), Score reporting research and applications (pp. 80–107). Routledge.
78.
SkaggsG.HeinS. F.WilkinsJ. L. M. (2016). Diagnostic profiles: A standard setting method for use with a cognitive diagnostic model. Journal of Educational Measurement, 53(4), 448–458. http://doi.org/10.1111/jedm.12125
79.
TannenbaumR. (2019). Validity aspects of score reporting. In Zapata-RiveraD. (Ed.), Score reporting research and applications (pp. 33–52). Routledge.
80.
TannenbaumR. J.ChoY. (2014). Critical factors to consider in evaluating standard-setting studies to map language test scores to frameworks of language proficiency. Language Assessment Quarterly, 11(3), 233–249. http://doi.org/10.1080/15434303.2013.869815
81.
TannenbaumR. J.WylieE. C. (2008). Linking English language test scores onto the Common European Framework of Reference: An application of standard setting methodology (ETS Research Report 08-34). Educational Testing Service. https://www.ets.org/Media/Research/pdf/RR-08-34.pdf
82.
TemplinJ.BradshawL. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251–275. http://doi.org/10.1007/s00357-013-9129-4
83.
TemplinJ.BradshawL. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2), 317–339. http://doi.org/10.1007/s11336-013-9362-0
84.
TemplinJ.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305. http://doi.org/10.1037/1082-989X.11.3.287
85.
ToprakE.AryadoustV.GohC. (2019). The log-linear cognitive diagnosis modeling (LCDM) in second language listening assessment. In AryadoustV.RaquelM. (Eds.), Quantitative data analysis for language assessment, Vol. II: Advanced methods (pp. 56–78). Routledge.
86.
VandergriftL. (2007). Recent developments in second and foreign language listening comprehension research. Language Teaching, 40(3), 191–210. http://doi.org/10.1017/S0261444807004338
87.
VandergriftL.GohC. C. M. (2012). Teaching and learning second language listening. Routledge.
88.
WainerH.WangX.SkorupskiW. P.BradlowE. T. (2005). A Bayesian method for evaluating passing scores: The PPoP curve. Journal of Educational Measurement, 42(3), 271–281. http://doi.org/10.1111/j.1745-3984.2005.00014.x
WangW.SongL.ChenP.MengY.DingS. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for diagnostic assessment. Journal of Educational Measurement, 52(4), 457–476. http://doi.org/10.1111/jedm.12096
91.
XieQ. (2017). Diagnosing university students' academic writing in English: Is cognitive diagnostic modelling the way forward?Educational Psychology, 37(1), 26–47. http://doi.org/10.1080/01443410.2016.1202900
92.
YiY.-S. (2017). Probing the relative importance of different attributes in L2 reading and listening comprehension items: An application of cognitive diagnostic models. Language Testing, 34(3), 1–19. http://doi.org/10.1177/0265532216646141
93.
YoungM. J.YoonB. (1998). Estimating the consistency and accuracy of classifications in a standards-referenced assessment (CSE technical report 475). National Center for Research on Evaluation, Standards, and Student Testing. https://files.eric.ed.gov/fulltext/ED426067.pdf
94.
ZhangB. (2010). Assessing the accuracy and consistency of language proficiency classification under competing measurement models. Language Testing, 27(1), 119–140. http://doi.org/10.1177/0265532209347363
95.
ZiekyM. J.PerieM.LivingstonS. A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Service.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
