The argument-based approach to validity (Kane, 2013) focuses on two steps: (1) making claims about the proposed interpretation and use of test scores as a coherent, interpretive argument; and (2) evaluating those claims based on theoretical and empirical evidence related to test performances and scores. This paper discusses the role of multidimensional (MD) analysis (Biber, 1988), a corpus analytic approach to examining linguistic and discourse characteristics of language performance, in an argument-based validation framework for writing assessment. We propose a set of warrants, assumptions and evidence for inferential steps pertaining to investigations of performance characteristics in language assessments. Then, we illustrate how MD analysis can provide evidence for generalization and explanation inferences through a case study that examines lexico-grammatical features of writing performances on the Examination for the Certification of Proficiency in English (ECPE).
BanerjeeJ.YanX.ChapmanM.ElliottH. (2015). Keeping up with the times: Revising and refreshing a rating scale. Assessing Writing, 26, 5–19. https://doi.org/10.1016/j.asw.2015.07.001
BiberD.ConradS.CortesV. (2004). If you look at. . .: Lexical bundles in university teaching and textbooks. Applied Linguistics, 25(3), 371–405. https://doi.org/10.1093/applin/25.3.371
BiberD.GrayB.StaplesS. (2016). Predicting patterns of grammatical complexity across textual task types and proficiency levels. Applied Linguistics, 37(5), 639–668. https://doi.org/10.1093/applin/amu059
8.
BrooksL.SwainM. (2014). Contextualizing performances: Comparing performances during TOEFL iBT and real-life academic speaking activities. Language Assessment Quarterly, 11(4), 353–373. https://doi.org/10.1080/15434303.2014.947532
9.
ChapelleC. A.ChungY. R.HegelheimerV.PendarN.XuJ. (2010). Towards a computer-delivered test of productive grammatical ability. Language Testing, 27(4), 443–469. https://doi.org/10.1177/0265532210367633
10.
ChapelleC. A.CotosE.LeeJ. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32(3), 385–405. https://doi.org/10.1177/0265532214565386
11.
ChapelleC. A.EnrightM. K.JamiesonJ. M. (2008). Building a validity argument for the Test of English as a Foreign Language. New York: Routledge.
12.
ChenY. H.BakerP. (2016). Investigating criterial discourse features across second language development: Lexical bundles in rated learner essays, CEFR B1, B2 and C1. Applied Linguistics, 37(6), 849–880. https://doi.org/10.1093/applin/amu065
13.
CohenJ. (1988), Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
14.
EgbertJ. (2014). Reader perceptions of linguistic variation in published academic writing. (Unpublished doctoral dissertation). Northern Arizona University, Flagstaff, AZ.
15.
HardyJ. A.RömerU. (2013). Revealing disciplinary variation in student writing: A multi-dimensional analysis of the Michigan Corpus of Upper-level Student Papers (MICUSP). Corpora, 8(2), 183–207. https://doi.org/10.3366/cor.2013.0040
KaneM. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73. https://doi.org/10.1111/jedm.12000
20.
KnochU. (2011). Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from?Assessing Writing, 16(2), 81–96. https://doi.org/10.1016/j.asw.2011.02.003
21.
KnochU.ChapelleC. A. (2018). Validation of rating processes within an argument-based framework. Language Testing, 35(4), 477–499. https://doi.org/10.1177/0265532217710049
22.
KyleK.CrossleyS. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49(4), 757–786. https://doi.org/10.1002/tesq.194
23.
LaFlairG. T.StaplesS. (2017). Using corpus linguistics to examine the extrapolation inference in the validity argument for a high-stakes speaking assessment. Language Testing, 34(4), 451–475. https://doi.org/10.1177/0265532217713951
24.
LaFlairG.T.StaplesS.YanX. (in press). Triangulating corpus linguistics and language assessment: Using corpus linguistics to enhance validity arguments. In BakerP.EgbertJ. (Eds.), Using corpus methods to triangulate linguistic analysis. New York: Routledge.
25.
R Core Team. (2015). R: A language and environment for statistical computing [Internet]. Vienna, Austria: R Foundation for Statistical Computing. http://www.R-project.org/
26.
RevelleW. (2016). Psych: Procedures for psychological, psychometric, and personality research (Version 1.6.9) [Computer software]. Evanston, IL: Northwestern University. http://CRAN.R-project.org/package=psych
27.
SchoonenR.van GelderenA.StoelR. D.HulstijnJ.de GlopperK. (2011). Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, 61(1), 31–79. https://doi.org/10.1111/j.1467-9922.2010.00590.x
Simpson-VlachR.EllisN. C. (2010). An academic formulas list: New methods in phraseology research. Applied Linguistics, 31(4), 487–512. https://doi.org/10.1093/applin/amp058
30.
StaplesS.BiberD.ReppenR. (2018). Using corpus-based register analysis to explore authenticity of high-stakes language exams: A register comparison of TOEFL iBT and disciplinary writing tasks. The Modern Language Journal, 102(2), 310–332. https://doi.org/10.1111/modl.12465
31.
StaplesS.EgbertJ.BiberD.McClairA. (2013). Formulaic sequences and academic writing development: Lexical bundles in the TOEFL iBT writing section. Journal of English for Academic Purposes, 12(3), 214–225. https://doi.org/10.1016/j.jeap.2013.05.002
32.
StaplesS.LaFlairG.EgbertJ. (2014). Investigating the multi-faceted nature of speaking performance with a multivariate method. Paper presented at the meeting of the Midwest Association of Language Testers. Ann Arbor, MI.
WeigleS. C.FriginalE. (2015). Linguistic dimensions of impromptu test essays compared with successful student disciplinary writing: Effects of language background, topic, and L2 proficiency. Journal of English for Academic Purposes, 18, 25–39. https://doi.org/10.1016/j.jeap.2015.03.006
35.
YuG. (2009). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31(2), 236–259. https://doi.org/10.1093/applin/amp024
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
