In second language writing assessment, rating scales and scores from human-mediated assessment have been criticized for a number of shortcomings including problems with adequacy, relevance, and reliability (Hamp-Lyons, 1990; McNamara, 1996; Weigle, 2002). In its testing practice, Euroexam International also detected that the rating scales for writing at B2 had limited discriminating power and did not adequately reflect finer shades of candidate ability. This study sought to investigate whether a level-specific checklist of binary choice items could be designed to yield results that accurately reflect differential degrees of ability in EFL essay writing at level B2. The participants were four language teachers working as independent raters. The study involved the task materials, operational rating scales, reported scores, and candidate scripts from the May 2017 test administration. In a mixed-methods strategy of inquiry, qualitative data from stimulated recall, think-aloud protocols, and semi-structured interviews informed statistical test and item analyses. The results indicated that the checklist items were more transparent, led to increased variance, and contributed to a more coherent candidate language profile than scores from the rating scales. The implications support the recommendation that checklists should be used for level-specific language proficiency testing (Council of Europe, 2001, p. 189).
AldersonJ. C.ClaphamC.WallD. (1995). Language test construction and evaluation. Cambridge University Press.
5.
AllowayA.BowingP.OsváthE.ÖstörZs. (2005). Eurocity teacher’s book: B2 vantage (3rd ed.). Euro Examination Centre.
6.
BachmanL. F. (2004). Statistical analyses for language assessment. Cambridge University Press.
7.
BachmanL. F.PalmerA. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.
8.
BiberD.GrayB.PoonponK. (2011). Should we use characteristics of conversation to measure grammatical complexity in L2 writing development?TESOL Quarterly, 45, 5–35. https://doi.org/10.5054/tq.2011.244483
9.
BonkW.OakeyG. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing, 20(1), 89–110. https://doi.org/10.1191/0265532203lt245oa
10.
BrindleyG. (1991). Defining language ability: The criteria for criteria. In AnivanS. (Ed.), Current developments in language testing. SEAMEO Regional Language Centre.
11.
BrindleyG. (1998). Describing language development? Rating scales and SLA. In BachmanL. F.CohenA. D. (Eds.), Interfaces between second language acquisition and language testing research (pp. 112-140). Cambridge University Press.
12.
BrindleyG. (2000). Task difficulty and task generalizability in competency-based writing assessment. In BrindleyG. (Ed.), Studies in immigrant English language assessment (Vol. 1, pp. 125–157). National Centre for English Language Teaching and Research, Macquarie University.
13.
BroadB. (2003). What we really value: Beyond rubrics in teaching and assessing writing. Utah State Press.
EastM.WeigleS. C. (2016). Innovation in rubric use: Exploring different dimensions [Editorial]. Assessing Writing, 30, 1–2. https://doi.org/10.1016/j.asw.2016.09.001
19.
EbelR. L.FrisbieD. A. (1986). Essentials of educational measurement. Prentice-Hall.
20.
EberharterK. (2018, May). Judgement and decision-making in rating speaking: Exploring the role of cognitive attributes. Paper presented at the meeting of EALTA, Bochum, Germany.
21.
EckesT. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly: An International Journal, 2(3), 197–221. https://doi.org/10.1207/s15434311laq0203_2
22.
EngberC. A. (1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4(2), 139–155. https://doi.org/10.1016/1060-3743(95)90004-7
FerrisD. R. (1994). Lexical and syntactic features of ESL writing students at different levels of L2 proficiency. TESOL Quarterly, 28(2), 414–420. https://doi.org/10.2307/3587446
26.
FulcherG. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13(2), 208–238. https://doi.org/10.1177/026553229601300205
27.
FulcherG.DavidsonF.KempJ. (2011). Effective rating scale development for speaking tests: Performance decision trees. Language Testing, 28(1), 5–29. https://doi.org/10.1177/0265532209359514
28.
GrantL.GintherA. (2000). Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing, 9(2), 123–145. https://doi.org/10.1016/S1060-3743(00)00019-9
29.
Hamp-LyonsL. (1990). Second language writing: Assessment issues. In KrollB. (Ed.), Second language writing: Research insights from the classroom. Cambridge University Press.
30.
Hamp-LyonsL.KrollB. (1996). Issues in ESL writing assessment: An overview. College ESL, 6(1), 52–72.
31.
HarschC.MartinG. (2012). Adapting CEF-descriptors for rating purposes: Validation by a combined rater training and scale revision approach. Assessing Writing, 17, 228–250. https://doi.org/10.1016/j.asw.2012.06.003
32.
HarschC.RuppA. A. (2011). Designing and scaling level-specific writing tasks in alignment with the CEFR: A test-centered approach. Language Assessment Quarterly, 8(1), 1–33. https://doi.org/10.1080/15434303.2010.535575
33.
HughesA. (1989). Testing for language teachers. Cambridge University Press.
34.
IntaraprawatP.SteffensenM. (1995). The use of metadiscourse in good and poor ESL essays. Journal of Second Language Writing, 4(3), 253–272. https://doi.org/10.1016/1060-3743(95)90012-8
35.
KaneM. (2006). Validation. In BrennanR. (Ed.), Educational measurement (4th ed., pp. 17–64). American Council on Education / Praeger.
KimY.-H. (2010). An argument-based validity inquiry into the Empirically-derived Descriptor-based Diagnostic (EDD) assessment in ESL academic writing [Unpublished doctoral dissertation]. University of Toronto.
38.
KimY.-H. (2011). Diagnosing EAP writing ability using the Reduced Reparametrized Unified Model. Language Testing, 28(4), 509–541. https://doi.org/10.1177/0265532211400860
39.
KnochU. (2009). Diagnostic assessment of writing: The development and validation of a rating scale. Peter Lang.
40.
KnochU. (2011). Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from?Assessing Writing, 16, 81–96. https://doi.org/10.1016/j.asw.2011.02.003
41.
KyleK.CrossleyS. (2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34, 12–24. https://doi.org/10.1016/j.jslw.2016.10.003
42.
LinacreJ. M. (2014). The sin of false precision: Too many rating-scale categories. Rasch Measurement Transactions, 28(2), 1463. https://www.rasch.org/rmt/rmt282.pdf
43.
LinacreJ. M. (2017). Facets computer program for many-facet Rasch measurement (Version 3.80.0). Beaverton, OR: https://www.winsteps.com/
LukácsiZ. (2016). Cohesion and writing quality. GlobeEdit.
46.
LumleyT. (2005). Assessing second language writing. The rater’s perspective. Peter Lang.
47.
LumleyT.McNamaraT. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12(1), 54–71. https://doi.org/10.1177/026553229501200104
48.
MaxwellM.FalickT. (1992). Cohesion and quality in deaf and hearing children’s written English. Sign Language Studies, 77, 345–372. https://doi.org/10.1353/sls.1992.0039
49.
McNamaraT. F. (1996). Measuring second language performance. Longman.
50.
McNamaraT. F. (2000). Language testing. Oxford University Press.
51.
MehrensW. A.LehmannI. J. (1991). Measurement and evaluation in education and psychology (4th ed.). Holt, Rinehart and Winston.
52.
MickanP. (2003). “What’s your score?” An investigation into language descriptors for rating written performance. IELTS Australia.
53.
NationI. S. P. (2009). Teaching ESL / EFL reading and writing. Routledge.
54.
NorthB. (1997). The development of a common framework scale or descriptors of language proficiency based on a theory of measurement. In HuhtaA.KohonenV.Kurki-SuonioL.LuomaS. (Eds.), Current developments and alternatives in language assessment (pp. 423–447). University of Jyväskylä.
55.
NorthB. (2000). The development of a common framework scale of language proficiency. Peter Lang.
56.
OrtegaL. (2003). Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics, 24, 492–518. https://doi.org/10.1093/applin/24.4.492
57.
PollittA.MurrayN. (1996). What raters really pay attention to. In MilanovicM.SavilleN. (Eds.), Language testing 3: Performance testing, cognition and assessment (pp. 74–91). Cambridge University Press.
58.
PophamW. J. (1978). Criterion-referenced measurement. Educational Testing Service.
59.
SasakiM. (2000). Toward an empirical model of EFL writing processes: An exploratory study. Journal of Second Language Writing, 9(3), 259–291. https://doi.org/10.1016/S1060-3743(00)00028-X
60.
SchneiderG.LenzP. (2001). A guide for developers of European Language Portfolios. Council of Europe.
61.
StruthersL.LapadatJ. C.MacMillanP. D. (2013). Assessing cohesion in children’s writing: Development of a checklist. Assessing Writing, 18, 187–201. https://doi.org/10.1016/j.asw.2013.05.001
62.
TraceJ.MeierV.JanssenG. (2016). “I can see that”: Developing shared rubric category interpretations through score negotiation. Assessing Writing, 30, 32–43. https://doi.org/10.1016/j.asw.2016.08.001
63.
TurnerC. E.UpshurJ. A. (2002). Rating scales derived from student samples: Effects of the scale maker and the student sample on scale content and student scores. TESOL Quarterly, 36(1), 49–70. https://doi.org/10.2307/3588360
64.
UpshurJ. A.TurnerC. E. (1995). Constructing rating scales for second language tests. ELT Journal, 49(1), 3–12. https://doi.org/10.1093/elt/49.1.3
WeigleS. C. (2002). Assessing writing. Cambridge University Press.
67.
WeirC. J. (2005). Language testing and validation: An evidence-based approach. Palgrave Macmillan.
68.
WigglesworthG. (1993). Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing, 10(3), 305–319. https://doi.org/10.1177/026553229301000306
69.
WrightB.LinacreJ. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.
70.
ZhangM. (2000). Cohesive features in the expository writing of undergraduates in two Chinese universities. RELC Journal, 31(1), 61–95. https://doi.org/10.1177/003368820003100104
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
