This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program. Using many-facet Rasch modeling, rater performance was examined in terms of rater agreement, rater consistency, and rater severity. These measurement estimates of rating quality were subjected to multivariate analysis to examine whether and how rater performance changes across rounds. Rater comments on the essays were qualitatively analyzed to obtain a deeper understanding of how raters learn to use the scale over time. The quantitative results showed a non-linear, three-staged developmental pattern of rater performance for all three groups of raters. Findings of this study suggest that rater development resembles a learning curve similar to how one acquires a language and other skills. We argue that understanding the developmental pattern of rater behavior is crucial not only to understanding the effectiveness of rater training, but also to the investigation of rater cognition and development. We will also discuss the practical implications of this study in relation to the effort and expectations needed for rater training for writing assessments.
AttaliY. (2016). A comparison of newly-trained and experienced raters on a standardized writing assessment. Language Testing, 33(1), 99–115. https://doi.org/10.1177/0265532215582283
2.
BachmanL. F.LynchB. K.MasonM. (1995). Investigating variability in tasks and rater judgements in a performance test of foreign language speaking. Language Testing, 12(2), 238–257. https://doi.org/10.1177/026553229501200206
3.
BarkaouiK. (2010). Do ESL essay raters’ evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44(1), 31–57. https://doi.org/10.5054/tq.2010.214047
4.
BonkW. J.OckeyG. J. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing, 20(1), 89–110. https://doi.org/10.1191/0265532203lt245oa
5.
ChoY. (2001). Examining a process-oriented writing assessment in a large-scale ESL testing context [Unpublished Doctoral dissertation]. University of Illinois at Urbana-Champaign.
6.
ColeM.ColeS.LightfootC. (2005). The development of children (5th ed.). Worth.
7.
CongdonP. J.McQueenJ. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163–178. https://doi.org/10.1111/j.1745-3984.2000.tb01081.x
8.
CronbachL. J. (1990). Essentials of psychological testing (5th ed.). Harper & Row.
CummingA.KantorR.PowersD. E. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. The Modern Language Journal, 86(1), 67–96. https://doi.org/10.1111/1540-4781.00137
11.
DavisL. (2016). The influence of training and experience on rater performance in scoring spoken language. Language Testing, 33(1), 117–135. https://doi.org/10.1177/0265532215582282
EckesT. (2011). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments. Peter Lang. https://doi.org/10.3726/978-3-653-04844-5
14.
ElderC.BarkhuizenG.KnochU.Von RandowJ. (2007). Evaluating rater responses to an online training program for L2 writing assessment. Language Testing, 24(1), 37–64. https://doi.org/10.1177/0265532207071511
15.
ElderC.KnochU.BarkhuizenG.von RandowJ. (2005). Individual feedback to enhance rater training: Does it work?Language Assessment Quarterly, 2(3), 175–196. https://doi.org/10.1207/s15434311laq0203_1
16.
Hamp-LyonsL. (1995). Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29(4), 759–762. https://doi.org/10.2307/3588173
17.
JohnsonJ. S.LimG. S. (2009). The influence of rater language background on writing performance assessment. Language Testing, 26(4), 485–505. https://doi.org/10.1177/0265532209340186
18.
KangO. (2008). Ratings of L2 oral performance in English: Relative impact of rater characteristics and acoustic measures of accentedness [Unpublished doctoral dissertation]. University of Georgia.
19.
Karmiloff-SmithA. (1992). Beyond modularity: A developmental perspective on cognitive science. MIT Press.
20.
KauperN. L. (2012). Development and implementation of an ESL classroom assessment of face-to-face conversational interaction [Unpublished doctoral dissertation]. Purdue University.
KimH.BowlesM.YanX.ChungS. (2018). Examining the comparability between paper–pencil and computer-mediated versions of an integrated writing placement. Assessing Writing, 36, 49–62. https://doi.org/10.1016/j.asw.2018.03.006
23.
KnochU. (2011). Investigating the effectiveness of individualized feedback to rating behavior—A longitudinal study. Language Testing, 28(2), 179–200. https://doi.org/10.1177/0265532210384252
24.
LamprianouI.TsagariD.KyriakouN. (2021). The longitudinal stability of rating characteristics in an EFL examination: Methodological and substantive considerations. Language Testing, 38(2), 273–301. https://doi.org/10.1177/0265532220940960
25.
LimG. S. (2011). The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters. Language Testing, 28(4), 543–560. https://doi.org/10.1177/0265532211406422
26.
LinacreJ. M. (1989). Many-faceted Rasch measurement. MESA.
27.
LinacreJ. M. (2010). FACETS (Version 3.67.1) [Computer software]. SWReg Digital River.
LingG.MollaunP.XiX. (2014). A study on the impact of fatigue on human raters when scoring speaking responses. Language Testing, 31(4), 479–499. https://doi.org/10.1177/0265532214530699
31.
LumleyT. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters?Language Testing, 19(3), 246–276. https://doi.org/10.1191/0265532202lt230oa
32.
LumleyT. (2005). Assessing second language writing: The rater’s perspective. Peter Lang.
33.
LumleyT.McNamaraT. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12(1), 54–71. https://doi.org/10.1177/026553229501200104
McNamaraT.KnochU. (2012). The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing, 29(4), 555–576. https://doi.org/10.1177/0265532211430367
36.
McNamaraT. F. (1996). Measuring second language performance. Addison Wesley Longman.
37.
MilesM. B.HubermanA. M. (1994). Qualitative data analysis: An expanded sourcebook. SAGE.
38.
MyfordC. M.WolfeE. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422.
ParkH.YanX. (2019). An investigation into rater performance with a holistic scale and a binary, analytic scale on an ESL writing placement test. Papers in Language Testing and Assessment, 8(2), 34–64.
41.
ShawS. (2002). The effect of training and standardisation on rater judgement and inter-rater reliability. Research Notes, 8(5), 13–17.
42.
SongB.CarusoI. (1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students?Journal of Second Language Writing, 5(2), 163–182. https://doi.org/10.1016/S1060-3743(96)90023-5
VaughanC. (1991). Holistic assessment: What goes on in the rater’s mind? In Hamp-LyonsL. (Ed.), Assessing second language writing in academic contexts (pp. 111–125). Ablex.
45.
WangJ.EngelhardG.WolfeE. (2016). Evaluating rater accuracy in rater-mediated assessments using an unfolding model. Educational and Psychological Measurement, 76(6), 1005–1025. https://doi.org/10.1177/0013164415621606
WigglesworthG. (1993). Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing, 10(3), 305–319. https://doi.org/10.1177/026553229301000306
48.
WoehrD. J.HuffcuttA. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189–205. https://doi.org/10.1111/j.2044-8325.1994.tb00562.x
49.
XiX.MollaunP. (2009). How do raters from India perform in scoring the TOEFL iBT speaking section and what kind of training helps? (TOEFL iBT Research Report RR-09-31). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2009.tb02188.x
50.
YanX. (2014). An examination of rater performance on a local oral English proficiency test: A mixed-methods approach. Language Testing, 31(4), 501–527. https://doi.org/10.1177/0265532214536171
51.
YanX.GintherA. (2017). Listeners and raters: Similarities and differences in evaluation of accented speech. In KangO.GintherA. (Eds.), Assessment in L2 pronunciation (pp. 67–88). Routledge.
52.
YanX.KimH. R.KotnarowskiJ. (2021). Development of a Profile-Based Writing Scale: How collaboration with teachers enhanced assessment practice in a post-admission ESL writing program at a USA University. In LanteigneB.CoombeC.BrownJ. (Eds.), Challenges in language testing around the world (pp. 529–546). Springer. https://doi.org/10.1007/978-981-33-4232-3_36
53.
ZhangJ. (2016). Same text, different processing? Exploring how raters’ cognitive and meta-cognitive strategies influence rating accuracy in essay scoring. Assessing Writing, 27, 37–53. https://doi.org/10.1016/j.asw.2015.11.001
