Raters are central to writing performance assessment, and rater development – training, experience, and expertise – involves a temporal dimension. However, few studies have examined new and experienced raters’ rating performance longitudinally over multiple time points. This study uses operational data from the writing section of the MELAB (n = 20,662 ratings), an international exam of English proficiency, to investigate the rating quality of new and experienced raters over three time periods of 12 to 21 months. Rating quality was operationalized in terms of rater severity and consistency, and estimates of those modeled using multi-facet Rasch methodology. Results indicate that, within one particular rating context, (1) novice raters, where initially differing in performance, learn to rate appropriately relatively quickly, (2) raters are able to maintain rating quality over time, and (3) rating volume and rating quality may be related. Implications for rater preparation, rater certification, and the notion of expert rater are discussed.
BarkaouiK. (2007). Participants, texts, and processes in ESL/EFL essay tests: A narrative review of the literature. Canadian Modern Language Review, 64(1), 99–134.
2.
BarkaouiK. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54–74.
3.
BondT. G.FoxC. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
4.
BrelandH.LeeY. W.NajarianM.MurakiE. (2004). An analysis of TOEFL CBT writing prompt difficulty and comparability for different gender groups. TOEFL Research Reports, RR-04-05. Princeton, NJ: Educational Testing Service.
5.
BroerM.LeeY. W.RizaviS.PowersD. (2005). Ensuring the fairness of GRE writing prompts: Assessing differential difficulty. ETS Research Report, RR 05-11. Princeton, NJ: Educational Testing Service.
6.
Chalhoub-DevilleM. (2003). Fundamentals of ESL admissions tests: MELAB, IELTS, and TOEFL. In DouglasD. (Ed.), English language testing in US colleges and universities (2nd ed., pp. 11–35). Washington, DC: NAFSA.
7.
ChoD. W. (1999). A study of ESL writing assessment: Intra-rater reliability of ESL compositions. Melbourne Papers in Language Testing, 8(1), 1–24.
8.
CongdonP. J.McQueenJ. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163–178.
9.
CummingA. (1990). Expertise in evaluating second language compositions. Language Testing, 7(1), 31–51.
10.
CummingA.KantorR.PowersD. E. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. The Modern Language Journal, 86(1), 67–96.
11.
EllisN. C. (2002). Frequency effects in language processing: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24(2), 143–188.
12.
EngelhardG. (1992). The measurement of writing ability with a many-faceted Rasch model. Applied Measurement in Education, 5(3), 171–191.
13.
English Language Institute, University of Michigan. (2005). Michigan English language assessment battery: Technical manual 2003. Ann Arbor, MI: English Language Institute, University of Michigan.
14.
EricssonK. A.CharnessN.FeltovichP. J.HoffmanR. R. (Eds.) (2006). The Cambridge handbook of expertise and expert performance. Cambridge: Cambridge University Press.
15.
FoxJ. (2003). From products to process: An ecological approach to bias detection. International Journal of Testing, 3(1), 21–48.
16.
Hamp-LyonsL. (1991). Basic concepts. In Hamp-LyonsL. (Ed.), Assessing second language writing in academic contexts (pp. 5–15). Norwood, NJ: Ablex.
17.
HenningG. (1992). Scalar analysis of the Test of Written English. TOEFL Research Reports, RR-92-30. Princeton, NJ: Educational Testing Service.
18.
HuotB. A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In WilliamsonM. M.HuotB. A. (Eds.), Validating holistic scoring for writing assessment (pp. 206–232). Cresskill, NJ: Hampton Press.
19.
JohnsonJ. S. (2004). Score gains for repeat MELAB administrations. ELI Research Reports 2004-04. Ann Arbor, MI: English Language Institute, University of Michigan.
20.
JohnsonJ. S. (2005). MELAB 2004 descriptive statistics and reliability estimates. ELI Research Reports 2005-03. Ann Arbor, MI: English Language Institute, University of Michigan.
21.
JohnsonJ. S. (2006). MELAB 2005 descriptive statistics and reliability estimates. ELI Research Reports 2006-02. Ann Arbor, MI: English Language Institute, University of Michigan.
22.
JohnsonJ. S. (2007). MELAB 2006 descriptive statistics and reliability estimates. ELI Research Reports 2007-03. Ann Arbor, MI: English Language Institute, University of Michigan.
23.
JohnsonJ. S.LimG. S. (2009). The influence of rater language background on writing performance assessment. Language Testing, 26(4), 485–505.
24.
JohnsonJ. S.SongT. (2008). MELAB 2007 descriptive statistics and reliability estimates. ELI Research Reports 2008-03. Ann Arbor, MI: English Language Institute, University of Michigan.
25.
Kondo-BrownK. (2002). A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing, 19(1), 3–31.
26.
LaneS.StoneC. A. (2006). Performance assessment. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 387–431). Westport, CT: American Council on Education.
27.
LeeY. W.BrelandH.MurakiE. (2004). Comparability of TOEFL CBT prompts for different native language groups. TOEFL Research Reports, RR-04-24. Princeton, NJ: Educational Testing Service.
28.
LimG. S. (2009). Prompt and rater effects in second language writing performance assessment. Retrieved from ProQuest Dissertations and Theses (Accession Order No. AAT 3392954).
29.
LimG. S. (2010). Investigating prompt effects in writing performance assessment. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 8, 95–116.
30.
LinacreJ. M. (1989). Many-facet Rasch measurement. Chicago: MESA Press.
31.
LinacreJ. M. (2002). What do infit, outfit, mean-square and standardized mean?Rasch Measurement Transactions, 16, 878.
32.
LinacreJ. M. (2006). Facets Rasch measurement computer program. Chicago: Winsteps.com.
33.
LumleyT. (2005). Assessing second language writing: The rater’s perspective. Frankfurt am Main: Peter Lang.
34.
MislevyR. J. (2008, September). Some implications of expertise research for educational assessment. Keynote address delivered at the 34th International Association for Educational Assessment Conference, Cambridge, UK.
35.
MyfordC. M.WolfeE. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement, 46(4), 371–389.
36.
O‘SullivanB.RignallM. (2007). Assessing the value of bias analysis feedback to raters for the IELTS writing module. In TaylorL.FalveyP. (Eds.), IELTS collected papers. Research in speaking and writing performance (pp. 446–478). Cambridge: Cambridge University Press.
37.
PulaJ. J.HuotB. A. (1993). A model of background influences on holistic raters. In WilliamsonM. M.HuotB. A. (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 237–265). Cresskill, NJ: Hampton Press.
38.
PurpuraJ. E. (2005). Michigan English language assessment battery (MELAB). In StoynoffS.ChapelleC. A. (Eds.), ESOL tests and testing (pp. 87–91). Alexandria, VA: TESOL.
39.
SantosT. (1988). Professors’ reactions to the academic writing of non-native-speaking students. TESOL Quarterly, 22(1), 69–90.
40.
ShawS. D.WeirC. J. (2007). Examining writing: Research and practice in assessing second language writing. Cambridge: Cambridge University Press.
41.
ShiL. (2001). Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing. Language Testing, 18(3), 303–325.
42.
ShohamyE.GordonC. M.KraemerR. (1992). The effect of raters’ background and training on the reliability of direct writing tests. The Modern Language Journal, 76(1), 27–33.
43.
WeigleS. C. (1994). Effects of training on raters of ESL compositions. Language Testing, 11(2), 197–223.
44.
WeigleS. C. (1998). Using FACETS to model rater training effects. Language Testing, 15(2), 263–287.
45.
WeigleS. C. (2002). Assessing writing. Cambridge: Cambridge University Press.
46.
WolfeE. W.KaoC. W.RanneyM. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15(4), 465–492.
