Several effects of training on composition raters have been hypothesized but not investigated empirically. This article presents an analysis of the verbal protocols of four inexperienced raters of ESL placement compositions scoring the same essays both before and after rater training. The verbal protocols show that training clarified the intended scoring criteria for raters, modified their expectations of student writing and provided a reference group of other raters with which raters could compare themselves, although agreement with peers was not an over-riding concern. These results are generally in accordance with hypothesized effects of rater training.
Get full access to this article
View all access options for this article.
References
1.
Barritt, L., Stock, P. and Clark, F.1986: Researching practice: evaluating assessment essays. College Composition and Communication37, 315-27.
2.
Charney, D.1984: The validity of using holistic scoring to evaluate writing: a critical overview. Research in the Teaching of English18, 65-81.
3.
Coffman, W.1971: On the reliability of ratings of essay examinations in English. Research in the Teaching of English7, 356-71.
4.
Cooper, C.R.1977: Holistic evaluation of writing. In Cooper, C.R. and Odell, L., editors, Evaluating writing: describing, measuring, judging. Urbana, IL: NCTE, 3-31.
5.
Cumming, A.1990: Expertise in evaluating second language compositions. Language Testing7,31-51.
6.
Ericsson, K.A. and Simon, H.1993: Protocol analysis: verbal reports as data. Cambridge, MA: MIT Press.
7.
Follman, J. and Anderson, J.1967: An investigation of the reliability of five procedures for grading English themes. Research in the Teaching of English1,190-200.
8.
Freedman, S.W. and Calfee, R.C.1983: Holistic assessment of writing: experimental design and cognitive theory. In Mosenthal, P., Tamor, L. and Walmsley, S. , editors, Research on writing: principles and methods. New York: Longman, 75-98.
9.
Gere, A.R.1980: Written composition: towards a theory of evaluation. College English42, 44-58.
10.
Hamp-Lyons, L.1990: Second language writing: assessment issues. In Kroll, B., editor, Second language writing: research insights for the classroom. Cambridge: Cambridge University Press, 69-87.
11.
Homburg, T.J.1984: Holistic evaluation of ESL compositions: can it be validated objectively?TESOL Quarterly18, 87-107.
12.
Huot, B.1988: The validity of holistic scoring: a comparison of the talk-aloud protocols of expert and novice holistic raters. Unpublished PhD dissertation, Indiana University of Pennsylvania.
13.
— 1990: Reliability, validity, and holistic scoring: what we know and what we need to know. College Composition and Communication41, 201-13.
Nisbett, R.E. and Wilson, T.D.1977: Telling more than we can know: verbal reports on mental processes. Psychological Review84, 231-59.
16.
Vaughan, C.1992: Holistic assessment: what goes on in the rater's mind ? In Hamp-Lyons, L., editor, Assessing second language writing in academic contexts. Norwood, NJ: Ablex, 111-26.
17.
Weigle, S.1994: Effects of training on raters of ESL compositions: quantitative and qualitative approaches. PhD dissertation, University of California, Los Angeles.
