Assessment criteria in a large-scale writing test: what do they really mean to the raters?

Abstract

The process of rating written language performance is still not well understood, despite a body of work investigating this issue over the last decade or so (e.g., Cumming, 1990; Huot, 1990; Vaughan, 1991; Weigle, 1994a; Milanovic et al., 1996). The purpose of this study is to investigate the process by which raters of texts written by ESL learners make their scoring decisions using an analytic rating scale designed for multiple test forms. The context is the Special Test of English Proficiency (step), which is used by the Australian government to assist in immigration decisions. Four trained, experienced and reliable step raters took part in the study, providing scores for two sets of 24 texts. The first set was scored as in an operational rating session. Raters then provided think-aloud protocols describing the rating process as they rated the second set. A coding scheme developed to describe the think-aloud data allowed analysis of the sequence of rating, the interpretations the raters made of the scoring categories in the analytic rating scale, and the difficulties raters faced in rating.

Data show that although raters follow a fundamentally similar rating process in three stages, the relationship between scale contents and text quality remains obscure. The study demonstrates that the task raters face is to reconcile their impression of the text, the specific features of the text, and the wordings of the rating scale, thereby producing a set of scores. The rules and the scale do not cover all eventualities, forcing the raters to develop various strategies to help them cope with problematic aspects of the rating process. In doing this they try to remain close to the scale, but are also heavily influenced by the complex intuitive impression of the text obtained when they first read it. This sets up a tension between the rules and the intuitive impression, which raters resolve by what is ultimately a somewhat indeterminate process. In spite of this tension and indeterminacy, rating can succeed in yielding consistent scores provided raters are supported by adequate training, with additional guidelines to assist them in dealing with problems. Rating requires such constraining procedures to produce reliable measurement.

Get full access to this article

View all access options for this article.

References

Alderson, J. C. 1991: Bands and scores. In Alderson, J. C. and North, B. , editors, Language testing in the 1990s. London: Macmillan , 71-86.

Brindley, G. and Wigglesworth, G. , editors, 1997: access: Issues in English language test design and delivery. Sydney: (NCELTR), Macquarie University .

Charney, D. 1984: The validity of using holistic scoring to evaluate writing: a critical overview . Research in the Teaching of English 18, 65-81 .

Clapham, C. and Alderson, J. C. , editors, 1996: Constructing and trialling the IELTS test. IELTS Research Report 3. Cambridge: The British Council, The University of Cambridge Local Examinations Syndicate and the International Development Program of Australian Universities and Colleges .

Clark, J. L. D. and Clifford, R. T. 1988: The FSI/ILR/ACTFL proficiency scales and testing techniques: development, current status, and needed research . Studies in Second Language Acquisition 10, 129-147 .

Connor-Linton, J. 1995: Looking behind the curtain: what do L2 composition ratings really mean? TESOL Quarterly 29: 762-765 .

Cumming, A. 1990: Expertise in evaluating second language compositions . Language Testing 7, 31-51 .

Cumming, A. 1997: The testing of writing in a second language. In Clapham, C. and Corson, D. , editors, Encyclopedia of language and education, Volume 7: Language testing and assessment. Dordrecht, Netherlands: Kluwer , 51-63.

Cumming, A. , Kantor, R. and Powers, D. 2001: Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters’ decision making, and development of a preliminary analytic framework. TOEFL Monograph Series. Princeton, NJ: Educational Testing Service .

10.

DIEA 1995: 1 November 1993: Decisions, Fact Sheet 20, Revised 23 March 1995. Canberra: Department of Immigration and Ethnic Affairs.

11.

Ericsson, K. A. and Simon, H. A. 1993: Protocol analysis: verbal reports as data (2nd edn). Cambridge, MA: MIT Press .

12.

Freedman, S. W. and Calfee, R. C. 1983: Holistic assessment of writing: experimental design and cognitive theory. In Mosenthal, P. , Tamor, L. and Walmsley, S. , editors, Research on writing: principles and methods. New York: Longman , 75-98.

13.

Green, A. J. K. 1997: Verbal protocol analysis in language testing research. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate .

14.

Hamp-Lyons, L. 1990: Second language writing: assessment issues. In Kroll, B. , editor, Second language writing. Cambridge: Cambridge University Press , 69-87.

15.

Hamp-Lyons, L. 1991: Scoring procedures. In Hamp-Lyons, L. , editor, Assessing second language writing in academic contexts. Norwood, NJ: Ablex , 241-276.

16.

Hamp-Lyons, L. 1996: The challenge of second-language writing assessment. In White, E. M. , Lutz, W. D. and Kamusikiri, S. , editors, Assessment of writing: politics, policies, practices. New York: Modern Language Association of America , 226-240.

17.

Hawthorne, L. 1996: The politicisation of English: the case of the step test and the Chinese students . Australian Review of Applied Linguistics, Series S 13, 13-32 .

18.

Huot, B.A. 1988: The validity of holistic scoring: a comparison of the talk-aloud protocols of expert and novice holistic raters. Unpublished PhD thesis. Indiana University of Pennsylvania.

19.

Hawthorne, L. 1990: Reliability, validity and holistic scoring: what we know and what we need to know . College Composition and Communication 41, 201-213 .

20.

Hawthorne, L. 1993: The influence of holistic scoring procedures on reading and rating student essays. In Williamson, M. and Huot, B. , editors, Validating holistic scoring for writing assessment: theoretical and empirical foundations. Cresskill, NJ: Hampton Press .

21.

Ingram, D. E. and Wylie, E. 1984: Australian second language proficiency ratings. Canberra: Australian Department of Immigration and Ethnic Affairs .

22.

Jacobs, H. L. , Zinkgraf, S. A. , Wormuth, D. R. , Hartfiel, V. F. and Hughey, J. B. 1981: Testing ESL composition: a practical approach. Rowley, MA: Newbury House .

23.

Kroll, B. 1998: Assessing writing abilities . Annual Review of Applied Linguistics 18, 219-240 .

24.

Kroll, B. and Reid, J. 1994: Guidelines for designing writing prompts: clari-fications, caveats and cautions . Journal of Second Language Writing 3, 231-255 .

25.

Linacre, J. M. and Wright, B. 1992-96: FACETS. Chicago, IL: MESA Press .

26.

Lumley, T. 2000: The process of the assessment of writing performance: the rater’s perspective. Unpublished PhD thesis, Department of Linguistics and Applied Linguistics, The University of Melbourne.

27.

McIntyre, P.N. 1993: The importance and effectiveness of moderation training on the reliability of teachers’ assessments of ESL writing samples. Unpublished MA thesis, Department of Applied Linguistics, University of Melbourne.

28.

McNamara, T.F. 1990: Assessing the second language proficiency of health professionals. Unpublished PhD thesis, University of Melbourne.

29.

Linacre, J. M. and Wright, B. 1996: Measuring second language performance. London: Longman .

30.

Messick, S. 1989: Validity. In Linn, R. L. , editor, Educational measurement (3rd edn). Washington, DC: The American Council on Education and the National Council on Measurement in Education , 13-103.

31.

Selected Papers from the 15th Language Testing Research Colloquium, Cambridge and Arnhem.

32.

Moss, P. A. 1994: Can there be validity without reliability? Educational Researcher 23, 5-12 .

33.

Moss, P. A. 1996: Enlarging the dialogue in educational measurement: voices from interpretive research traditions . Educational Researcher 25, 20-28 .

34.

North, B. 1995: Scales of language proficiency . Melbourne Papers in Language Testing 4, 60-111 .

35.

North, B. and Schneider, G. 1998: Scaling descriptors for language proficiency scales . Language Testing 15, 217-262 .

36.

Pressley, M. and Afflerbach, P. 1995: Verbal protocols of reading: the nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum .

37.

Pula, J. J and Huot, B. A. 1993: A model of background influences on holistic raters. In Williamson, M. M. and Huot, B. A. , editors, Validating holistic scoring for writing assessment: theoretical and empirical foundations. Cresskill, NJ: Hampton Press , 237-265.

38.

Shohamy, E. 2001: The power of tests: a critical perspective on the uses and consequences of language tests. Harlow: Longman/Pearson .

39.

Shohamy, E. , Gordon, C. and Kraemer, R. 1992: The effect of raters’ background and training on the reliability of direct writing tests . Modern Language Journal 76, 27-33 .

40.

Smagorinsky, P. 1994: Potential problems and problematic potentials of using talk about writing as data about writing processes. In Smagorinsky, P. , editor, Speaking about writing: reflections on research methodology. Thousand Oaks, CA: Sage , ix-xviii.

41.

Spolsky, B. 1995. Measured words. Oxford: Oxford University Press .

42.

Torrance, H. 1998: Learning from research in assessment: a response to writing assessment - raters’ elaboration of the rating task . Assessing Writing 5, 31-37 .

43.

Vaughan, C. 1991: Holistic assessment: what goes on in the rater’s mind? In Hamp-Lyons, L. , editor, Assessing second language writing in academic contexts. Norwood, NJ: Ablex , 111-125.

44.

Weigle, S.C. 1994a: Effects of training on raters of English as a second language compositions: quantitative and qualitative approaches. Unpublished PhD dissertation, University of California, Los Angeles.

45.

Vaughan, C. 1994b: Effects of training on raters of ESL compositions . Language Testing 11, 197-223 .

46.

Vaughan, C. 1998: Using FACETS to model rater training effects . Language Testing 15, 263-287 .

47.

Widdowson, H. G. 1983: Learning purpose and language use. Oxford: Oxford University Press .

48.

Zhang, W. 1998: The rhetorical patterns found in Chinese EFL student writers’ examination essays in English and the influence of these patterns on rater response. Unpublished PhD thesis, Hong Kong Polytechnic University.