This study (1) assesses whether raters implicitly distinguish students' writing expertise and second language proficiency while evaluating ESL compositions holistically and (2) seeks to describe the decision-making behaviours used by experienced and inexperienced raters in this process. The performance of 7 novice and 6 expert ESL teachers was assessed while they evaluated 12 compositions written by adult students with differing levels of ESL proficiency (intermediate and advanced) and writing expertise (average and professionally experienced writers) in their mother tongues. Multivariate analyses of rating scores indicated that both groups' evaluations distinguished students' second language proficiency and writing skills as separate, non-interacting factors. Descriptive analyses of the raters' concurrent verbal reports revealed 28 common decision-making behaviours, many of which varied significantly in use between novice and expert groups. Implications discuss biases in holistic evaluations of second language compositions, aspects of expertise in this skill, and potential uses of this research for the training of composition raters and student-teachers.
Get full access to this article
View all access options for this article.
References
1.
Alexander, P. and Judy, J.1988: The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research58(4), 375-404.
2.
Bereiter, C. and Scardamalia, M.1987: The psychology of written composition. Hillsdale, NJ: Erlbaum.
3.
Canale, M.1981: Communication: How to evaluate it? Bulletin of the Canadian Association ofApplied Linguistics3(2), 77-94.
Chi, M., Glaser, R. and Rees, E.1982: Expertise in problem solving. In Sternberg, R., editor, Advances in the psychology of human intelligence, vol. 1, pp. 7-75. Hillsdale, NJ: Erlbaum.
6.
Clark, C. and Peterson, P.1986: Teachers' thought processes. In Wittrock, M., editor, Handbook of research on teaching, 3rd edition, pp. 255-96. New York, NY: Macmillan.
7.
Cohen, A.1987: Student processing of feedback on their compositions . In Wenden, A. and Rubin J., editors, Learner strategies in language learning , pp. 57-69. Englewood Cliffs, NJ: Prentice-Hall.
8.
Cohen, A.in press:The taking and rating of summary tasks on a foreign-language reading comprehension test. To appear in Hill, C. and Parry, K., editors, The test at the gate: Towards an ethnography of reading assessment.
9.
Cumming, A.1985: Responding to the writing of ESL students. In Maguire, M. and Pare, A. , editors, Patterns of development, pp. 58-75. Ottawa : Canadian Council of Teachers of English.
10.
— 1989: Writing expertise and second language proficiency . Language Learning39(1), 81-141.
11.
— 1990: Student-teachers' conceptions of curriculum: Toward an understanding of language teacher development. TESL Canada Journal7(1), 33-51.
12.
Cumming, A., Rebuffot, J. and Ledwell, M.1989: Reading and summarizing challenging texts in first and second languages. Reading and Writing2(2), 201-19.
13.
Cummins, J. and Swain, M.1986: Bilingualism in education. New York : Longman.
14.
Ericsson, A. and Simon, H.1984: Verbal reports as data. Cambridge, Mass.MIT Press.
15.
Freeman, D.1989: Teacher training, development, and decision making: a model of teaching and related strategies for language teacher education . TESOL Quarterly23(1), 27-45.
16.
Freedman, S.1987: Response to student writing. Urbana, IL: National Council of Teachers of English .
17.
Homburg, T.1984: Holistic evaluation of ESL compositions: Can it be validated objectively?TESOL Quarterly18(1), 87-107.
18.
Jacobs, H., Zinkgraf, A., Wormuth, D., Hartfiel, V. and Hughey, J.1981: Testing ESL composition. Rowley, MA : Newbury House.
19.
Janopoulos, M.1989: Reader comprehension and holistic assessment of second language writing proficiency. Written Communication6(2), 218-37.
20.
Kagan, D.1988: Teaching as clinical problem solving: a critical examination of the analogy and its implications. Review of Educational Research58(4), 482-505.
21.
Kintsch, W.1988: The role of knowledge in discourse comprehension: a construction-integration model. Psychological Review95(2), 163-82.
22.
— 1989: Learning from text. In Resnick, L., editor, Knowing, learning, and instruction , pp. 25-46. Hillsdale, NJ: Erlbaum.
23.
Land, R. and Whitley, C.1989: Evaluating second language essays in regular composition classes: toward a pluralistic U.S. rhetoric. In Johnson, D. and Roen, D., editors, Richness in writing, pp.284-93. New York: Longman.
24.
Leinhardt, G. and Greeno, J.1986: The cognitive skill of teaching. Journal of Educational Psychology78(1), 75-95.
25.
Mendelsohn, D. and Cumming, A.1987: Professors' ratings of language use and rhetorical organization in ESL compositions. TESL Canada Journal5(1), 9-26.
26.
Perfetti, C.1989: There are generalized abilities and one of them is reading . In Resnick, L., editor, Knowing, learning, and instruction, pp. 307-35. Hillsdale, NJ : Erlbaum.
27.
Perkins, D. and Salomon, G.1989: Are cognitive skills context-bound?Educational Researcher18(1), 16-25.
28.
Perkins, K.1983: On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly17(4), 651-71.
29.
Purves, A.1984: In search of an internationally-valid scheme for scoring compositions. College composition and communication35(4), 426-38.
30.
Quellmalz, E.1980: Problems in stabilizing the judgment process. Center for the Study of Evaluation report no. 136. Los Angeles CA: University of California .
31.
Santos, T.1988: Professors' reactions to the writing of nonnative-speaking students. TESOL Quarterly22(1), 69-90.
32.
Stern, H.1983: Language teacher education: an approach to the issues and a framework for discussion. In Alatis, J., Stern, H. and Strevens, P., editors, Applied linguistics and the preparation of second language teachers: Georgetown University round table on languages and linguistics, pp.342-61. Washington, DC : Georgetown University Press.
33.
Wesche, M., Canale, M., Cray, E., Jones, S., Mendelsohn, D., Tumpane, M., Tyacke, M.1987: The Ontario test of English as a second language: A report on the research. Toronto: Ontario Ministry of Colleges and Universities.
34.
Weir, C.1988: Academic writing - can we please all the people all the time? ELT Documents129, 24-34.
35.
Yinger, R.1987: Learning the language of practice. Curriculum Inquiry17(3), 293-318.
36.
Zamel, V.1985: Responding to student writing. TESOL Quarterly19(1), 79-101.
