Two kinds of scoring templates were empirically derived from summaries written by experts and students to evaluate the quality of summaries written by the students. This paper reports students' attitudes towards the use of the two templates and its differential statistical effects on the judgment of students' summarization performance. It was found that a summary consistently received higher scores when judged by the popular template, regardless of the language and the language order (English then Chinese or Chinese then English) in which a summary was produced. However, the scores from the expert template were slightly better predictors of students' reading abilities as measured by FCE and TOEFL. The majority of the students strongly preferred the expert template. Their arguments centered mainly on the differences between students and experts in their language abilities and experience, their stereotypical status in educational assessment, and the dialectical interpretations of `quantity' and `quality'. Implications of these findings are discussed with specific reference to the value of involving test-takers in assessment criteria development.
Get full access to this article
View all access options for this article.
References
1.
Afflerbach, P.P.1990: The influence of prior knowledge on expert readers' main idea construction strategies. Reading Research Quarterly25: 31—46.
2.
Alderson, J.C.1996: The testing of reading. In Nuttall, C., editor, Teaching reading skills in a foreign language. London: Heinemann.
3.
——— 2000: Assessing reading. Cambridge: Cambridge University Press.
4.
Alderson, J.C., Clapham, C. and Wall, D.1995: Language test construction and evaluation. Cambridge: Cambridge University Press.
5.
Axelrod, J.1975: Getting the main idea is still the main idea. Journal of Reading18: 383—87.
6.
Bachman, L.F.1990: Fundamental considerations in language testing. Oxford: Oxford University Press.
7.
Bachman, L.F. and Palmer, A.S.1996: Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.
8.
Bachman, L.F., Davidson, F., Ryan, K. and Choi, I.-C.1995: An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study. Cambridge: Cambridge University Press.
9.
Balajthy, E. and Weisberg, R.1990: Transfer effects of prior knowledge and use of graphic organizers on college developmental readers' summarization and comprehension of expository text. National Reading Conference Yearbook 39: 339—45.
10.
Barnwell, D.1989: “Naïve” native speakers and judgments of oral proficiency in Spanish. Language Testing6: 152—63.
11.
Basham, C.S.1987: Summary writing as cultural artifact. Fairbanks, AK: Cross Cultural Communications, University of Alaska.
12.
Bensoussan, M. and Kreindler, I.1990: Improving advanced reading comprehension in a foreign language: Summaries vs. Short-answer questions. Journal of Research in Reading13: 55—68.
13.
Bernhardt, E.B.1991: Reading development in a second language: Theoretical, empirical, and classroom perspectives. Norwood, NJ: Ablex.
14.
Birenbaum, M.1996: Assessment 2000: Towards a pluralistic approach to assessment . In Birenbaum, M. and Dochy, F.R.J., editors, Alternatives in assessment of achievements, learning processes, and prior knowledge. Dordrecht: Kluwer, 3—29.
15.
Boyd, K. and Davies, A.2002: Doctors' orders for language testers: The origin and purpose of ethical codes. Language Testing19: 296—322.
16.
Brown, A.L., Campione, J.C. and Day, J.D.1981: Learning to learn: On training students to learn from texts. Educational Researcher10: 14—24.
17.
Brown, A.L. and Day, J.D.1983: Macrorules for summarizing texts: The development of expertise . Journal of Verbal Learning and Verbal Behavior22: 1—14.
18.
Brown, A.L., Day, J.D., and Jones, R.S.1983: The development of plans for summarizing texts. Child Development54: 968—79.
19.
Brown, A.L. and Smiley, S.S.1978: The development of strategies for studying text. Child Development49: 1076—88.
20.
Bueckendorf, J.M.1992: Comparative assessment of reading comprehension: Using multiple-choice and written summary formats with narrative and expository texts. Unpublished EdD thesis, University of Missouri, St Louis.
21.
Byrd, M.1985: Age differences in the ability to recall and summarize textual information. Experimental Aging Research11: 87—91.
22.
Cohen, A.D.1993: The role of instructions in testing summarizing ability . In Douglas, D. and Chapelle, C., editors, A new decade of language testing research. Washington, DC: TESOL, 132—59.
23.
——— 1994: English for academic purposes in Brazil: The use of summary tasks. In Hill, C. and Parry, K., editors, From testing to assessment: English as an international language. London: Longman, 174—204.
24.
Cohen, J.1988: Statistical power analysis for the behavioral sciences . Hillsdale, NJ: Lawrence Erlbaum .
25.
Connor, U. and McCagg, P.1983: Cross-cultural differences and perceived quality in written paraphrases of English expository prose. Applied Linguistics4: 259—68.
26.
Corder, S.P.1973: Interpretive procedures: Seeing, reading and understanding discourse. In Anderson, R., editor, New dimensions in second language acquisition research. Rowley, MS: Newbury House.
27.
Courchêne, R. and Bayliss, D.1995: Summary cloze: What is it? What does it measure? In Courchene, R., Burger, S., Cornaire, C., LeBlanc, R., Paribakht, S. and Seguin, H., editors, Twenty-five years of second language teaching at the University of Ottawa . Ottawa: Second Language Institute , University of Ottawa, 305—27.
28.
Cronbach, L.J., Linn, R.L., Brennan, R.L. and Haertel, E.1995: Generalizability analysis for educational assessment. Los Angeles: UCLA's Center for the Study of Evaluation & National Center for Research on Evaluation, Standards, and Student Testing.
29.
Crowley, E.1987: The relationship between ability to summarize and field independence or dependence. Unpublished PhD thesis, Illinois State University.
30.
Cumming, A., Rebuffot, J. and Ledwell, M. 1989: Reading and summarizing challenging texts in first and second languages. Reading and Writing: An Interdisciplinary Journal2: 201—19.
31.
Davies, A.2003: The native speaker: Myth and reality. Clevedon, Bristol: Multilingual Matters.
32.
Deville, C. and Chalhoub-Deville, M.1993: Modified scoring, traditional item analysis and Sato's caution index used to investigate the reading recall protocol. Language Testing10: 117—32.
33.
Douglas, D.2001: Language for specific purposes assessment criteria: Where do they come from?Language Testing18: 171—85.
34.
Douglas, D. and Myers, R.2000: Assessing the communication skills of veterinary students: Whose criteria? In Kunnan, A., editor, Fairness and validation in language assessment: Selected papers from the 19th language testing research colloquium. Cambridge : University of Cambridge Press with University of Cambridge Local Examinations Syndicate, 60—81.
35.
Ets2002: LanguEdgeTM Courseware Handbook for Scoring Speaking and Writing. Princeton, NJ : Educational Testing Service.
36.
Garner, R.1982: Efficient text summarization: Costs and benefits. Journal of Educational Research75: 275—79.
37.
Grenz, S.J.1996: A primer on postmodernism. Grand Rapids, MI: William B. Eerdmans.
38.
Hare, V.C. and Borchardt, K.M.1984: Direct instruction of summarization skills. Reading Research Quarterly20: 62—78.
39.
Head, M.H., Readence, J.E. and Buss, R.R.1989: An examination of summary writing as a measure of reading comprehension. Reading Research & Instruction28: 1—11.
40.
Hill, K.1997: Who should be the judge? The use of non-native speakers as raters on a test of English as an international language. In Huhta, A., Kohonen, V., Kurki-Suonio, L. and Luoma, S., editors, Current developments and alternatives in language assessment. Tampere: Universities of Tampere and Jyväskylä.
41.
Huhta, A. and Randell, E.1996: Multiple-choice summary: A measure of text comprehension . In Cumming, A. and Berwick, R., editors, Validation in language testing. Clevedon, UK: Multilingual Matters, 94—110.
42.
Jacoby, S. and McNamara, T.F.1999: Locating competence. English for Specific Purposes18: 213—41.
43.
Johns, A.M.1985: Summary protocols of “underprepared” and “adept” university students: Replications and distortions of the original. Language Learning35: 495—517.
44.
Johns, A.M. and Mayes, P.1990: An analysis of summary protocols of university ESL students . Applied Linguistics11: 253—71.
45.
Kiewit, S.F.1997: The relationship of prior background knowledge to the summarization skills of developmental college students. Unpublished PhD thesis , University of Akron.
46.
Kintsch, W.1974: The representation of meaning in memory. Hillsdale, NJ: Lawrence Erlbaum.
47.
Kintsch, W. and Kozminsky, E.1977: Summarizing stories after reading and listening. Journal of Educational Psychology69: 491—99.
48.
Kintsch, W. and van Dijk, T.A.1978: Toward a model of text comprehension and production. Psychological Review85: 363—94.
49.
Kobayashi, M.2002: Method effects on reading comprehension test performance: Text organization and response format. Language Testing19: 193—220.
50.
Kuckartz, U.1998: winMAX: Scientific text analysis for the social sciences (Version 98 Pro). Berlin: VERBI Software-Consult-Research GmbH (Germany).
51.
Lado, R.1961: Language testing: The construction and use of foreign language tests. A teacher's book. London: Longman.
52.
——— 1986: Analysis of native speaker performance on a cloze test . Language Testing3: 130—46.
53.
McNamara, T.2001: Language assessment as social practice: Challenges for research. Language Testing18: 333—49.
54.
Mast, C.O.1988: The effects of cognitive styles on summarization of expository text. Unpublished PhD thesis, University of North Texas .
55.
Meyer, B.J.F.1975: The organization of prose and its effects on memory. Amsterdam: North-Holland.
56.
Mills, C.B., Diehl, V.A., Birkmire, D.P. and Mou, L.C.1993: Procedural text: Predictions of importance ratings and recall by models of reading comprehension. Discourse Processes16: 279—315.
57.
Moore, T.1997: From text to note: Cultural variation in summarization practices. Prospect12: 54—63.
58.
Nuttall, C.1996: Teaching reading skills in a foreign language. Oxford: Macmillan Heinemann.
59.
Penny, J., Johnson, R.L. and Gordon, B.2000: The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric. Assessing Writing7: 143—64.
60.
Pollitt, A. and Hutchinson, C.1987: Calibrating graded assessments: Rasch partial credit analysis of performance in writing. Language Testing4: 72—92.
61.
Riley, G.L. and Lee, J.F.1996: A comparison of recall and summary protocols as measures of second language reading comprehension. Language Testing13: 173—89.
62.
Rivard, L.P.2001: Summary writing: A multi-grade study of French-immersion and Francophone secondary students. Language, Culture and Curriculum14: 171—86.
63.
Rosenau, P.M.1992: Post-modernism and the social sciences: Insights, inroads and intrusions. Princeton, NJ: Princeton University Press.
64.
Sarig, G.1989: Testing meaning construction: Can we do it fairly?Language Testing6: 77—94.
65.
Savignon, S.2003: Teaching English as communication: A global perspective . World Englishes22: 55—66.
66.
Sawaki, Y.2003: A comparison of summarization and free recall as reading comprehension tasks in web-based assessment of Japanese as a foreign language . Unpublished PhD thesis, University of California, Los Angeles (UCLA), Los Angeles .
67.
Schnotz, W.1983: On the influence of text organization on learning outcomes . In Rickheit, G. and Bock, M., editors, Psycholinguistic studies in language processing. Berlin and New York: de Gruyter, 152—81.
68.
Scott, M.L., Stansfield, C.W. and Kenyon, D.M.1996: Examining validity in a performance test: The listening summary translation exam (LSTE) — Spanish version. Language Testing13: 83—109.
69.
Shi, L.2001: Native- and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing. Language Testing18: 303—25.
70.
Shohamy, E.1984: Does the testing method make a difference? The case of reading comprehension. Language Testing1: 147—70.
71.
Stansfield, C.W., Wu, W.M. and van der Heide, M.2000: A job-relevant listening summary translation exam in Minnan. In Kunnan, A.J., editor, Fairness and validation in language assessment: Selected papers from the 19th LTRC. Cambridge: Cambridge University Press with University of Cambridge Local Examinations Syndicate, 177—200.
72.
Stevens, J.2002: Applied multivariate statistics for the social sciences . Hillsdale, NJ: Lawrence Erlbaum .
73.
Taylor, L.B.1996: An investigation of text-removed summary completion as a means of assessing reading comprehension. Unpublished PhD thesis, University of Cambridge.
74.
The New Encyclopaedia Britannica: Macropaedia: Knowledge in Depth 1992: 15th edition,Vol. 18: pp. 87—88.
75.
Turner, C.E. and Upshur, J.A.2002: Rating scales derived from student samples: Effects of the scale maker and the student sample on scale content and student scores . TESOL Quarterly36: 49—70.
76.
Urquhart, A.H. and Weir, C.J.1998: Reading in a second language: Process, product and practice . London: Longman.
77.
van Dijk, T.A. and Kintsch , W.1977: Cognitive psychology and discourse: Recalling and summarizing stories. In Dressler, W.U., editor, Current trends in textlinguistics. New York: de Gruyter, 61—80.
78.
——— 1983: Strategies of discourse comprehension. New York: Academic Press.
79.
Wade, S.E., Buxton, W.M. and Kelly, M.1999: Using think-alouds to examine reader-text interest. Reading Research Quarterly34: 194—216.
80.
Weir, C.J.1993: Understanding and developing language tests. Hemel Hempstead: Prentice Hall International.
81.
Wenger, E.1998: Communities of practice: Learning, meaning and identity . Cambridge: Cambridge University Press.
82.
Winograd, P.N.1984: Strategic difficulties in summarizing texts. Reading Research Quarterly19: 404—25.
83.
Wolf, D.F., Bixby, J., Glenn, J.I. and Gardener, H.1991: To use their minds well: Investigating new forms of student assessment. Review of Research in Education17: 31—74.
84.
Yu, G.2005: Towards a model of using summarization tasks as a measure of reading comprehension. Unpublished PhD thesis, University of Bristol.
85.
Zuck, L.V. and Zuck, J.G.1984: The main idea: Specialist and non-specialist judgements . In Ulijn, J. and Pugh, A., editors, Reading for professional purposes: Studies and practices in native and foreign languages. London: Heinemann, 130—35.
