Sage Journals: Discover world-class research

Abstract

This study examines the decision-making behaviors of raters with varying levels of experience while assessing EFL essays of distinct qualities. The data were collected from 28 raters with varying levels of rating experience and working at the English language departments of different universities in Turkey. Using a 10-point analytic rubric, each rater voice-recorded their thoughts through think-aloud protocols (TAPs) while scoring 16 essays of distinct text qualities and provided brief score explanations. Data collected from TAPs were analyzed by using a coding scheme adapted from Cumming, Kantor, and Powers (2002). The results revealed that text quality has a larger effect than rating experience on raters’ decision-making behaviors. In addition, raters prioritized aspects of style, grammar, and mechanics when rating low-quality essays, but emphasized rhetoric and their general impressions of the text for high-quality essays. Furthermore, low-experienced raters differed more in their behaviors while assessing scripts of distinct qualities than did the medium- and high-experienced groups. The findings suggest that raters’ scoring behaviors might evolve with practice, resulting in less variation in their decisions. As such, this research provides implications for developing strategy-based rater training programs, which might help to increase consistency across raters of different experience levels.

Keywords

EFL writing assessment raters’ decision-making behaviors rating experience text quality think-aloud protocols

Get full access to this article

View all access options for this article.

References

Bachman

L. F.

(1990). Fundamental considerations in language testing. Oxford University Press.

Baker

A. B.

(2012). Individual differences in rater decision-making style: An exploratory mixed-methods study. Language Assessment Quarterly, 9(3), 225–248. https://doi.org/10.1080/15434303.2011.637262

Barkaoui

(2007). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12(2), 86–107. https://doi.org/10.1016/j.asw.2007.07.001

Barkaoui

(2010a). Do ESL essays raters’ evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44(1), 31–57. https://doi.org/10.5054/tq.2010.214047

Barkaoui

(2010b). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54–74. https://doi.org/10.1080/15434300903464418

Cumming

(1990). Expertise in evaluating second language composition. Language Testing, 7(1), 31–51. https://doi.org/10.1177/026553229000700104

Cumming

Kantor

Powers

(2001). Scoring TOEFL essays and TOEFL 2000 prototype tasks: An investigation into raters’ decision making and development of a preliminary analytic framework (TOEFL Monograph Series, Report No: 22). Princeton, NJ: Educational Testing Service. https://www.ets.org/Media/Research/pdf/RM-01-04.pdf

Cumming

Kantor

Powers

(2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86(1), 67–96. https://doi.org/10.1111/1540–4781.00137

Daly

J. A.

Dickson-Markman

(1982). Contrast effects in evaluating essays. Journal of Educational Measurement, 19(4), 309–316. https://doi.org/10.1111/j.1745–3984.1982.tb00136.x

10.

DeRemer

M. L.

(1998). Writing assessment: Raters’ elaboration of the rating task. Assessing Writing, 5(1), 7–29. https://doi.org/10.1016/S1075–2935(99)80003–8

11.

Eckes

(2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185. https://doi.org/10.1177/0265532207086780

12.

Engber

C. A.

(1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4(2), 139–155. https://doi.org/10.1016/1060–3743(95)90004–7

13.

Ferris

D. R.

(1994). Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly, 28(2), 414–420. https://doi.org/10.2307/3587446

14.

Freedman

S. W.

(1981). Influences on evaluators of expository essays: Beyond the text. Research in the Teaching of English, 15(3), 245–255. https://www.jstor.org/stable/40170792

15.

Gebril

Plakans

(2014). Assembling validity evidence for assessing academic writing: Rater reactions to integrated tasks. Assessing Writing, 21(2), 56–73. https://doi.org/10.1016/j.asw.2014.03.002

16.

Hamp-Lyons

(1990). Second language writing: Assessment issues. In Kroll

(Ed.), Second language writing: Research insights for the classroom (pp. 69–87). Cambridge University Press.

17.

Hamp-Lyons

(1991). Basic concepts. In Hamp-Lyons

(Ed.), Assessing second language writing in academic contexts (pp. 5–15). Ablex.

18.

Han

(2017). Scores assigned by inexpert raters to different quality of EFL compositions, and the raters’ decision-making behaviors. International Journal of Progressive Education, 13(1), 136–152. http://ijpe.penpublishing.net/makale/229

19.

Henning

(1991). Issues in evaluating and maintaining an ESL writing assessment program. In Hamp-Lyons

(Ed.), Assessing second language writing in academic contexts (pp. 279–292). Ablex.

20.

Huang

(2008). How accurate are ESL students’ holistic writing scores on large-scale assessments? —A generalizability theory approach. Assessing Writing, 13(3), 201–218. https://doi.org/10.1016/j.asw.2008.10.002

21.

Huang

(2009). Factors affecting the assessment of ESL students’ writing. International Journal of Applied Educational Studies, 5(1), 1–17.

22.

Huang

Foote

C. J.

(2010). Grading between lines: What really impacts professors’ holistic evaluation of ESL graduate student writing? Language Assessment Quarterly, 7(3), 219–233. https://doi.org/10.1080/15434300903540894

23.

Huang

Han

Tavano

Hairston

(2014). Using generalizability theory to examine the impact of essay quality on rating variability and reliability of ESOL writing. In Huang

Han

(Eds.), Empirical quantitative research in social sciences: Examining significant differences and relationships (pp. 127–149). Untested Ideas Research Center.

24.

Hughes

D. E.

Keeling

(1984). The use of models to reduce context effects in essay scoring. Journal of Educational Measurement, 21(3), 277–281. https://doi.org/10.1111/j.1745–3984.1984.tb01034.x

25.

Hyland

(2003). Second language writing. New York: Cambridge University Press.

26.

Kenyon

(1992, February). Introductory remarks at symposium on development and use of rating scales in language testing. Paper presented at the 14th Language Testing Research Colloquium, Vancouver, British Columbia.

27.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174. https://doi.org/10.2307/2529310

28.

Lim

G. S.

(2011). The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters. Language Testing, 28(4), 543–560. https://doi.org/10.1177/0265532211406422

29.

Lumley

(2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–276. https://doi.org/10.1191/0265532202lt230oa

30.

McNamara

T. F.

(2000). Language testing. Oxford University Press.

31.

Milanovic

Seville

Shuhong

(1996). A study of the decision-making behavior of composition markers. In Milanovic

Seville

(Eds.), Performance testing, cognition, and assessment: Selected papers from the 15th Language Testing Colloquium (LTRC), Cambridge and Arnhem (pp. 92–114). Cambridge University Press.

32.

Raimes

(1990). The TOEFL test of written English: Causes for concern. TESOL Quarterly, 24(3), 427–442. https://doi.org/10.2307/3587228

33.

Rinnert

Kobayashi

(2001). Differing perceptions of EFL writing among readers in Japan. The Modern Language Journal, 85(2), 189–209. https://doi.org/10.1111/0026–7902.00104

34.

Şahan

. (2016, June 23). A sample think-aloud protocol [Video file]. https://youtu.be/hoJxNZFdT4Q

35.

Şahan

. (2019). The impact of rater experience and essay quality on the variability of EFL writing scores. In Papageorgiou

Bailey

K. M.

(Eds.), Global perspectives on language assessment: Research, theory, and practice (pp. 32–46). Routledge.

36.

Shi

(2001). Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing. Language Testing, 18(3), 303–325. https://doi.org/10.1177/026553220101800303

37.

Vaughan

(1991). Holistic assessment: What goes on in the raters’ minds? In Hamp-Lyons

(Ed.), Assessing second language writing in academic contexts (pp. 11–126). Ablex.

38.

Weigle

S. C.

(2002). Assessing writing. Cambridge University Press.

39.

Wolfe

E. F.

Feltovich

(1994, April). Learning to rate essays: A study of scorer cognition. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA. https://files.eric.ed.gov/fulltext/ED368777.pdf

40.

Wolfe

E. W.

Kao

Ranney

(1998). Cognitive differences in proficient and non-proficient essay scorers. Written Communication, 15(4), 465–492. https://doi.org/10.1177/0741088398015004002

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB

0.14 MB

0.46 MB

Do experience and text quality matter for raters’ decision-making behaviors?

Abstract

Keywords

Get full access to this article

References

Supplementary Material