Do source use features impact raters’ judgment of argumentation? An experimental study

Abstract

This experimental study explores how source use features impact raters’ judgment of argumentation in a second language (L2) integrated writing test. One hundred four experienced and novice raters were recruited to complete a rating task that simulated the scoring assignment of a local English Placement Test (EPT). Sixty written responses were adapted from essays written by EPT test-takers. These responses were crafted to reflect different conditions of source use features, namely source use quantity and quality. Rater scores were analyzed using the many-facet Rasch model and mixed two-way analyses of variance (ANOVAs) to examine how they are affected by source use features and rater experience. Results show that source use features impacted the argumentation scores assigned by raters. Paragraphs with more source text ideas that are better incorporated received the highest argumentation scores, and vice versa for those with limited, poorly integrated source information. Rater experience impacted scores but did not influence rater performance meaningfully. The findings of this study connect specific source use features with raters’ evaluation of argumentation, helping to further disentangle the relationships among examinee performance, rater decision, and task features of integrated argumentative writing tests. They also provide meaningful implications for writing assessment research and practices.

Keywords

Argumentation experiment integrated writing rater behavior source use

Get full access to this article

View all access options for this article.

References

Arunachalam

(2013). Experimental methods for linguists. Language and Linguistics Compass, 7(4), 221–232. https://doi.org/10.1111/lnc3.12021

Attali

(2016). A comparison of newly-trained and experienced raters on a standardized writing assessment. Language Testing, 33(1), 99–115. https://doi.org/10.1177/0265532215582283

Barkaoui

(2010). Do ESL essay raters’ evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44(1), 31–57. https://doi.org/10.5054/tq.2010.214047

Bosker

H. R.

Pinget

A.-F.

Quené

Sanders

de Jong

N. H.

(2013). What makes speech sound fluent? The contributions of pauses, speed and repairs. Language Testing, 30(2), 159–175. https://doi.org/10.1177/0265532212455394

Brown

J. D.

(1991). Do English and ESL faculties rate writing samples differently? TESOL Quarterly, 25(4), 587–603. https://doi.org/10.2307/3587078

Chuang

P.-L.

Yan

(2022). An investigation of the relationship between argument structure and essay quality in assessed writing. Journal of Second Language Writing, 56, 100892. https://doi.org/10.1016/j.jslw.2022.100892

Chuang

P.-L.

Yan

(2023). Connecting source use and argumentation in L2 integrated argumentative writing performance. Journal of Second Language Writing, 60, 101003. https://doi.org/10.1016/j.jslw.2023.101003

Cumming

(1990). Expertise in evaluating second language compositions. Language Testing, 7(1), 31–51. https://doi.org/10.1177/026553229000700104

Cumming

Kantor

Baba

Erdosy

Eouanzoui

James

(2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, 10(1), 5–43. https://doi.org/10.1016/j.asw.2005.02.001

10.

Cumming

Kantor

Powers

D. E.

(2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. The Modern Language Journal, 86(1), 67–96. https://doi.org/10.1111/1540-4781.00137

11.

Currie

(1998). Staying out of trouble: Apparent plagiarism and academic survival. Journal of Second Language Writing, 7(1), 1–18. https://doi.org/10.1016/S1060-3743(98)90003-0

12.

De La Paz

Ferretti

Wissinger

Yee

MacArthur

(2012). Adolescents’ disciplinary use of evidence, argumentative strategies, and organizational structure in writing about historical controversies. Written Communication, 29(4), 412–454. https://doi.org/10.1177/0741088312461591

13.

Gebril

Plakans

(2009). Investigating source use, discourse features, and process in integrated writing tests. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 7(1), 47–84.

14.

Gebril

Plakans

(2014). Assembling validity evidence for assessing academic writing: Rater reactions to integrated tasks. Assessing Writing, 21, 56–73. https://doi.org/10.1016/j.asw.2014.03.002

15.

Jacobs

H. L.

Zingraf

S. A.

Wormuth

D. R.

Hartfiel

V. F.

Hughey

J. B.

(1981). Testing ESL composition: A practical approach. Newbury House.

16.

Kelly

G. J.

Takao

(2002). Epistemic levels in argument: An analysis of university oceanography students’ use of evidence in writing. Science Education, 86(3), 314–342. https://doi.org/10.1002/sce.10024

17.

Leckie

Baird

J.-A.

(2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48(4), 399–418. https://doi.org/10.1111/j.1745-3984.2011.00152.x

18.

Linacre

J. M.

(2014). Facets computer program for many-facet Rasch measurement (Version 3.71.4). https://www.winsteps.com/facets.htm

19.

Linacre

J. M.

(2023a, November 28). Fit diagnosis: Infit outfit mean-square standardized. https://www.winsteps.com/winman/misfitdiagnosis.htm

20.

Linacre

J. M.

(2023b, November 28). Reliability—separation—strata. https://www.winsteps.com/facetman/reliability.htm

21.

Myford

C. M.

Wolfe

E. W.

(2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422.

22.

Polio

Shi

, (Eds.). (2012). Textual appropriation and source use in L2 writing [Special issue]. Journal of Second Language Writing, 21(2), 95–101.

23.

Qin

Karabacak

(2010). The analysis of Toulmin elements in Chinese EFL university. System, 38(3), 444–456. https://doi.org/10.1016/j.system.2010.06.012

24.

Schmid

M. S.

Hopp

(2014). Comparing foreign accent in L1 attrition and L2 acquisition: Range and rater effects. Language Testing, 31(3), 367–388. https://doi.org/10.1177/0265532214526175

25.

Seo

S.-Y.

(2024). Development of clausal and phrasal complexity in L2 writing: A case of argumentative essays of Korean college students. English for Specific Purposes, 73, 46–60. https://doi.org/10.1016/j.esp.2023.09.003

26.

Shi

(2004). Textual borrowing in second-language writing. Written Communication, 21(2), 171–200. https://doi.org/10.1177/0741088303262846

27.

Song

Caruso

(1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing, 5(2), 163–182. https://doi.org/10.1016/S1060-3743(96)90023-5

28.

Takao

A. Y.

Kelly

G. J.

(2003). Assessment of evidence in university students’ scientific writing. Science & Education, 12(4), 341–363. https://doi.org/10.1023/A:1024450509847

29.

Toulmin

S. E.

(2003). The uses of argument (2nd ed.). Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511840005

30.

Uludag

Lindberg

McDonough

Payant

(2019). Exploring L2 writers’ source-text use in an integrated writing assessment. Journal of Second Language Writing, 46, 100670. https://doi.org/10.1016/j.jslw.2019.100670

31.

Wang

Engelhard

Raczynski

Song

Wolfe

E. W.

(2017). Evaluating rater accuracy and perception for integrated writing assessments using a mixed-methods approach. Assessing Writing, 33, 36–47. https://doi.org/10.1016/j.asw.2017.03.003

32.

Weigle

S. C.

Montee

(2012). Raters’ perceptions of textual borrowing in integrated writing tasks. In Tillema

Van Steendam

Rijlaarsdam

van den Bergh

(Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 117–151). Emerald Books.

33.

Weigle

S. C.

Parker

(2012). Source text borrowing in an integrated reading/writing assessment. Journal of Second Language Writing, 21(2), 118–133. https://doi.org/10.1016/j.jslw.2012.03.004

34.

Winke

Lim

. (2015). ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing, 25, 38–54. https://doi.org/10.1016/j.asw.2015.05.002

35.

Wright

B. D.

Linacre

J. M.

(1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370. https://www.rasch.org/rmt/rmt83b.htm

36.

Xie

(2023). Assessing source use: Summary vs. reading-to-write argumentative essay. Assessing Writing, 57, 100755. https://doi.org/10.1016/j.asw.2023.100755

37.

Yan

Kim

H. R.

Kotnarowski

(2021). Development of a profile-based writing scale: How collaboration with teachers enhanced assessment practice in a post-admission ESL writing program at a USA University. In Lanteigne

Coombe

D. Brown

(Eds.), Challenges in language testing around the world (pp. 529–546). Springer. https://doi.org/10.1007/978-981-33-4232-3_36

38.

Yoon

H.-J.

(2017). Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality. System, 66, 130–141. https://doi.org/10.1016/j.system.2017.03.007

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB