Assessing learners’ writing skills in a SLA study: Validating the rating process across tasks,scales and languages

Abstract

There is still relatively little research on how well the CEFR and similar holistic scales work when they are used to rate L2 texts. Using both multifaceted Rasch analyses and qualitative data from rater comments and interviews, the ratings obtained by using a CEFR-based writing scale and the Finnish National Core Curriculum scale for L2 writing were examined to validate the rating process used in the study of the linguistic basis of the CEFR in L2 Finnish and English. More specifically, we explored the quality of the ratings and the rating scales across different tasks and across the two languages. As the task is an integral part of the data-gathering procedure, the relationship of task peformance across the scales and languages was also examined. We believe the kinds of analyses reported here are also relevant to other SLA studies that use rating scales in their data-gathering process.

Keywords

CEFR scales L2 learning L2 writing rating process tasks validation

Get full access to this article

View all access options for this article.

References

Alanen

Huhta

Tarnanen

(2010). Designing and assessing L2 writing tasks across CEFR proficiency levels. In Bartning

Martin

Vedder

(Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 21–56). EUROSLA Monograph Series, 1. Retrieved from http://eurosla.org/monographs/EM01/EM01home.html

Alanen

Huhta

Jarvis

Martin

Tarnanen

(2012). Issues and challenges in combining SLA research and language testing. In Tsagari

Csepes

(Eds.), Collaboration in language testing and assessment (pp. 15–30). Frankfurt am Main: Peter Lang.

Alderson

J. C

. (1991). Bands and scores. In Alderson

J. C.

North

(Eds.) Language testing in the 1990s: The communicative legacy (pp. 71–86). London: Macmillan.

Alderson

J. C

. (2007). The CEFR and the need for more research. Modern Language Journal, 91, 659–663.

Bachman

Cohen

(1998). Interfaces between second language learning and language testing research. Cambridge: Cambridge University Press.

Barkaoui

(2010). Explaining ESL essay holistic scores: A multilevel modeling approach. Language Testing 27, 515–535.

Bartning

Martin

Vedder

(Eds.) (2010). Communicative proficiency and linguistic development: Intersections between SLA and language testing research. EUROSLA Monograph series 1. Retrieved from http://eurosla.org/monographs/EM01/EM01home.html

Carlsen

(2010). Discourse connectives across CEFR-levels: A corpus based study. In Bartning

Martin

Vedder

(Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 191–209). EUROSLA Monograph Series, 1. Retrieved from http://eurosla.org/monographs/EM01/EM01home.html

Chalhoub-Deville

(2009). Content validity considerations in language testing contexts. In Lissitz

R. W.

(Ed.), The concept of validity: Revisions, new directions and applications. Charlotte, NC: Information Age Publishing.

10.

Chen

Y.-H.

(2009). Investigating lexical bundles across learner writing development. (Doctoral dissertation). Lancaster University, UK.

11.

Eckes

(2012). Examinee-centered standard setting for large-scale assessments: The prototype group method. Psychological Test and Assessment Modeling, 54(3), 257–283.

12.

Forsberg

Bartning

(2010). Can linguistic features discriminate between the communicative CEFR-levels? A pilot study of written L2 French. In Bartning

Martin

Vedder

(Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 133–157). EUROSLA Monograph Series, 1. Retrieved from http://eurosla.org/monographs/EM01/EM01home.html

13.

Galaczi

(2013). Interactional competence across proficiency levels: How do learners manage interaction in paired speaking tests? Applied Linguistics, 1–23. doi:10.1093/applin/amt017

14.

Harsch

Martin

(2012). Adapting CEF-descriptors for rating purposes: Validation by a combined rater training and scale revision approach. Assessing Writing, 17, 228–250.

15.

Hildén

Takala

(2007). Relating descriptors of the Finnish school scale to the CEF overall scales for communicative activities. In Koskensalo

Smeds

Kaikkonen

Kohonen

(Eds.), Foreign languages and multicultural perspectives in the European context (pp. 291–300). Reihe: Dichtung – Wahrheit – Sprache Bd. 7. Münster: LIT Verlag.

16.

Huhta

(2012). Common European Framework of Reference. In Chapelle

(Ed.), The encyclopedia of applied linguistics. Oxford: Wiley-Blackwell.

17.

Hulstijn

J. H

. (2007). The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency. The Modern Language Journal, 91, 663–667.

18.

Hulstijn

J. H

. (2010). Linking L2 proficiency to L2 acquisition: Opportunities and challenges of profiling research. In Bartning

Martin

Vedder

(Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 233–238). EUROSLA Monograph series 1. Retrieved from http://eurosla.org/monographs/EM01/EM01home.html

19.

Kaftandjieva

Takala

(2002). Council of Europe scales of language proficiency: A validation study. In Alderson

J. C.

(Ed.), Common European framework of reference for languages: Learning, teaching, assessment. Case studies. (pp. 106–129) Strasbourg: Council of Europe.

20.

Knoch

(2010). Investigating the effectiveness of individualized feedback to rating behavior: A longitudinal study. Language Testing, 28, 179–200.

21.

Kuiken

Vedder

Gilabert

(2010). Communicative adequacy and linguistic complexity in L2 writing. In Bartning

Martin

Vedder

(Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 81–99). EUROSLA Monograph series 1. Retrieved from http://eurosla.org/monographs/EM01/EM01home.html

22.

Linacre

(2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement 3, 85–106.

23.

Linacre

(2009). A user’s guide to FACETS v 3.66.0. Chicago, IL: Winsteps.

24.

Little

(2007). The Common European framework of reference for languages: Perspectives on the making of supranational language education policy. Modern Language Journal, 91, 645–655.

25.

Lunz

Wright

Linacre

(1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education 3, 331–345.

26.

Martin

. (2013). The complex simple –a problematic adjective in the CEFR writing scales. Nordand –Nordisk tidsskrift for andrespråksforskning [Nordic Journal of Second Language Research], 8(2), 63–85.

27.

McNamara

(1996). Measuring second language performance. Boston, MA: Addison-Wesley Longman.

28.

North

(2000). The development of a common scale of language proficiency. New York: Peter Lang.

29.

North

(2007). The CEFR illustrative descriptor scales. Modern Language Journal, 91, 656–659.

30.

Pallotti

(2009). CAF: Defining, refining and differentiating constructs. Applied Linguistics, 30, 590–601.

31.

Tarnanen

Huhta

(2008). Interaction of language policy and assessment in Finland. Current Issues in Language Planning, 9(3), 262–281.

32.

Toropainen

Härmälä

Lahtinen

. (2012). Kaksi asteikkoa, kaksi eri tilannetta: Äidinkielellä ja vieraalla kielellä kirjoitettujen tekstien kriteeripohjaisen arvioinnin haasteita [Two scales, two different situations: Challenges in criterion-related assessment of texts written in the first and a foreign language]. AFinLA-e: Soveltavan kielitieteen tutkimuksia 4, 60–79.

33.

Tremblay

Garrison

M. D.

(2010). Cloze tests: A tool for proficiency assessment in research on L2 French. In Prior

M. T.

Watanabe

Lee

S.-K.

(Eds.), Selected proceedings of the 2008 Second Language Research Forum. Exploring SLA perspectives, positions, and practices (pp. 73–88). Somerville, MA: Cascadilla Proceedings Project.