Creating a self-rating instrument for second language writing: from idea to implementation

Abstract

This article reports on the development of a self-rating instrument for writing. The instrument engages learners in responding to a writing task and assessing their own proficiency against a set of benchmarks. The technical and conceptual framework for the instrument is the DIALANG project, a diagnostic language assessment system for 14 languages on the internet. The report comprises a description of the self-rating procedure, an account of instrument development, a report on a usability study with six learners of Finnish as a second language, and our test-developer analysis of the process and products of the self-rating instrument. The data consists of records of task and benchmark development, participant background data, the texts that the learners wrote, video recordings of their screen actions during the self-rating process, immediate retrospective interviews with the learners, and teacher ratings of the learner texts. The discussion concentrates on the concepts that underlie the self-rating process and the potential offered by this approach for the learning, teaching, and assessment of second language writing.

Get full access to this article

View all access options for this article.

References

Alderson, J. C. , Clapham, C. and Wall, D. 1995: Language test construction and evaluation. Cambridge: Cambridge University Press .

Bachman, L. and Palmer, A. S. 1989: The construct validation of self-ratings of communicative language ability . Language Testing 6, 14-25 .

Bachman, L. and Palmer, A. S. 1996: Language testing in practice. Oxford: Oxford University Press .

Benson, P. and Voller, P. 1997: Introduction: autonomy and independence in language learning. In Benson, P. and Voller, P. , editors, Autonomy and independence in language learning. London: Longman , 1-12.

Benson, P. 2001: Teaching and researching autonomy in language learning. Harlow: Longman .

Bruner, J. 1986: The culture of education. Cambridge, MA: Harvard University Press .

Clark, J. L. D. 1981: Language. In Burroughs, T. S. et al. , editors, College students’ knowledge and beliefs: a survey of global understanding. New Rochelle, NY: Change Magazine Press , 25-35.

Council of Europe 2001: Common European framework of reference: learning, teaching, assessment. Cambridge: Cambridge University Press .

Dandenault, E. 1997: Self-assessment of communicative ability: investigation of a novel tool for ESL learners. Unpublished MA thesis. Concordia University, Montreal, Canada.

10.

Fletcher, C. and Baldry, C. 2000: A study of individual differences and self-awareness in the context of multi-source feedback . Journal of Occupational and Organizational Psychology 73, 303-319 .

11.

LeBlanc, R. and Painchaud, G. 1985: Self-assessment as a second language placement instrument . TESOL Quarterly 19, 673-687 .

12.

Little, D. 1991: Learner autonomy. Dublin: Authentik .

13.

Milanovic, M. , Saville N. and Shuhong, S. 1996: A study of the decision-making behaviour of composition markers . In Milanovic, M. and Saville, N. , editors, Performance testing, cognition and assessment: selected papers from the 15th Language Testing Research Colloquium, Cambridge and Arnhem. Cambridge: Cambridge University Press, 92-114 .

14.

Moss, P. 1994: Can there be validity without reliability? Educational Researcher 31, 5-12 .

15.

Nielsen, J. 1994: Estimating the number of subjects for thinking aloud test . International Journal of Human-Computer Studies 41, 385-397 .

16.

Oscarson, M. 1978: Approaches to self-assessment in foreign language learning. Strasbourg: Council of Europe, Council for Cultural Cooperation .

17.

Oscarson, M. 1999: Review of R&D work in the area of self-assessment of foreign language ability. The DIALANG Project, project-internal working paper. Unpublished manuscript, Department of Education, University of Göteborg .

18.

Rolfe, T. 1990: Self- and peer assessment in the ESL curriculum. In Brindley, G. , editor, The second language curriculum in action. Sydney: National Centre for English Language Teaching and Research, Macquarie University , 163-186.

19.

Segalowitz, N. 1976: Communicative incompetence and the nonfluent bilingual . Canadian Journal of Behavioural Science 8, 122-131 .

20.

Shepard, L. 2000: The role of assessment in a learning culture . Educational Researcher 29, 4-14 .

21.

Skehan, P. 1998: Processing perspectives to second language development, instruction, performance, and assessment . Working Papers in Applied Linguistics 4, 70-88 . London: Thames Valley University .

22.

Smith, D. 2000: Rater judgement in the direct assessment of competency-based second language writing ability. In Brindley, G. , editor, Studies in immigrant English language assessment. Volume 1. Research series 11. Sydney: National Centre for English Language Teaching and Research, Macquarie University , 159-189.

23.

Tarnanen, M. 2002: Arvioija valokiilassa: suomi toisena kielenä - kirjoittamisen arviointia [The rater in a spotlight: Rating Finnish-asa-second-language writing]. PhD thesis, University of Jyväskylä, Finland.

24.

Teasdale, A. and Leung, C. 2000: Teacher assessment and psychometric theory: a case of paradigm crossing? Language Testing 17, 163-184 .

25.

Vaughan, C. 1991: Holistic assessment: what goes on in the rater’s mind? In Hamp-Lyons, L. , editor, Assessing second language writing in academic contexts. Norwood, NJ: Ablex , 111-126.

26.

Virzi, R. A. 1992: Refining the test phase of usability evaluation: how many subjects is enough? Human Factors 34, 457-486 .