The effect of rater variables in the development of an occupation-specific language performance test

Abstract

There has been little research into the effect of raters' background on assessments made in language tests. This article explores the effect that the rater's occupational and linguistic background has on assessments made in an occupation-specific oral language test, the Japanese Language Test for Tour Guides. Assessments of 51 test candidates made by 33 such assessors, including native and near-native speakers of Japanese with backgrounds either in teaching Japanese as a foreign language or in tour guiding in Japanese, were compared in order to determine what effect background has on assessments made on both linguistic and 'real-world' criteria. Multifaceted Rasch analysis was used, as this allows a range of facets to be modelled; for the purposes of this research we were interested in the facet 'raters', and in particular how the scale was applied for each of the assessment categories. These were then compared across the different types of rater. While differences were found in the assessments according to rater type, these were minor and did not point to the unsuitability of any group as a whole; there were no significant differences between the different types of rater in terms of the overall grade awarded. However, there were significant differences in ratings awarded for some individual criteria, and the fact that these reveal different perceptions of what constitutes good performances has obvious implications not only for the selection and training of raters for occupation- specific language performance tests but also for the development of assess ment procedures.

Get full access to this article

View all access options for this article.

References

Barnwell, D. 1989: 'Naive' native speakers and judgements of oral proficiency in Spanish Language Testing 6, 152-63.

Brown, A. 1993: LSP testing: the role of linguistic and 'real world' criteria Paper presented at the annual RELC seminar, Singapore, April.

Elder, C. 1993: Assessing the language proficiency of second language teachers: an LSP approach to test design Paper presented at the annual RELC seminar, Singapore, April.

Engelhard, G. jr 1991: The measurement of writing ability with a many-faceted Rasch model Applied Measurement in Education 5, 171-91.

Fayer, J.M. and Krasinski , E. 1987: Native and nonnative judgements of intelligibility and irritation Language Learning 37, 313-26.

Galloway, V. 1977: Analysis of evaluations of the oral communicative competence in Spanish of University of South Carolina students Dissertation Abstracts International 38, 7255A (University Microfilm 78-07, 900).

— 1980: Perceptions of the communication efforts of American students of Spanish Modern Language Journal 64, 428-33.

Hadden, B.L. 1991: Teacher and nonteacher perceptions of second-language communication Language Learning 41, 1-24.

Linacre, J.M. 1989: Many-facet Rasch measurement Chicago, IL: Mesa Press.

10.

Lunz, M.E. and Stahl , J.A. 1990: Judge consistency and severity across grading periods Evaluation and the Health Professionals 13, 425-44.

11.

McNamara, T.F. 1990: Assessing the second language proficiency of health professionals. PhD thesis, University of Melbourne.

12.

McNamara, T.F. and Adams , RJ. 1991: Exploring rater behaviour with Rasch techniques ERIC Document Reproduction Service ED 345 498.

13.

Myford, CJ. 1991: Assessment of acting ability Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL, April.

14.

Sheorey, R . 1985: Goof gravity in ESL: native vs. nonnative perceptions Paper presented at the 19th annual TESOL convention, New York.

15.

Shohamy, E. , Gordon, C.M. and Kraemer, R. 1992: The effect of raters' background and training on the reliability of direct writing tests The Modern Language Journal 76, 27-33.

16.

Wesche, M.B. 1992: Performance testing for work-related second language assessment In Shohamy, E. and Walton, R., editors, Language assessment for feedback: testing and other strategies. Washington, DC: National Foreign Language Center Publications.