The role of test-taker feedback in the test development process: test-takers' reactions to a tape-mediated test of proficiency in spoken Japanese

Abstract

This study explores how test-taker reactions in a specific purpose testing context may vary according to characteristics of the test-taker. Such reactions are of interest both theoretically to the researcher and practically to the test developer, who is concerned to ensure that the test is fair and appropriate for all candidates and acceptable to the range of test-takers. The reactions may also be of use in the improvement of test items (as a supplement to item analysis) and test rubrics.

The article reports on the use of test-taker feedback in the development of the occupational foreign language test (Japanese), a tape-mediated test of spoken Japanese for the tourism and hospitality industry. Some 53 trial subjects completed a post-test questionnaire, providing reactions to the test as a whole, to task types, and to individual test items. Relationships are investigated between responses and a number of test-taker characteristics, including gender, type of course undertaken (general or specific purpose), amount of study of the language, time spent in Japan and relevant occupa tional experience. Responses are also considered for items and person shown by Rasch IRT analysis as indicating significant misfit. Aspects of the content, construct and face validity of the test are considered in the light of the analysis. The role of feedback in the revision of test items, in the writing of test rubrics and in the development of the test-user handbook is also discussed.

Get full access to this article

View all access options for this article.

References

Adams, R. and Khoo, S.T. 1990: TITAN: The Interactive Test Analysis System. Hawthorn: ACER.

Alderson, J.C. 1988: New procedures for validating proficiency tests of ESP? Theory and practice. Language Testing 5, 220-32.

ARELS Examinations Trust 1989: The Oxford-ARELS examinations in English as a foreign language: rationales, regulations and syllabuses. Oxford: Author.

Bradshaw, J. 1990: Test-takers' reactions to a placement test. Language Testing 7, 13-30.

Brown, A. 1991a: A Japanese test for tourism and hospitality. Paper presented at the Australia-Japan Foundation Conference for Teachers of Japanese, August 1991.

— 1991b: The development of the Occupational Foreign Language Test. Report to the Victorian Education Foundation. Melbourne : Language Testing Centre, University of Melbourne .

Clark, J.L.D. 1985: Development of a tape-mediated, ACTFL/ILR scale-based test of Chinese speaking proficiency. In Stansfield, C.W. , editor, Technology and language testing . Princeton, NJ: Education Testing Service.

— 1986: Handbook for the development of tape-mediated, ACTFL/ILR scale-based tests of speaking proficiency in the less commonly taught languages . Washington, DC: Center for Applied Linguistics.

Clark, J.L.D. and Swinton, S. 1979: An exploration of speaking proficiency measures in the TOEFL context. TOEFL Research Report 4 Princeton, NJ: Educational Testing Service.

10.

Davies, A. and Brown, A. 1990: Designing instruments to measure language proficiency . In Wylie, E. , editors, Assessment of proficiency in Japanese as a foreign language. Canberra: Asian Studies Council.

11.

Ingram, D.E. 1984: Australian Second Language Proficiency Ratings. Canberra: Australian Government Publishing Service .

12.

James, G. 1988: Development of an oral proficiency component in a test in English for academic purposes. In Hughes, A. , editor, Testing English for university study. ELT Documents 127. Oxford: Modern English Publications and the British Council.

13.

Kenyon, D.M. and Stansfield, C. 1991: A method for improving tasks on performance assessments through field testing. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL, April 1991.

14.

Lowe, P. jr and Clifford, R.T. 1980: Developing an indirect measure of overall oral proficiency . In Frith, J.R. , editor, Measuring spoken language proficiency. Washington, DC: Georgetown University Press.

15.

McNamara, T.F. 1990: Assessing the second language proficiency of health professionals. Unpublished Ph.D. thesis, University of Melbourne.

16.

Savignon, S.J. 1972: Communicative competence: an experiment in foreign language teaching. Philadelphia, PA: The Center for Curriculum Development.

17.

Scott, M.L. 1986: Student affective reactions to oral language tests. Language Testing 3, 99-118.

18.

Shohamy, E. 1982: Affective considerations in language testing. Modern Language Journal 66, 13-17.

19.

Stansfield, C.W. 1991: A comparative analysis of simulated and direct oral proficiency interviews. In Anivan, S. , editor, Current developments in language testing. Singapore: SEAMEO-RELC.

20.

Stansfield, C.W. , Kenyon, D.M. , Paiva, R. , Doyle, F. , Ulsh, I. and Cowles, M.A. 1990: The development and validation of the Portuguese speaking test. Hispania 73, 641-51.

21.

Wright, B.D. and Stone, M.H. 1979: Best test design. Chicago: MESA Press.

22.

Zeidner, M. 1987: Essay versus multiple-choice type classroom exam: the students' perspective. Journal of Educational Research 80, 352-58.

23.

— 1988a: Classroom testing: the examinees' perspective. Studies in Educational Evaluation 14, 215-33.

24.

— 1988b: Sociocultural differences in examinees' attitudes toward scholastic ability exams. Journal of Educational Measurement 25, 67-75.

25.

— 1990: College students' reactions towards key facets of classroom testing. Assessment and Evaluation in Higher Education 15, 151-69.

26.

Zeidner, M. and Bensoussan, M. 1988: College students' attitudes towards written versus oral tests of English as a foreign language. Language Testing 5, 100-114.