Examinee abilities and task difficulty in task-based second language performance assessment

Abstract

This article summarizes findings from investigations into the development and use of a prototype English language task-based performance test. Data included performances by 90 examinees on 13 complex and skills-integrativetasks, a priori estimations of examinee proficiency differences, a priori estimations of task difficulty based on cognitive processing demands, performance ratings according to task-specific as well as holistic scales and criteria, and examinee self-ratings. Findings indicated that the task-based test could inform intended inferences about examinees’ abilities to accomplish specific tasks as well as inferences about examinees’ likely abilities with a domain of tasks. Although a relationship between task difficulty estimates and examinee performances was observed, these estimates were not found to provide a trustworthy basis for inferring examinees’ likely abilities with other tasks. These findings, as well as study limitations, are further discussed in light of the intended uses for performance assessment within language education, and recommendations are made for needed research into the interaction between task features, cognitive processing and language performance.

Get full access to this article

View all access options for this article.

References

Brindley, G. 1994: Task-centred assessment in language learning: the promise and the challenge . In Bird, N. , Falvey, P. , Tsui, A. , Allison, D. and McNeill, A. , editors, Language and learning: papers presented at the Annual International Language in Education Conference (Hong Kong, 1993). Hong Kong: Hong Kong Education Department, 73-94 .

Brown, J. D. and Hudson, T. D. 1998: The alternatives in language assessment: advantages and disadvantages . TESOL Quarterly 30, 653-675 .

Brown, J. D. , Hudson, T. D. , Norris, J. M. and Bonk, W. in press: An investigation of second language task-based performance assessments. Honolulu: University of Hawaii Press .

Byrnes, H. 1998: Constructing curricula in collegiate foreign language departments. In Byrnes, H. , editor, Learning foreign and second languages. New York: The Modern Language Association of America , 262-296.

Clark, J. L. D. and Grognet, A. G. 1985: Development and validation of a performance-based test of ESL ‘survival skills’. In Hauptman, P. , LeBlanc, R. and Wesche, M. , editors, Second language performance testing. Ottawa: University of Ottawa Press , 89-110.

Delandshere, G. and Petrosky, A. 1998: Assessment of complex performances: limitations of key measurement assumptions . Educational Researcher 27, 14-24 .

Douglas, D. 2000: LSP assessment criteria: where do they come from ? Paper presented at the Second Asian Language Assessment Research Forum, January 11-12, 2000.

Douglas, D. and Myers, R. 2000: Assessing the communication skills of veterinary students: whose criteria? In Kunnan, A. J. , editor, Fairness and validation in language assessment: selected papers from the 19th Language Testing Research Colloquium, Orlando, Florida. New York: Cambridge University Press , 60-81.

Dunn-Rankin, P. 1983: Scaling methods. Hillsdale, NJ: Lawrence Erlbaum .

10.

Eisner, E. W. 1999: The uses and limits of performance assessment . Phi Delta Kappan 80, 658-660 .

11.

Guttman, L. 1950: The basis of scalogram analysis. In Stouffer, S. A. , editor, Measurement and prediction. Princeton, NJ: Princeton University .

12.

Haertel, E. H. 1999: Performance assessment and education reform . Phi Delta Kappan 80, 662-666 .

13.

Jacoby, S. and McNamara, T. 1999: Locating competence . English for Specific Purposes 18, 213-241 .

14.

Khattri, N. , Reeve, A. and Kane, M. 1998: Principles and practices of performance assessment. Mahwah, NJ: Lawrence Erlbaum .

15.

Linn, R. L. and Burton, E. 1994: Performance-based assessment: implications of task specificity . Educational Measurement: Issues and Practice 13, 5-15 .

16.

Long, M. H. 1985: A role for instruction in second language acquisition: task-based training. In Hyltenstam, K. and Pienemann, M. , editors, Modelling and assessing second language acquisition. Clevedon, Avon: Multilingual Matters , 77-100.

17.

Long, M. and Crookes, G. 1993: Units of analysis in syllabus design. In Crookes, G. , and Gass, S. , editors, Tasks in a pedagogical context: integrating theory and practice. Clevedon: Multilingual Matters , 9-54.

18.

Long, M. H. and Norris, J. M. 2000: Task-based language teaching and assessment. In Byram, M. , editor, Encyclopedia of language teaching. London: Routledge , 597-603.

19.

McNamara, T. 1996: Measuring second language performance. New York: Longman .

20.

Messick, S. 1994: The interplay of evidence and consequences in the validation of performance assessments . Educational Researcher 23, 13-23 .

21.

Mislevy, R. J. , Steinberg, L. S. , Breyer, R. J. , Almond, R. G. and Johnson, L. 1998: A cognitive task analysis, with implications for designing a simulation-based performance assessment. Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST) .

22.

Nichols, P. and Sugrue, B. 1999: The lack of fidelity between cognitively complex constructs and conventional test development practice . Educational Measurement: Issues and Practice 18, 18-29 .

23.

Norris, J. M. 2000: Purposeful language assessment . English Teaching Forum 38, 18-23 .

24.

Norris, J. M. 2001: Identifying rating criteria for task-based EAP assessment. In Hudson, T. D. and Brown, J. D. , editors, A focus on language test development: expanding the language proficiency construct across a variety of tests. Honolulu: University of Hawaii Press , 163-204.

25.

Norris, J. M. , Brown, J. D. , Hudson, T. D. and Yoshioka, J. 1998: Designing second language performance assessments. Honolulu: University of Hawaii Press .

26.

Norris, J. M. and Ortega, L. (in press). Defining and measuring L2 acquisition. In Doughty, C. and Long, M. H. , editors, Handbook of second language acquisition. London: Blackwell .

27.

Norton, B. 2000: Writing assessment: language, meaning, and marking memoranda . In Kunnan, A. J. , editor, Fairness and validation in language assessment: selected papers from the 19th Language Testing Research Colloquium, Orlando, Florida. New York: Cambridge University Press, 20-29 .

28.

Popham, W. 1997: What’s wrong - and what’s right - with rubrics. Educational Leadership , October, 72-75.

29.

Robinson, P. 2001: Task complexity, task difficulty, and task production: exploring interactions in a componential framework . Applied Linguistics 22, 27-57 .

30.

Robinson, P. and Ross, S. 1996: The development of task-based testing in English for academic purposes contexts . Applied Linguistics 17, 455-476 .

31.

Royer, J. M. , Cisero, C. A. and Carlo, M. S. 1993: Techniques and procedures for assessing cognitive skills . Review of Educational Research 63, 201-243 .

32.

Shohamy, E. 1995: Performance assessment in language testing . Annual Review of Applied Linguistics 15, 188-211 .

33.

Skehan, P. 1996: A framework for the implementation of task-based instruction . Applied Linguistics 17, 38-62 .

34.

Skehan, P. 1998: A cognitive approach to language learning. Oxford: Oxford University Press .

35.

Wiggins, G. 1993: Assessing student performance: exploring the purpose and limits of testing. San Francisco, CA: Jossey-Bass .