Test-task validation has been an important strand in recent revision projects for
University of Cambridge Local Examinations Syndicate (UCLES) examinations. This
article addresses the relatively neglected area of validating the match between
intended and actual test-taker language with respect to a blueprint of language
functions representing the construct of spoken language ability. An observation
checklist designed for both a priori and a posteriori analysis of speaking task
output has been developed. This checklist enables language samples elicited by the
task to be scanned for these functions in real time, without resorting to the
laborious and somewhat limited analysis of transcripts. The process and results of
its development, implications and further applications are discussed.
Get full access to this article
View all access options for this article.
References
1.
Anastasi, A.1988: Psychological testing. 6th edition.
New York: Macmillan
.
2.
Bachman, L. F.1990: Fundamental considerations in language testing.
Oxford: Oxford University Press
.
3.
Bachman, L. F.
and
Palmer, A. S.1981:
The construct validation of the FSI oral interview
. Language Learning31,
67-86
.
4.
Bachman, L. F.
and
Palmer, A. S.1996: Language testing in practice.
Oxford: Oxford University Press
.
5.
Ballman, T. L.1991: The oral task of picture description: similarities and
differences in native and nonnative speakers of Spanish. In
Teschner, R. V.
, editor, Assessing foreign language proficiency of undergraduates.
AAUSC Issues in Language Program Direction.
Boston: Heinle and Heinle
, 221-231.
6.
Brown, A.
1998:
Interviewer style and candidate performance in the IELST oral interview
. Paper presented at the Language Testing Research
Colloquium, Monterey, CA.
7.
Bygate, M.1988: Speaking.
Oxford: Oxford University Press
.
8.
Bygate, M.1999:
Quality of language and purpose of task: patterns of learners’
language on two oral communication tasks
. Language Teaching Research3,
185-214
.
9.
Chalhoub-Deville, M.1995a:
Deriving oral assessment scales across different tests and rater groups
. Language Testing12,
16-33
.
10.
Chalhoub-Deville, M.1995b:
A contextualized approach to describing oral language proficiency
. Language Learning45,
251-281
.
11.
Clark, J. L. D.1979: Direct vs. semi-direct tests of speaking ability.
In
Briere, E. J.
and
Hinofotis, F. B.
, editors, Concepts in language testing: some recent studies.
Washington DC: TESOL
.
12.
Clark, J. L. D.1988:
Validation of a tape-mediated ACTFL/ILR scale based test of Chinese
speaking proficiency
. Language Testing5,
187-205
.
13.
Clark, J. L. D.
and
Hooshmand, D.1992:
‘Screen to Screen’ testing: an exploratory study of
oral proficiency interviewing using video teleconferencing
. System20,
293-304
.
14.
Cronbach, L. J.1971: Validity. In
Thorndike, R. L.
, editor, Educational measurement. 2nd edition.
Washington DC: American Council on Education
, 443-597.
15.
Cronbach, L. J.1990: Essentials of psychological testing. 5th
edition.
New York: Harper & Row
.
16.
Davies, A.1977: The construction of language tests. In
Allen, J. P. B.
and
Davies, A.
, editors, Testing and experimental methods. The Edinburgh Course in
Applied Linguistics, Volume 4.
London: Oxford University Press
, 38-194.
17.
Davies, A.1990: Principles of language testing.
Oxford: Blackwell
.
18.
Ellerton, A.W. 1997: Considerations in the validation of semi-direct oral
testing. Unpublished PhD thesis, CALS, University of Reading.
19.
ffrench, A.
1999:
Language functions and UCLES speaking tests
. Seminar in Athens, Greece.
October 1999.
20.
Foster, P.
and
Skehan, P.1996:
The influence of planning and task type on second language performance
. Studies in Second Language Acquisition18,
299-323
.
21.
Foster, P.
and
Skehan, P.1999:
The influence of source of planning and focus of planning on task-based performance
. Language Teaching Research3,
215-247
.
22.
Fulcher, G.1994:
Some priority areas for oral language testing
. Language Testing Update15,
39-47
.
23.
Fulcher, G.1996:
Testing tasks: issues in task design and the group oral
. Language Testing13,
23-51
.
24.
Fulcher, G.1999:
Assessment in English for academic purposes: putting content validity in
its place
. Applied Linguistics20,
221-236
.
25.
Hayashi, M.1995: Conversational repair: a contrastive study of Japanese and
English. MA Project Report,
University of Canberra
.
26.
Henning, G.1983:
Oral proficiency testing: comparative validities of interview, imitation,
and completion methods
. Language Learning33,
315-332
.
27.
Henning, G.1987: A guide to language testing.
Cambridge, MA: Newbury House
.
28.
Kelly, R. 1978: On the construct validation of comprehension tests: an
exercise in applied linguistics. Unpublished PhD thesis, University of Queensland.
29.
Kenyon, D.
1995:
An investigation of the validity of task demands on performance-based tests
of oral proficiency
. In
Kunnan, A. J.
, editor, Validation in language assessment: selected papers from the
17th Language Testing Research Colloquium, Long
Beach. Mahwah, NJ: Lawrence Erlbaum,
19-40
.
30.
Kormos, J.1999:
Simulating conversations in oral-proficiency assessment: a conversation
analysis of role plays and non-scripted interviews in language exams
. Language Testing16,
163-188
.
31.
Lazaraton, A.1992:
The structural organisation of a language interview: a conversational
analytic perspective
. System20,
373-386
.
32.
Lazaraton, A.
1996:
A qualitative approach to monitoring examiner conduct in the Cambridge
assessment of spoken English (CASE)
. In
Milanovic, M.
and
Saville, N.
, editors, Performance testing, cognition and assessment: selected
papers from the 15th Language Testing Research Colloquium,
Cambridge and Arnhem. Studies in Language Testing 3.
Cambridge: University of Cambridge Local Examinations Syndicate,
18-33
.
33.
Lazaraton, A.2000: A qualitative approach to the validation of oral language
tests. Studies in Language Testing, Volume 14.
Cambridge: Cambridge University Press
.
34.
Lumley, T.
and
O’Sullivan, B.
2000:
The effect of speaker and topic variables on task performance in a
tape-mediated assessment of speaking
. Paper presented at the 2nd Annual Asian Language Assessment
Research Forum, The Hong Kong Polytechnic University.
35.
Luoma, S. 1997: Comparability of a tape-mediated and a face-to-face test of
speaking: a triangulation study. Unpublished Licentiate Thesis, Centre for Applied
Language Studies, Jyväskylä University, Finland.
36.
McNamara, T.1996: Measuring second language performance.
London: Longman
.
37.
Mehnert, U.1998:
The effects of different lengths of time for planning on second language performance
. Studies in Second Language Acquisition20,
83-108
.
38.
Messick, S.1975:
The standard problem: meaning and values in measurement and evaluation
. American Psychologist30,
955-966
.
39.
Messick, S.1989: Validity. In
Linn, R. L.
, editor, Educational measurement. 3rd edition.
New York: Macmillan
.
40.
Milanovic, M.
and
Saville, N.1996: Introduction. Performance testing, cognition
and assessment. Studies in Language Testing, Volume 3.
Cambridge: University of Cambridge Local
Examinations Syndicate
, 1-17.
41.
Moller, A. D. 1982: A study in the validation of proficiency tests of English
as a Foreign Language. Unpublished PhD thesis, University of Edinburgh.
42.
Norris, J
,
Brown, J. D.
,
Hudson, T.
and
Yoshioka, J.1998: Designing second language performance assessments.
Technical Report 18.
Honolulu, HI: University of Hawaii Press
.
43.
O’Loughlin, K.1995:
Lexical density in candidate output on direct and semi-direct versions of
an oral proficiency test
. Language Testing12,
217-237
.
44.
O’Loughlin, K. 1997: The comparability of direct and semi-direct
speaking tests: a case study. Unpublished PhD Thesis, University of Melbourne, Melbourne.
45.
O’Loughlin, K.2001: An investigatory study of the equivalence of direct and
semi-direct speaking skills. Studies in Language Testing 13.
Cambridge: Cambridge University Press/UCLES
.
46.
Ortega, L.1999:
Planning and focus on form in L2 oral performance
. Studies in Second Language Acquisition20,
109-148
.
47.
O’Sullivan, B. 2000: Towards a model of performance in oral
language testing. Unpublished PhD dissertation, CALS, University of Reading.
48.
Robinson, P.1995:
Task complexity and second language narrative discourse
. Language Learning45,
99-140
.
49.
Ross, S.
and
Berwick, R.1992:
The discourse of accommodation in oral proficiency interviews
. Studies in Second Language Acquisition14,
159-176
.
50.
Saville, N.
and
Hargreaves, P.1999:
Assessing speaking in the revised FCE
. ELT Journal53,
42-51
.
51.
Schegloff, E.
,
Jefferson, G.
and
Sachs, H.1977:
The preference for self-correction in the organisation of repair in conversation
. Language53,
361-382
.
52.
Schwartz, J.1980: The negotiation for meaning: repair in conversations
between second language learners of English. In
Larsen-Freeman, D.
, editor, Discourse analysis in second language research.
Rowley, MA: Newbury House
.
53.
Shohamy. E.1983.
The stability of oral language proficiency assessment in the oral interview
testing procedure
. Language Learning33,
527-540
.
54.
Shohamy. E.1988:
A proposed framework for testing the oral language of second/foreign
language learners
. Studies in Second Language Acquisition10,
165-179
.
55.
Shohamy. E.1994:
The validity of direct versus semi-direct oral tests
. Language Testing11,
99-123
.
56.
Shohamy, E.
,
Reves, T.
and
Bejarano, Y.1986:
Introducing a new comprehensive test of oral proficiency
. ELT Journal40,
212-220
.
57.
Skehan, P.1996:
A framework for the implementation of task based instruction
. Applied Linguistics17,
38-62
.
58.
Skehan, P.1998: A cognitive approach to language learning.
Oxford: Oxford University Press
.
59.
Stansfield, C. W.
and
Kenyon, D. M.1992:
Research on the comparability of the oral proficiency interview and the
simulated oral proficiency interview
. System20,
347-364
.
60.
Stenström, A.1994: An introduction to spoken interaction.
London: Long-man
.
61.
Suhua H. 1998: A communicative test of spoken English for the CET 6.
Unpublished PhD Thesis, Shanghai Jiao Tong University, Shanghai.
62.
Upshur, J. A.
and
Turner, C.1999:
Systematic effects in the rating of second-language speaking ability: test
method and learner discourse
. Language Testing16,
82-111
.
63.
van Ek, J. A.
and
Trim J. L. M.
, editors, 1984: Across the threshold.
Oxford: Pergamon
.
64.
van Lier, L.1989:
Reeling, writhing, drawling, stretching, and fainting in coils: oral
proficiency interviews as conversation
. TESOL Quarterly, 23,
489-508
.
Weir, C.J. 1983: Identifying the language needs of overseas students in
tertiary education in the United Kingdom. Unpublished PhD thesis, University of London.
67.
Walker, C.1993: Understanding and developing language tests.
Hemel Hempstead: Prentice Hall
.
68.
Wigglesworth, G.1997:
An investigation of planning time and proficiency level on oral test discourse
. Language Testing14,
85-106
.
69.
Wigglesworth, G.
and
O’Loughlin, K.1993:
An investigation into the comparability of direct and semi-direct versions
of an oral interaction test in English
. Melbourne Papers in Language Testing2,
56-67
.
70.
Young, R.1995:
Conversational styles in language proficiency interviews
. Language Learning45,
3-42
.
71.
Young, R.
and
Milanovic, M.1992:
Discourse variation in oral proficiency interviews
. Studies in Second Language Acquisition14,
403-424
.
