The content that test-takers attempt to convey is not always included in the construct definition of general English oral proficiency tests, although some English-for-academic-purposes (EAP) speaking tests and most writing tests tend to place great emphasis on the evaluation of the content or ideas in the performance. This study investigated the relative contribution of linguistic criteria and the elaboration of speech content to scores on a test of speaking proficiency. A speaking test was designed and administered to Japanese undergraduates to determine what criteria English teachers associate with general oral proficiency. Nine raters were recruited to rate 30 students’ monologues on three topics, using intuitive judgments of oral proficiency (referred to as Overall communicative effectiveness). Following this, they assigned scores to the monologues using five criteria: Grammatical accuracy, Fluency, Vocabulary range, Pronunciation, and Content elaboration/development. The raters were also asked to provide open-ended written comments on the factors contributing to their intuitive judgments. Statistical analyses of the scores – Rasch measurement, multiple regression, and multivariate generalizability (G) theory analysis – revealed that Content elaboration/development made a substantive contribution to the intuitive judgments and composite score. The present study enriches our understanding of general oral proficiency and the construct definition of proficiency tests.
BachmanL. F. (2004). Statistical analyses for language assessment. Cambridge: Cambridge University Press.
2.
BachmanL. F.PalmerA. S. (1996). Language testing in practice. Oxford: Oxford University Press.
3.
BachmanL. F.PalmerA. S. (2010). Language assessment in practice. Oxford: Oxford University Press.
4.
BarnwellD. (1989). ‘Naive’ native speakers and judgements of oral proficiency in Spanish. Language Testing, 6(2), 152–163.
5.
BerryV. (2007). Personality differences and oral test performance. Frankfurt am Main: Peter Lang.
6.
BondT. G.FoxC. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
7.
BrennanR. L. (2001a). Generalizability theory. New York: Springer-Verlag.
8.
BrennanR. L. (2001b). mGENOVA (Version 2.1) [Computer software]. Iowa City, IA: Iowa Testing Programs, University of Iowa.
9.
BrownA. (2007). An investigation of the rating process in the IELTS oral interview. In TaylorL.FalveyP. (Eds.), Studies in language testing 19: IELTS collected papers: Research in speaking and writing assessment (pp. 98–139). Cambridge: Cambridge University Press.
10.
BrownA.IwashitaN.McNamaraT. (2005). An examination of rater orientations and test-taker performance on English-for-academic-purposes speaking tasks. [Monograph]. Retrieved from www.ets.org/Media/Research/pdf/RR-05-05.pdf
11.
CanaleM.SwainM. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1–47.
12.
Chalhoub-DevilleM. (1996). Performance assessment and the components of the oral construct across different tests and rater groups. In MilanovicM.SavilleN. (Eds.), Studies in language testing 3: Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Research Colloquium, Cambridge and Arnhem (pp. 55–73). Cambridge: Cambridge University Press.
13.
CohenJ. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.
14.
de JongJ. H. A. L.van GinkelL. W. (1992). Dimensions in oral foreign language proficiency. In VerhoevenL.de JongJ. H. A. L. (Eds.), The construct of language proficiency: Application of psychological models to language assessment. Amsterdam: John Benjamins.
15.
DucasseA. M. (2009). Raters as scale makers for an L2 Spanish speaking test: Using paired test discourse to develop a rating scale for communicative interaction. In BrownA.HillK. (Eds.), Language testing and evaluation 13: Tasks and criteria in performance assessment (pp. 1–22). Frankfurt am Main: Peter Lang.
16.
DucasseA. M.BrownA. (2009). Assessing paired orals: Raters’ orientation to interaction. Language Testing, 26(3), 423–443.
17.
EckesT. (2009). On common ground? How raters perceive scoring criteria in oral proficiency testing. In BrownA.HillK. (Eds.), Language testing and evaluation 13: Tasks and criteria in performance assessment (pp. 43–73). Frankfurt am Main: Peter Lang.
LinacreJ. M. (2009). Facets Rasch measurement computer program (Version 3.65.0) [Computer software]. Chicago, IL: Winsteps.com.
25.
LuomaS. (2004). Assessing speaking. Cambridge: Cambridge University Press.
26.
MayL. (2006). An examination of rater orientations on a paired candidate discussion task through stimulated verbal recall. Melbourne Papers in Language Testing, 11(1), 29–51.
27.
MayL. (2009). Co-constructed interaction in a paired speaking test: The raters’ perspective. Language Testing, 26(3), 397–421.
28.
McNamaraT. F. (1990). Item Response Theory and the validation of an ESP test for health professionals. Language Testing, 7(1), 52–75.
29.
McNamaraT. F. (1996). Measuring second language performance. Harlow, Essex: Pearson Education.
30.
MyfordC. M.WolfeE. W. (2000). Monitoring sources of variability within the Test of Spoken English Assessment System (TOEFL Research Report No. RR-65). Retrieved from Educational Testing Service website: www.ets.org/Media/Research/pdf/RR-00-06-Myford.pdf
31.
OrrM. (2002). The FCE speaking test: Using rater reports to help interpret test scores. System, 30(2), 143–154.
32.
PierceD.KinsellS. (2006). Cracking the TOEFL iBT. New York: The Princeton Review.
33.
PollittA.MurrayN. L. (1996). What raters really pay attention to. In MilanovicM.SavilleN. (Eds.), Studies in language testing 3: Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Research Colloquium, Cambridge and Arnhem (pp. 74–91). Cambridge: Cambridge University Press.
34.
PurpuraJ. E. (2008). Assessing communicative language ability: Models and their components. In ShohamyE.HornbergerN. H. (Eds.), Encyclopedia of language and education, vol.7: Language testing and assessment (2nd ed., pp. 53–68). New York: Springer Science+Business Media LLC.
35.
SawakiY. (2007). Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite. Language Testing, 24(3), 355–390.
36.
Society for Testing English Proficiency. (2008). The EIKEN can-do list. Tokyo: The Society for Testing English Proficiency.
37.
SpandelV.StigginsR. J. (1997). Creating writers: Linking writing assessment and instruction (2nd ed.). White Plains, NY: Longman.
38.
UnderhillN. (1987). Testing spoken language: A handbook of oral testing techniques. Cambridge: Cambridge University Press.
39.
WebbN. M.ShavelsonR. J.MaddahianE. (1983). Multivariate generalizability theory. In FyansL. J.Jr. (Ed.), Generalizability theory: Inferences and practical applications (pp. 67–81). San Francisco, CA: Jossey-Bass.
40.
WigglesworthG. (2008). Task and performance based assessment. In ShohamyE.HornbergerN. H. (Eds.), Encyclopedia of language and education, vol. 7: Language testing and assessment (2nd ed., pp. 111–122). New York: Springer Science+Business Media LLC.
