Second-language (L2) testing researchers have explored the relationship between speakers’ overall speaking ability, reflected by holistic scores, and the speakers’ performance on speaking subcomponents, reflected by analytic scores (e.g., McNamara, 1990; Sato, 2011). These research studies have advanced applied linguists’ understanding of how raters view the components of effective speaking skills, but the authors of the studies either used analytic composite scores, instead of true holistic ratings, or ran regression analyses with highly correlated subscores, which is problematic. To address these issues, 10 experienced ITA raters rated the speaking of 127 international teaching assistant (ITA) candidates using a four-component analytic rubric. In addition, holistic ratings were provided for the 127 test takers from a separate (earlier) scoring by two experienced ITA raters. The two types of scores differentiated examinees in similar ways. The variability observed in students’ holistic scores was reflected in their analytic scores. However, among the four analytic subscales, examinees’ scores on Lexical and Grammatical Competence had the greatest differentiating power. Its scores indicated with a high level of accuracy who passed the test and who did not. The paper discusses the components contributing to ITAs’ L2 oral speaking proficiency, and reviews pedagogical implications.
AldersonC. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. (1st ed.). A&C Black.
2.
AllisonD.TaurozaS. (1995). The effect of discourse organisation on lecture comprehension. English for Specific Purposes, 14(2), 157–173. https://doi.org/10.1016/0889-4906(95)00007-E
BachmanL. F. (1990). Fundamental considerations in language testing. Oxford University Press.
6.
BachmanL. F.MasonM.LynchB. K. (1995). Investigating variability in tasks and rater judgements in a performance test of foreign language speaking. Language Testing, 12(2), 238–257. https://doi.org/10.1177/026553229501200206
7.
BachmanL. F.SavignonS. J. (1986). The evaluation of communicative language proficiency: A critique of the ACTFL oral interview. The Modern Language Journal, 70(4), 380–390. https://doi.org/10.2307/326817
8.
BaileyK. (1983). Foreign teaching assistants at U.S. universities: Problems in interaction and communication. TESOL Quarterly, 17(2), 308–310. https://doi.org/10.2307/3586658
9.
BarkaouiK. (2011). Effects of marking method and rater experience on ESL essay scores and rater performance. Assessment in Education: Principles, Policy and Practice, 18(3), 279–293. https://doi.org/10.1080/0969594X.2010.526585
10.
BondT.FoxC. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Lawrence Erlbaum.
11.
ChoiI. (2017). Empirical profiles of academic oral English proficiency from an international teaching assistant screening test. Language Testing, 34(1), 49–82. https://doi.org/10.1177/0265532215601881
12.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
13.
Council of Europe. (2001). Common European Framework of Reference for Language. Cambridge University Press.
14.
FarnsworthT. L. (2004). The effect of teaching skills on holistic ratings of language ability in performance tests for international teaching assistant selection [Unpublished master’s thesis]. University of California, Los Angeles, USA. https://doi.org/10.1111/j.0033-0124.1957.93_28.x
15.
FeinbergR. A.JurichD. P. (2017). Guidelines for interpreting and reporting subscores. Educational Measurement: Issues and Practice, 36(1), 5–13. https://doi.org/10.1111/emip.12142
16.
FlowerdewJ.TaurozaS. (1995). The effect of discourse markers on second language lecture comprehension. Studies in Second Language Acquisition, 17(4), 435–458. https://doi.org/10.1017/S0272263100014406
17.
GintherA. (2003). International teaching assistant testing: Policies and methods. In DouglasD. (Ed.), English language testing in U.S. colleges and universities (2nd ed.). Association of International Educators.
HahnL. D. (2004). Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38(2), 201–223. https://doi.org/10.2307/3588378
21.
Hamp-LyonsL. (Ed.) (1991). Assessing second language writing in academic contexts. Ablex.
22.
HarschC.MartinG. (2013). Comparing holistic and analytic scoring methods: Issues of validity and reliability. Assessment in Education: Principles, Policy and Practice, 20(3), 281–307. https://doi.org/10.1080/0969594X.2012.742422
23.
HsiehC.-N. (2011). Rater effects in ITA testing: ESL teachers’ versus American undergraduates’ judgements of accentedness, comprehensibility, and oral proficiency [Unpublished doctoral dissertation]. Michigan State University.
24.
IsaacsT. (2008). Towards defining a valid assessment criterion of pronunciation proficiency in non-native English-speaking graduate students. Canadian Modern Language Review, 64(4), 555–580. https://doi.org/10.3138/cmlr.64.4.555
25.
KangO. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38(2), 301–315. https://doi.org/10.1016/j.system.2010.01.005
26.
KangO.RubinD.PickeringL. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. Modern Language Journal, 94(4), 554–566. https://doi.org/10.1111/j.1540-4781.2010.01091.x
27.
KhabbazbashiN.GalacziE. D. (2020). A comparison of holistic, analytic, and part marking models in speaking assessment. Language Testing, 37(3), 333–360. https://doi.org/10.1177/0265532219898635
KnochU.McNamaraT. (2015). Rasch analysis. In PlonskyL. (Ed.), Advancing quantitative methods in second language research (pp. 275–304). Routledge.
30.
LeeY. W.GentileC.KantorR. (2009). Toward automated multi-trait scoring of essays: Investigating links among holistic, analytic, and text feature scores. Applied Linguistics, 31(3), 391–417. https://doi.org/10.1093/applin/amp040
31.
LuomaS. (2002). Assessing speaking. Cambridge University Press.
32.
MaW. (2021). Data and code book for Language Testing article: What the analytic versus holistic scoring of international teaching assistants’ can reveal: Lexical grammar matters. [Data set]. Inter-university Consortium for Political and Social Research. https://doi.org/10.3886/E115966V2
33.
McGregorL. (2007). An examination of comprehensibility in a high stakes oral proficiency assessment for prospective international teaching assistants [Unpublished doctoral dissertation]. University of Texas at Austin.
34.
McNamaraT. F. (1990). Item Response Theory and the validation of an ESP test for health professionals. Language Testing, 7(1), 52–76. https://doi.org/10.1177/026553229000700105
35.
PickeringL. (2001). The role of tone choice in improving ITA communication in the classroom. TESOL Quarterly, 35(2), 233–255. https://doi.org/10.2307/3587647
36.
PlakansB. S. (1997). Undergraduates’ experiences with and attitudes toward international teaching assistants. TESOL Quarterly, 31(1), 95–119. https://doi.org/10.2307/3587976
37.
PlonskyL. (Ed.) (2015). Advancing quantitative methods in second language research. Routledge.
38.
RoundsP. L. (1987). Characterizing successful classroom discourse for NNS teaching assistant training. TESOL Quarterly, 21(4), 643–671. https://doi.org/10.2307/3586987
39.
SalomoneA. M. (1998). Communicative grammar teaching: A problem for and a message from international teaching assistants. Foreign Language Annals, 31(4), 552–568. https://doi.org/10.1111/j.1944-9720.1998.tb00599.x
40.
SatoT. (2011). The contribution of test-takers’ speech content to scores on an English oral proficiency test. Language Testing, 29(2), 223–241. https://doi.org/10.1177/0265532211421162
TaylorL.GalacziE. (2011). Scoring validity. In TaylorL. (Ed.), Studies in language testing, Vol. 30 (pp. 171–233). Cambridge University Press.
43.
TylerA. (1992). Discourse structure and the perception of incoherence in international teaching assistants’ spoken discourse. TESOL Quarterly, 26(4), 713–729. https://doi.org/10.2307/3586870
44.
WeigleS. C. (2002). Assessing writing. Cambridge University Press.
45.
WeirC. J. (1990). Communication language testing. Prentice-Hall.
46.
XiX. (2005). Do visual chunks and planning impact performance on the graph description task in the SPEAK exam?Language Testing, 22(4), 463–508. https://doi.org/10.1191/0265532205lt305oa
47.
XiX. (2007). Investigating the utility of analytic scoring for the TOEFL Academic Speaking Test (TAST). Language Testing, 24(2), 251–286. https://doi.org/10.1177/0265532207076365
48.
XiX.MollaunP. (2006). Evaluating analytic scoring for the TOEFL® Academic Speaking Test (TAST) for operational use. TOEFL IBT Research Report, 1, 1–71. https://doi.org/10.1002/j.2333-8504.2006.tb02013.x
