This study investigated the construct validity of a local speaking test for international teaching assistants (ITAs) from a fairness perspective, by employing a multi-group confirmatory factor analysis (CFA) to examine the impact of task type and examinee first language (L1) background on the internal structure of the test. The test consists of three types of integrated speaking tasks (i.e., text-speaking, graph-speaking, and listening-speaking) and the three L1s that are most represented among the examinees are Mandarin, Hindi, and Korean. Using scores of 1804 examinees across three years, the CFA indicated a two-factor model with a general speaking factor and a listening task factor as the best-fitting internal structure for the test. The factor structure was invariant for examinees across academic disciplines and L1 backgrounds, although the three examinee L1 groups demonstrated different factor variances and factor means. Specifically, while Korean examinees showed a larger variance in oral English proficiency, Hindi examinees demonstrated a higher level of oral proficiency than did Mandarin and Korean examinees. Overall, the lack of significance for multiple task factors and the invariance of factor structure suggest that the test measures the same set of oral English skills for all examinees. Although the factor variances and factor means for oral proficiency differed across examinee L1 subgroups, they reflect the general oral proficiency profiles of English speakers from these selected L1 backgrounds in the university and therefore do not pose serious threats to the fairness of the test. Findings of this study have useful implications for fairness investigations on ITA speaking tests.
AbbottM. L. (2007). A confirmatory approach to differential item functioning on an ESL reading assessment. Language Testing, 24(1), 7–36.
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for Educational and Psychological Testing. Washington, DC: Author.
3.
BachmanL. F.PalmerA. S. (1982). The Construct Validation of Some Components of Communicative Proficiency. TESOL Quarterly, 16(4), 449–465.
4.
BaeJ.BachmanL. F. (1998). A latent variable approach to listening and reading: Testing factorial invariance across two groups of children in the Korean/English two-way immersion program. Language Testing, 15(3), 380–414.
5.
BaeJ.BachmanL. F. (2010). An investigation of four writing traits and two tasks across two languages. Language Testing, 27(2), 213–234.
6.
BaileyK. (1984). The foreign ITA problem. In BaileyK.PialorsiF.Zukowski-FaustJ. (Eds.), Foreign teaching assistants in U.S. universities (pp. 3–15). Washington, DC: National Association for Foreign Affairs.
7.
ByrneB. M. (2006). Structural equation modeling with EQS. Basic concepts, applications, and programming (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
8.
ChenZ.HenningG. (1985). Linguistic and cultural bias in language proficiency tests. Language Testing, 2(2), 155–163.
9.
DuranR. P. (1989). Testing linguistic minorities. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 573–587). New York: American Council on Education, Macmillan.
10.
ElderC. (1996). The effect of language background on’ foreign’ language test performance: The case of Chinese Italian and Modern Greek. Language Learning, 46(2), 233–282.
11.
ElderC.McNamaraT.CongdonP. (2003). Rasch techniques for detecting bias in performance assessments: An example comparing performance of native and non-native speakers on a test of academic English. Journal of Applied Measurement, 4, 181–197.
12.
GintherA. J. (1994). The internal construct validity of the Advanced Placement Spanish Language Examination for examinees of different ethic and language backgrounds. Unpublished Doctoral dissertation. University of New Mexico.
13.
GintherA.StevensJ. (1998). Language background, ethnicity, and the internal construct validity of the advanced placement Spanish language examination. In KunnanA. J. (ed), Validation in language assessment (pp. 169–194). Mahwah, NJ: Lawrence Erlbaum.
14.
GintherA.YanX. (2017). Interpreting the relationships between TOEFL iBT scores and GPA: Language proficiency, policy, and profiles. Language Testing, 35(2), 271–295
15.
GuL. (2011). At the interface between language testing and second language acquisition: Communicative language ability and test-taker characteristics. Unpublished Doctoral Dissertation. University of Iowa.
16.
HaleG. A.RockD. A.JireleT. (1989). Confirmatory factor analysis of the Test of English as a Foreign Language (TOEFL Research Rep. No. RR-32; ETS RR-89–42). Princeton, NJ: Educational Testing Service.
17.
HaleG. A.StansfieldC. W.RockD. A.HicksM. M.ButlerF. A.OllerJ. W.Jr. (1988). Multiple-choice cloze items and the Test of English as a Foreign Language (TOEFL Research Rep. No. RR-26; ETS RR-88–2). Princeton, NJ: Educational Testing Service.
18.
HarleyB.CumminsJ.SwainM.AllenP. (1990). The nature of language proficiency. In HarleyB.AllenP.CumminsJ.SwainM. (Eds.), The development of second language proficiency (pp. 7–25). New York: Cambridge University Press.
19.
HaugenM. (2017). Goals, objectives and formative assessment for instruction. Unpublished doctoral dissertation. Purdue University.
20.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: a Multidisciplinary Journal, 6(1), 1–55.
21.
IBM Corp. (2012). IBM SPSS Statistics for Windows (Version 21.0) [computer software]. Armonk, NY: IBM Corp.
22.
JenkinsS. (2000). Cultural and linguistic miscues: A case study of international teaching assistant and academic faculty miscommunication. International Journal of Intercultural Relations, 24(4), 477–501.
23.
KaneM. (2010). Validity and fairness. Language Testing, 27(2), 177–182.
24.
KatzI. R.XiX.KimH.ChengP. (2004). Elicited speech from graph items on the Test of Spoken English. TOEFL Research Report No. 74. Princeton, NJ: Educational Testing Service.
25.
KoretzD. M. (2008). Measuring up. Boston, MA: Harvard University Press.
26.
KunnanA. J. (2004). Test fairness. In MilanovicM.WeirC. (Eds.), European language testing in a global context (pp. 27–48). Cambridge, UK: Cambridge University Press.
27.
KunnanA. J. (2008). Towards a model of test evaluation: Using the Test Fairness and Wider Context frameworks. In TaylorL.WeirC. (Eds.), Multilingualism and assessment: Achieving transparency, assuring quality, sustaining diversity (pp. 229–251). Cambridge, UK: Cambridge University Press.
28.
OppenheimN. (1997). How international teaching assistant programs can protect themselves from lawsuits. (ERIC Document Reproduction Service No. ED408886).
29.
PaeT. (2012). Causes of gender DIF on an EFL language test: A multiple-data analysis over nine years. Language Testing, 29, 533–554.
30.
PloughI. C.BriggsS. L.Van BonnS. (2010). A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors. Language Testing, 27(2), 235–260.
31.
PophamW. J. (2008). Classroom assessment: What teachers need to know (5th ed.). New York: Pearson.
32.
RawlsJ. (1971). A theory of justice. Cambridge, MA: Harvard University Press.
33.
RobinsonP. (2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22, 27–57.
34.
RobinsonP. (2007). Task complexity, theory of mind, and intentional reasoning: Effects on L2 speech production, interaction, uptake and perceptions of task difficulty. International Review of Applied Linguistics, 45, 237–257.
35.
RoeverC. (2007). DIF in the assessment of second language pragmatics. Language Assessment Quarterly, 4(2), 165–189.
36.
SkehanP.FosterP. (1999). The influence of task structure and processing conditions on narrative retellings. Language learning, 49(1), 93–120.
37.
SkehanP.FosterP.MehnertU. (1998). Assessing and using tasks. Anthology Series-Seameo Regional Language Centre, 227–248.
38.
SongM. Y. (2008). Do divisible subskills exist in second language (L2) comprehension? A structural equation modeling approach. Language Testing, 25(4), 435–464.
39.
StrickerL. J.RockD. A. (2008). Factor structure of the TOEFL Internet-Based Test across subgroups. (TOEFL iBT Research Report No. 07; ETS Research Report No. 08–66). Princeton, NJ: Educational Testing Service.
40.
SwainM. (1984). Teaching and testing communicatively. TESL Talk, 15, 7–18.
41.
TurnerC. E. (1989). The underlying factor structure of L2 cloze test performance in Francophone, university-level students: Causal modeling as an approach to construct validation. Language Testing, 6(2), 172–197. doi: 10.1177/026553228900600205
42.
UiterwijkH.VallenT. (2005). Linguistic sources of item bias for second generation immigrants in Dutch tests. Language Testing, 22, 211–234.
43.
WarneR. T.YoonM.PriceC. (2014). Exploring the various interpretations of “test bias”. Cultural Diversity and Ethnic Minority Psychology, 20(4), 570–582.
44.
WittA. S. (2010). Establishing the validity of the task-based English speaking test (TBEST) for international teaching assistants. Doctoral dissertation: The University of Arizona.
45.
XiX. (2005). Do visual chunks and planning impact the overall quality of oral descriptions of graphs?Language Testing, 22, 463–508.
46.
XiX. (2010a). How do we go about investigating test fairness?Language Testing, 27(3), 291–300.
47.
XiX. (2010b). Aspects of performance on line graph description tasks: Influenced by graph familiarity and different task features. Language Testing, 27(1), 73–100.
48.
YanX. (2014). An examination of rater performance on a local oral English proficiency test: A mixed-methods approach. Language Testing, 31(4), 501–527.
49.
YanX.ThirakunkovitS.KauperN.GintherA. (2016). What do test takers say: Test-taker feedback as in put for quality control. In ReadJ. (Ed.), Post-admission language assessments of university students. New York, NY: Springer.
50.
YangR. (2009). A confirmatory factory analysis of the construct validity of the oral English proficiency test. OEPP Technical Papers 09–01.
51.
YuanF.EllisR. (2003). The effects of pre-task planning and on-Line planning on fluency, complexity and accuracy in L2 monologic oral production. Applied linguistics, 24(1), 1–27.
