Based on evidence that listeners may favor certain foreign accents over others (Gass & Varonis, 1984; Major, Fitzmaurice, Bunta, & Balasubramanian, 2002; Tauroza & Luk, 1997) and that language-test raters may better comprehend and/or rate the speech of test takers whose native languages (L1s) are more familiar on some level (Carey, Mannell, & Dunn, 2011; Fayer & Krasinski, 1987; Scales, Wennerstrom, Richard, & Wu, 2006), we investigated whether accent familiarity (defined as having learned the test takers’ L1) leads to rater bias. We examined 107 raters’ ratings on 432 TOEFL iBTTM speech samples from 72 test takers. The raters of interest were L2 speakers of Spanish, Chinese, or Korean, while the test takers comprised three native-speaker groups (24 each) of Spanish, Chinese, and Korean. We analyzed the ratings using a multifaceted Rasch measurement approach. Results indicated that L2 Spanish raters were significantly more lenient with L1 Spanish test takers, as were L2 Chinese raters with L1 Chinese test takers. We conclude by concurring with Xi and Mollaun (2009, 2011) and Carey et al. that rater training should address raters’ linguistic background as a potential rater effect. Furthermore, we discuss the importance of recognizing rater L2 as a possible source of bias.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
BachmanL. F. (2004). Analyzing test tasks. Statistical analyses for language assessment (pp. 119–152). Cambridge: Cambridge University Press.
3.
BentT.BradlowA. R. (2003). The interlanguage speech intelligibility benefit. Journal of the Acoustical Society of America, 114(3), 1600–1610.
4.
BondT. G.FoxC. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
5.
BonkW. J.OckeyG. J. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing, 20(1), 89–110.
BrodkeyD. (1972). Dictation as a measure of mutual intelligibility: A pilot study. Language Learning, 22(2), 203–220.
8.
BrownA. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 12(1), 1–15.
9.
CareyM. D.MannellR. H.DunnP. K. (2011). Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews?Language Testing, 28(2), 201–219.
10.
CargileA. C.GilesH. (1998). Language attitudes toward varieties of English: An American-Japanese context. Journal of Applied Communication Research, 26(3), 338–356.
11.
Chalhoub-DevilleM. (1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12(1), 16–33.
12.
Chalhoub-DevilleM.TurnerC. E. (2000). What to look for in ESL admission tests: Cambridge Certificate Exams, IELTS, and TOEFL. System, 28(4), 523–539.
13.
ClarkC. M.GarrettM. F. (2004). Rapid adaptation to foreign-accented English. Journal of the Acoustical Society of America, 116(6), 3647–3658.
14.
DaviesA. (2011). Does language testing need the native speaker?Language Assessment Quarterly, 8(3), 291–308.
15.
DerwingT. M.MunroM. J. (1997). Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19(1), 1–16.
16.
DerwingT. M.MunroM. J. (2005). Second language accent and pronunciation teaching: A research-based approach. TESOL Quarterly, 39(3), 379–398.
EngelhardG.Jr.MyfordC. M. (2003). Monitoring faculty consultant performance in the Advanced Placement English Literature and Composition program with a many-faceted Rasch model. New York: College Board.
19.
FayerJ. M.KrasinskiE. (1987). Native and nonnative judgments of intelligibility and irritation. Language Learning, 37(3), 313–326.
20.
FisherW. P.Jr. (1992). Reliablity statistics. Rasch Measurement: Transactions of the Rasch Measurement SIG, 6(3), 238.
21.
FlocciaC.ButlerJ.GoslinJ.EllisL. (2009). Regional and foreign accent processing in English: Can listeners adapt?Journal of Psycholinguistic Research, 38(4), 379–412.
22.
GassS. M.VaronisE. M. (1984). The effect of familiarity on the comprehensibility of nonnative speech. Language Learning, 34(1), 65–89.
23.
HardingL. (2012). Accent, listening assessment and the potential for a shared-L1 advantage: A DIF perspective. Language Testing, 29(2), 163–180.
24.
HillK. (1996). Who should be the judge? The use of non-native speakers as raters on a test of English as an international language. Melbourne Papers in Language Testing, 5(2), 29–49.
25.
JohnsonJ. S.LimG. S. (2009). The influence of rater language background on writing performance assessment. Language Testing, 26(4), 485–505.
26.
KennedyS.TrofimovichP. (2008). Intelligibility, comprehensibility, and accentedness of L2 Speech: The role of listener experience and semantic context. The Canadian Modern Language Review, 64(3), 459–489.
27.
KimY. H. (2009). An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach. Language Testing, 26(2), 187–217.
28.
LinacreJ. M. (2009). FACETS Rasch-model computer program (Version 3.66.1) [Computer software]. Available at www.winsteps.com.
29.
LinacreJ. M. (2010). A user’s guide to FACETS Rasch-model computer programs. Program manual 3.67.0. [Software manual]. Available at www.winsteps.com.
30.
LumleyT.McNamaraT. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12(2), 238–257.
31.
LuomaS. (2004). Assessing speaking. Cambridge: Cambridge University Press.
32.
MajorR. C.FitzmauriceS. M.BuntaF.BalasubramanianC. (2002). The effects of nonnative accents on listening comprehension: Implications for ESL assessment. TESOL Quarterly, 36(2), 173–190.
33.
McNamaraT. (2001). Language assessment as social practice: Challenges for research. Language Testing, 18(4), 333–349.
34.
McNamaraT.HillK.MayL. (2002). Discourse and assessment. Annual Review of Applied Linguistics, 22, 221–242.
35.
MunroM. J. (2008). Foreign accent and speech intelligibility. In EdwardsJ. G. H.ZampiniM. L. (Eds.), Phonology and second language acquisition (pp. 193–218). Philadelphia, PA: John Benjamins.
36.
MunroM. J.DerwingT. M. (1995). Foreign accent, comprehensibility and intelligibility in the speech of second language learners. Language Learning, 48(1), 73–97.
37.
MunroM. J.DerwingT. M.MortonS. L. (2006). The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 28(1), 111–131.
38.
MyfordC. M.WolfeE. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part 1. Journal of Applied Measurement, 4(4), 386–422.
39.
MyfordC. M.WolfeE. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189–227.
40.
SantosT. (1988). Professors’ reactions to the academic writing of non-native speaking students. TESOL Quarterly, 22(1), 69–90.
41.
ScalesJ.WennerstromA.RichardD.WuS. H. (2006). Language learners’ perceptions of accent. TESOL Quarterly, 40(4), 715–738.
42.
SidarasS. K.AlexanderJ. E. D.NygaardL. C. (2009). Perceptual learning of systemic variation in Spanish-accented speech. The Journal of the Acoustical Society of America, 125(5), 3306–3316.
43.
SmithL. E.BisazzaJ. A. (1982). The comprehensibility of three varieties of English for college students in seven countries. Language Learning, 32(2), 259–269.
44.
SmithL. E.RafiqzadK. (1979). English for cross-cultural communication: The question of intelligibility. TESOL Quarterly, 13(3), 371–380.
45.
StoneM.WrightB. D. (1988). Separation statistics in Rasch measurement (Research Memorandum No. 51). Chicago. IL: MESA Press.
46.
TaurozaS.LukJ. (1997). Accent and second language listening comprehension. RELC Journal, 28(1), 54–71.
47.
TennantA.PallantJ. F. (2008). Unidimensionality matters! (A tale of two Smiths?). Rasch Measurement: Transactions of the Rasch Measurement SIG, 20(1), 1048–1051.
48.
WeigleS. C.BoldtH.ValsecchiM. I. (2003). Effects of task and rater background on the evaluation of ESL student writing: A pilot study. TESOL Quarterly, 37(2), 345–354.
49.
WeilS. A. (2001). Foreign-accented speech: Encoding and generalization. Journal of the Acoustical Society of America, 109, 2473 (A).
50.
WigglesworthG. (1994). Patterns of rater behaviour in the assessment of an oral interaction test. Australian Review of Applied Linguistics, 17(2), 77–103.
51.
WinkeP.GassS.MyfordC. (2011). The relationship between raters’ prior language study and the evaluation of foreign language speech samples. (TOEFL iBT Research Report RR-11–30). Princeton, NJ: Educational Testing Service.
52.
WolfeE. W.DobriaL. (2008). Applications of the multifaceted Rasch model. In OsborneJ. (Ed.), Best practices in quantitative methods (pp. 71–85). Thousand Oaks, CA: SAGE Publications.
53.
WrightB. D.MastersG. N. (2002). Number of person or item strata (4G+1)/3. Rasch Measurement: Transactions of the Rasch Measurement SIG, 16(3), 888.
54.
XiX.MollaunP. (2009). How do raters from India perform in scoring the TOEFL iBTTM speaking section and what kind of training helps? (TOEFL iBT Research Report RR-09–31). Princeton, NJ: Educational Testing Service.
55.
XiX.MollaunP. (2011). Using raters from India to score a large-scale speaking test. Language Learning, 61(4), 1222–1255.
56.
ZhangB.ElderC. (2011). Judgments of oral proficiency by non-native and native English speaking teacher ratings: Competing or complementary constructs?Language Testing, 28(1), 31–50.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
