Nonverbal behavior can impact language proficiency scores in speaking tests, but there is little empirical information of the size or consistency of its effects or whether language proficiency may be a moderating variable. In this study, 100 novice raters watched and scored 30 recordings of test takers taking an international, high stakes proficiency test. The speech samples were each 2 minutes long and ranged in proficiency levels. The raters scored each sample on fluency, vocabulary, grammar, and comprehensibility using 7-point semantic differential scales. Nonverbal behavior was extracted using an automated machine learning software called iMotions, and data was analyzed with ordinal mixed effects regression. Results showed that attentional variance predicted fluency, vocabulary, and grammar scores, but only when accounting for proficiency. Higher standard deviations of attention corresponded with lower scores for the lower-proficiency group, but not the mid/higher-proficiency group. Comprehensibility scores were only predicted by mean valence when proficiency was an interaction term. Higher mean valence, or positive emotional behavior, corresponded with higher scores in the lower-proficiency group, but not the mid/higher-proficiency group. Effect sizes for these predictors were quite small, with small amounts of variance explained. These results have implications for construct representation and test fairness.
BeringerM.SpohnF.HildebrandtA.WackerJ.RecioG. (2019). Reliability and validity of machine vision for the assessment of facial expressions. Cognitive Systems Research, 56, 119–132. https://doi.org/10.1016/j.cogsys.2019.03.009
4.
BrantR. (1990). Assessing proportionality in the proportional odds model for ordinal logistic regression. Biometrics, 46(4), 1171–1178. https://doi.org/10.2307/2532457
5.
BuckR.VanLearC. A. (2002). Verbal and nonverbal communication: Distinguishing symbolic, spontaneous, and pseudo-spontaneous nonverbal behavior. Journal of Communication, 52(3), 522–541. https://doi.org/10.1111/j.1460-2466.2002.tb02560.x
BurtonJ. D. (2023b). The impact of nonverbal behavior on ratings of second language proficiency (OSF SK726, Version V1) [Data set]. Open Science Framework. https://osf.io/sk726/
10.
BurtonJ. D. (2023c). The role of nonverbal behavior and affect on ratings of second language proficiency [Doctoral dissertation, Michigan State University]. ProQuest Dissertations & Theses Global.
11.
CareyM. D.SzocsS. (2024). Revisiting raters’ accent familiarity in speaking tests: Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings. Language Testing, 41, 290–315.
12.
ChoiJ. S. (2022). Investigating test delivery modes within video-conferenced English speaking proficiency assessment [Unpublished doctoral dissertation, Michigan State University].
13.
CrossM. P.AcevedoA. M.HunterJ. F. (2023). A critique of automated approaches to code facial expressions: What do researchers need to know?Affective Science, 4(3), 500–505. https://doi.org/10.1007/s42761-023-00195-0
14.
DahlT. I.LudvigsenS. (2014). How I see what you’re saying: The role of gestures in native and foreign language listening comprehension. The Modern Language Journal, 98(3), 813–833. https://doi.org/10.1111/modl.12124
15.
Doherty-SneddonG.PhelpsF. G. (2005). Gaze aversion: A response to cognitive or social difficulty?Memory and Cognition, 33(4), 727–733. https://doi.org/10.3758/bf03195338
16.
DrijversL.ÖzyürekA. (2017). Visual context enhanced: The joint contribution of iconic gestures and visible speech to degraded speech comprehension. Journal of Speech, Language, and Hearing Research, 60(1), 212–222. https://doi.org/10.1044/2016_JSLHR-H-16-0101
17.
DucasseA. M.BrownA. (2009). Assessing paired orals: Raters’ orientation to interaction. Language Testing, 26(3), 423–443. https://doi.org/10.1177/0265532209104669
18.
DupréD.KrumhuberE. G.KüsterD.McKeownG. J. (2020). A performance comparison of eight commercially available automatic classifiers for facial affect recognition. PLOS ONE, 15(4), Article e0231968. https://doi.org/10.1371/journal.pone.0231968
19.
GanZ.DavisonC. (2011). Gestural behavior in group oral assessment: A case study of higher- and lower-scoring students. International Journal of Applied Linguistics, 21(1), 95–120. https://doi.org/10.1111/j.1473-4192.2010.00264.x
GregersenT.Olivares-CuhatG.StormJ. (2009). An examination of L1 and L2 gesture use: What role does proficiency play?Modern Language Journal, 93(2), 195–208. https://doi.org/10.1111/j.1540-4781.2009.00856.x
iMotions. (2017). Facial expression analysis: The complete pocket guide.
27.
IsbellD. R.CrowtherD.NishizawaH. (2024). Speaking performances, stakeholder perceptions, and test scores: Extrapolating from the Duolingo English test to the university. Language Testing, 41, 233–262.
28.
JenkinsS.ParraI. (2003). Multiple layers of meaning in an oral proficiency test: The complementary roles of nonverbal, paralinguistic, and verbal behaviors in assessment decisions. The Modern Language Journal, 87(1), 90–107. https://doi.org/10.1111/1540-4781.00180
29.
KappasA.KrumhuberE.KüsterD. (2013). Facial behavior. In HallJ. A.KnappM. L. (Eds.), Nonverbal communication (pp. 131–165). De Gruyter. https://doi.org/10.1515/9783110238150.131
30.
KimY. L.LiuC.TrofimovichP.McDonoughK. (2024). Is nonverbal behavior during conversation related to perceived fluency?TESOL Journal. Article e795. Advance online publication. https://doi.org/10.1002/tesj.795
31.
KulkeL.FeyerabendD.SchachtA. (2020). A comparison of the Affectiva iMotions facial expression analysis software with EMG for identifying facial expressions of emotion. Frontiers in Psychology, 11(329), Article 329. https://doi.org/10.3389/fpsyg.2020.00329
32.
KüntzlerT.HöflingT. T. A.AlpersG. W. (2021). Automatic facial expression recognition in standardized and non-standardized emotional expressions. Frontiers in Psychology, 12, Article 627561. https://doi.org/10.3389/fpsyg.2021.627561
LumleyT. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters?Language Testing, 19(3), 246–276. https://doi.org/10.1191/0265532202lt230oa
35.
MatsumotoD.HwangH. S. (2012). Culture and emotion: The integration of biological and cultural contributions. Journal of Cross-Cultural Psychology, 43(1), 91–118. https://doi.org/10.1177/0022022111420147
36.
MayL. (2009). Co-constructed interaction in a paired speaking test: The raters’ perspective. Language Testing, 26(3), 397–421. https://doi.org/10.1177/0265532209104668
37.
MayL. (2011). Interactional competence in a paired speaking test: Features salient to raters. Language Assessment Quarterly, 8(2), 127–145. https://doi.org/10.1080/15434303.2011.565845
38.
MondadaL. (2016). Challenges of multimodality: Language and the body in social interaction. Journal of Sociolinguistics, 20(3), 336–366. https://doi.org/10.1016/j.pragma.2014.04.004
39.
NagleC. L.TrofimovichP.O’BrienM. G.KennedyS. (2022). Beyond linguistic features: Exploring behavioral and affective correlates of comprehensible second language speech. Studies in Second Language Acquisition, 44(1), 255–270. https://doi.org/10.1017/S0272263121000073
40.
NakatsuharaF.InoueC.BerryV.GalacziE. (2017). Exploring the use of video-conferencing technology in the assessment of spoken language: A mixed-methods study. Language Assessment Quarterly, 14(1), 1–18. https://doi.org/10.1080/15434303.2016.1263637
41.
NakatsuharaF.InoueC.TaylorL. (2021). Comparing rating modes: Analyzing live, audio, and video ratings of IELTS speaking test performances. Language Assessment Quarterly, 18(2), 83–106. https://doi.org/10.1080/15434303.2020.1799222
42.
NeuJ. (1990). Assessing the role of nonverbal communication in the acquisition of communicative competence in L2. In ScarcellaR. C.AndersenE. S.KrashenS. D. (Eds.), Developing communicative competence in a second language (pp. 121–138). Newbury House.
43.
PlonskyL.OswaldF. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. https://doi.org/10.1111/lang.12079
44.
SatoT.McNamaraT. (2019). What counts in second language oral communication ability? The perspective of linguistic laypersons. Applied Linguistics, 40(6), 894–916. https://doi.org/10.1093/applin/amy032
45.
SeoM. S.KoshikI. (2010). A conversation analytic study of gestures that engender repair in ESL conversational tutoring. Journal of Pragmatics, 42(8), 2219–2239. https://doi.org/10.1016/j.pragma.2010.01.021
46.
SimmsL. J.ZelaznyK.WilliamsT. F.BernsteinL. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557–566. https://doi.org/10.1037/pas0000648
47.
SniderJ. G.OsgoodC. E. (Eds.). (1969). Semantic differential technique: A sourcebook. Aldine.
48.
StamG. (2008). What gestures reveal about second language acquisition. In McCaffertyS. G.StamS. (Eds.), Gesture: Second language acquisition and classroom research (pp. 231–256). Routledge.
49.
StöckliS.Schulte-MecklenbeckM.BorerS.SamsonA. C. (2018). Facial expression analysis with Affdex and Facet: A validation study. Behavior Research Methods, 50, 1446–1460. https://doi.org/10.3758/s13428-017-0996-1
50.
ThompsonC. P. (2016). Preliminary study of the role of eye contact, gestures, and smiles produced by Chinese-as-a-first-language test takers on ratings assigned by English-as-a-first- language examiners during IELTS speaking tests [Unpublished MA thesis]. University of Victoria, Canada. http://hdl.handle.net/1828/7724
51.
TrofimovichP.TekinO.McDonoughK. (2021). Task engagement and comprehensibility in interaction: Moving from what second language speakers say to what they do. Journal of Second Language Pronunciation, 7(3), 435–461. https://doi.org/10.1075/jslp.21006.tro
52.
TsunemotoA.LindbergR.TrofimovichP.McDonoughK. (2022). Visual cues and rater perceptions of second language comprehensibility, accentedness, and fluency. Studies in Second Language Acquisition, 44(3), 659–684. https://doi.org/10.1017/S0272263121000425
53.
WestfallJ.KennyD. A.JuddC. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 2020–2045. https://doi.org/10.1037/xge0000014
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
