With recent technological advances, researchers have begun to explore the potential use of spoken dialog systems (SDSs) for L2 oral communication assessment. While several studies support the feasibility of building these systems for various types of oral tasks, research on the construct validity of SDS-delivered tasks is still limited. Thus, this study examines the cognitive processes engaged by an SDS-delivered paired discussion task, focusing on strategic competence, an essential component of L2 oral communication ability. Thirty adult test-takers completed a paired discussion task with an SDS acting as an interlocutor and provided stimulated recalls about their strategy use in the task. Three trained raters independently evaluated the test-takers’ oral task responses using a holistic rating scale devised for the task. Findings revealed the use of six categories of construct-relevant strategies during task performance. While no statistically significant differences were found in the use of these categories between high- and low-ability test-takers, marked differences were observed in the use of individual strategies within the categories between the test-takers at the two levels. These findings provide insight into how test-takers at different ability levels cognitively interact with SDS-delivered paired discussion tasks and offer implications for the design and validation of such tasks.
AmalbertiR.CarbonellN.FalzonP. (1993). User representations of computer systems in human-computer speech interaction. International Journal of Man-machine Studies, 38, 547–566. https://doi.org/10.1006/imms.1993.1026
2.
ArtuncE.HartD. (2020). Interactional competence in paired speaking tests: A study on proficiency-based pairings. System, 89, 102–194. https://doi.org/10.1016/j.system.2019.102194
3.
BachmanL. (1990). Fundamental considerations in language testing. Oxford University Press.
4.
BachmanL.PalmerA. (1996). Language testing in practice. Oxford University Press.
BrownA.IwashitaN.McNamaraT. (2005). An examination of rater orientations and test taker performance on English for academic purposes speaking tasks (TOEFL Monograph No. MS-29). ETS. https://doi.org/10.1002/j.2333-8504.2005.tb01982.x
8.
CampbellJ.QuincyC.OssermanJ.PedersenO. (2013). Coding in-depth semi-structured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research, 42(3), 294–320. https://doi.org/10.1177/0049124113500475
9.
CanaleM.SwainM. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1–47. https://doi.org/10.1093/applin/1.1.1
Chukharev-HudilainenE.OckeyG. (2021). The development and evaluation of Interactional Competence Elicitor for oral language assessments (TOEFL Research Report No. RR-92). ETS. https://doi.org/10.1002/ets2.12319
12.
CohenA. (2014). Using test-wiseness strategy research in task development. In KunnanA. (Ed.), The companion to language assessment (pp. 893–905). Wiley Blackwell. https://doi.org/10.1002/9781118411360.wbcla006
13.
DombiJ.SydorenkoT.Timpe-LaughlinV. (2022). Common ground, cooperation, and recipient design in human-computer interactions. Journal of Pragmatics, 193, 4–20. https://doi.org/10.1016/j.pragma.2022.03.001
ErcikanK.PellegrinoJ. (Eds.). (2017). Validation of score meaning in the next generation of assessments: The use of response processes. Routledge. https://doi.org/10.4324/9781315708591
17.
EvaniniK.SoY.TaoJ.Zapata-RiveraD.LuceC.BattistiniL.WangX. (2014). Performance of a trialogue-based prototype system for English language assessment for young learners. Proceedings of the Interspeech Workshop on Child Computer Interaction, 2, 79–84. https://www.isca-speech.org/archive_v0/wocci_2014/papers/wc14_079.pdf
18.
FieldA. (2018). Discovering statistics using IBM SPSS Statistics (5th ed.). SAGE.
19.
FischerK. (2016). Designing speech for a recipient: The roles of partner modelling, alignment, and feedback in so-called “simplified registers.” John Benjamins. https://doi.org/10.1075/pbns.270
20.
ForsythC.LuceC.Zapata-RiveraD.Tanner JacksonG.EvaniniK.SoY. (2018). Evaluating English language learners’ conversations: Man vs. Machine. Computer Assisted Language Learning, 32(4), 398–417. https://doi.org/10.1080/09588221.2018.1517126
21.
FrostK.WigglesworthG.ClothierJ. (2021). Relationships between comprehension, strategic behaviors, and content-related aspects of test performances in integrated speaking. Language Assessment Quarterly, 18(2), 133–153. https://doi.org/10.1080/15434303.2020.1835918
22.
FulcherG. (2003). Testing second language speaking. Longman/Pearson Education.
23.
GalacziE. (2014). Interactional competence across proficiency levels: How do learners manage interaction in paired speaking tests?Applied Linguistics, 35(5), 553–574. https://doi.org/10.1093/applin/amt017
24.
GalacziE.TaylorL. (2018). Interactional competence: Conceptualisations, operationalisations, and outstanding questions. Language Assessment Quarterly, 15(3), 219–236. https://doi.org/10.1080/15434303.2018.1453816
25.
GanZ. (2010). Interaction in group oral assessment: A case study of higher- and lower-scoring students. Language Testing, 27(4), 585–602. https://doi.org/10.1177/0265532210364049
26.
GassS.MackeyA. (2000). Stimulated recall methodology in second language research. Lawrence Erlbaum Associates Publishers.
27.
GokturkN. (2020). Development and evaluation of a spoken dialog system-mediated paired oral task for measuring second language oral communication ability in English (Publication No. 28153696) [Doctoral dissertation, Iowa State University]. ProQuest Dissertations Publishing. https://doi.org/10.31274/etd-20210114-53
28.
HayesA.KrippendorffK. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. https://doi.org/10.1080/19312450709336664
KaneM. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
33.
LamD. (2018). What counts as “responding”? Contingency on previous speaker contribution as a feature of interactional competence. Language Testing, 35(3), 377–401. https://doi.org/10.1177/0265532218758126
34.
LamD. (2021). Don’t turn a deaf ear: A case for assessing interactive listening. Applied Linguistics, 42(4), 740–764. https://doi.org/10.1093/applin/amaa064
35.
LeaperD.BrawnJ. (2019). Detecting development of speaking proficiency with a group oral test: A quantitative analysis. Language Testing, 36(2), 181–206. https://doi.org/10.1177/0265532218779626
36.
MayL.NakatsuharaF.LamD.GalacziE. (2020). Developing tools for learning oriented assessment of interactional competence: Bridging theory and practice. Language Testing, 37(2), 165–188. https://doi.org/10.1177/0265532219879044
37.
McGrawK.WongS. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30
38.
NassC.MoonY. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81–103. https://doi.org/10.1111/0022-4537.00153
39.
NassC.MoonY.CarneyP. (1999). Are people polite to computers? Responses to computer-based interviewing systems. Journal of Applied Psychology, 29(5), 1093–1109. https://doi.org/10.1111/j.1559-1816.1999.tb00142.x
40.
OckeyG. (2009). The effects of a test taker’s group members’ personalities on the test taker’s second language group oral discussion test scores. Language Testing, 26(2), 161–186. https://doi.org/10.1177/0265532208101005
41.
OckeyG.Chukharev-HudilainenE. (2021). Human versus computer partner in the paired oral discussion task. Applied Linguistics, 42(5), 924–944. https://doi.org/10.1093/applin/amaa067
O’SullivanB. (2002). Learner acquaintanceship and oral proficiency test pair-task performance. Language Testing, 19(3), 277–295. https://doi.org/10.1191/0265532202lt205oa
RamanarayananV.LangeP.EvaniniK.MolloyH.Suendermann-OeftD. (2017). Human an automated scoring of fluency, pronunciation, and intonation during human-machine spoken dialog interaction. Proceedings of the Interspeech, 2017, 1711–1715. https://doi.org/10.21437/Interspeech.2017-1213
46.
RamanarayananV.Suendermann-OeftD.IvanovA.EvaniniK. (2015). A distributed cloud-based dialog system for conversational application development. Proceedings of the 16th Annual Meeting of the Special interest Group on Discourse and Dialogue2, 432–434. https://doi.org/10.18653/v1/W15-46588
47.
RamanarayananV.Suendermann-OeftD.LangeP.IvanovA.EvaniniK.YuZ.QianY. (2016). Bootstrapping development of a cloud-based spoken dialog system in the educational domain from scratch using crowdsourced data (Research Report No. RR-16- 16). ETS. https://doi.org/10.1002/ets2.12105
48.
SacksH.SchegloffE.JeffersonG. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4), 696–735. https://doi.org/10.2307/412243
49.
ShechtmanN.HorowitzL. (2003). Media inequality in conversation: How people behave differently when interacting with computers and people. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1, 281–288. https://doi.org/10.1145/642611.642661
SwainM.HuangL.BarkaouiK.BrooksL.LapkinS. (2009). The speaking section of the TOEFL iBT (SSTiBT): Test-takers’ reported strategic behaviors (TOEFL Research Report No. 10). ETS. https://doi.org/10.1002/j.2333-8504.2009.tb02187.x
52.
TaylorL. (Ed.). (2011). Examining speaking: Research and practice in assessing second language speaking. UCLES/Cambridge University Press.
53.
Timpe-LaughlinV.EvaniniK.GreenA.BloodI.DombiJ.RamanarayananV. (2017). Designing interactive, automated dialogues for L2 pragmatics learning. Proceedings of the SEMDIAL 2017 Workshop on the Semantics and Pragmatics of Dialogue, 21, 116–125. https://doi.org/10.21437/SemDial.2017-13
54.
Van MoereA. (2007). Group oral tests: How does task affect candidate performance and test scores? (Publication No. 507288). [Doctoral dissertation, Lancaster University]. British Library E-Thesis Online Services.
55.
WooffittR.FraserN.GilbertN.McGlashanS. (1997). Humans, computers and wizards: Human (simulated) computer interaction. Routledge.
56.
XiX.NorrisJ.OckeyG.FulcherG.PurpuraJ. (2021). Assessing academic speaking. In XiX.NorrisJ. (Eds.), Assessing academic English for higher education admissions (pp. 152–199). Routledge. https://doi.org/10.4324/9781351142403-5
57.
Yoshida-MoriseY. (1998). The use of communication strategies in language proficiency interviews. In YoungR.HeA. (Eds.), Talking and testing: Discourse approaches to the assessment of oral proficiency (pp. 205–238). John Benjamins.
58.
YounS.BiN. (2019). Investigating test-takers’ strategy use in task-based L2 pragmatic speaking assessment. Intercultural Pragmatics, 16(2), 185–218. https://doi.org/10.1515/ip-2019-0009
59.
ZhuX.LiaoX.CheongC. M. (2019). Strategy use in oral communication with competent synthesis and complex interaction. Journal of Psycholinguistic Research, 48(5), 1163–1183. https://doi.org/10.1007/s10936-019-09651-0
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
