The present study explored anxiety and its fluctuation during an artificial intelligence (AI)-based English conversation test as a test validation study. Applying an idiodynamic approach, 13 Japanese university students learning English as a foreign language (EFL) took the test and participated in pre- and post-test interviews, which were analyzed by conducting reflexive thematic analysis. Individual differences in the pattern of anxiety experience fluctuations were observed, while factors affecting their anxiety experiences were also explored. These were classified into either authentic factors observed in both testing and non-testing contexts (e.g., perceived English proficiency, perceived communication skills, a two-way relationship between cognition and anxiety, and features of conversation) or factors unique to the test (evaluative situation and AI’s unnatural behavior). From the perspective of test validation, this study emphasized the importance of paying attention to construct-irrelevant variance, such as anxiety unnecessarily arising from test-specific factors.
AraiY. (2025a, September26). Anxiety in an artificial intelligence (AI)-based English conversation test: An idiodynamic approach to explore potential construct-irrelevant variance [Supplementary Materials]. osf.io/qr7j9
2.
AraiY. (2025b). Exploring flow as the consequences of placement decisions in an EFL speaking course. Language Testing in Asia, 15, 13. https://doi.org/10.1186/s40468-025-00347-y
3.
AubreyS. (2022). The relationship between anxiety, enjoyment, and breakdown fluency during second language speaking tasks: An idiodynamic investigation. Frontiers in Psychology, 13, Article 968946. https://doi.org/10.3389/fpsyg.2022.968946
4.
BachmanL.PalmerA. (2010). Language assessment in practice. Oxford University Press.
BielakJ.Mystkowska-WiertelakA. (2024). Emotions and emotion regulation in L2 classroom speaking tasks: A mixed-methods study combining the idiodynamic and quantitative perspectives. The Modern Language Journal, 108(3), 688–718. https://doi.org/10.1111/modl.12950
BraunV.ClarkeV. (2022). Thematic analysis: A practical guide. Sage.
9.
ChapelleC. A.EnrightM. K.JamiesonJ. (2008). Building a validity argument for the test of English as a foreign language. Routledge.
10.
Chukharev-HudilainenE.OckeyG. J. (2021). The development and evaluation of interactional competence elicitor for oral language assessments (ETS Research Report Series, RR-21-06). https://doi.org/10.1002/ets2.12319
11.
Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. https://rm.coe.int/16802fc1bf
12.
CreswellJ. W. (2015). A concise introduction to mixed methods research. Sage.
13.
CreswellJ. W.Plano ClarkV. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage.
14.
DaiD. W.SuzukiS.ChenG. (2025). Generative AI for professional communication training in intercultural contexts: Where are we now and where are we heading?Applied Linguistics Review, 16(2), 763–774. https://doi.org/10.1515/applirev-2024-0184
15.
DuJ.DanielB. K. (2024). Transforming language education: A systematic review of AI-powered chatbots for English as a foreign language speaking practice. Computers and Education: Artificial Intelligence, 6, 100230. https://doi.org/10.1016/j.caeai.2024.100230
16.
DuckerN. (2021). Protecting and enhancing willingness to communicate with idiodynamic peer-peer strategy sharing. System, 103, 102634. https://doi.org/10.1016/j.system.2021.102634
17.
Elahi ShirvanM.TalebzadehN. (2017). English as a foreign language learners’ anxiety and interlocutors’ status and familiarity: An idiodynamic perspective. Polish Psychological Bulletin, 48(4), 489–503. https://doi.org/10.1515/ppb-2017-0056
18.
El ShazlyR. (2021). Effects of artificial intelligence on English speaking anxiety and speaking performance: A case study. Expert Systems, 38(3), e12667. https://doi.org/10.1111/exsy.12667
19.
EricssonE.HashemiS. S.LundinJ. (2023). Fun and frustrating: Students’ perspectives on practising speaking English with virtual humans. Cogent Education, 10(1), 2170088. https://doi.org/10.1080/2331186X.2023.2170088
20.
ForsythC. M.LuceC.Zapata-RiveraD.JacksonG. T.EvaniniK.SoY. (2019). Evaluating English language learners’ conversations: Man vs. machine. Computer Assisted Language Learning, 32(4), 398–417. https://doi.org/10.1080/09588221.2018.1517126
21.
GarciaB. C.AppelC. (2021). Online speaking interaction in foreign languages: How and why do students experience anxiety? In SimonsM.SmitsT. (Eds.), Language education and emotions: Research into emotions and language learners, language teachers and educational processes (pp. 55–76). Routledge.
22.
GokturkN.ChukharevE. (2024). Exploring the potential of a spoken dialog system-delivered paired discussion task for assessing interactional competence. Language Assessment Quarterly, 21(1), 60–99. https://doi.org/10.1080/15434303.2023.2289173
23.
GregersenT.MacIntyreP. D.MezaM. D. (2014). The motion of emotion: Idiodynamic case studies of learners’ foreign language anxiety. The Modern Language Journal, 98(2), 574–588. https://doi.org/10.1111/modl.12084
HuangH.-T. D. (2018). Modeling the relationships between anxieties and performance in second/foreign language speaking assessment. Learning and Individual Differences, 63, 44–56. https://doi.org/10.1016/j.lindif.2018.03.002
28.
HuangH.-T. D.HungS.-T. A.HongH.-T. V. (2016). Test-taker characteristics and integrated speaking test performance: A path-analytic study. Language Assessment Quarterly, 13(4), 283–301. https://doi.org/10.1080/15434303.2016.1236111
29.
In’namiY.KoizumiR.JeonE. H.AraiY. (2022). L2 listening and its correlates: A meta-analysis. In JeonE. H.In’namiY. (Eds.), Understanding L2 proficiency: Theoretical and meta-analytic investigations (pp. 235–283). John Benjamins.
30.
IsaacsT. (2016). Assessing speaking. In TsagariD.BanerjeeJ. (Eds.), Handbook of second language assessment (pp. 131–146). DeGruyter Mouton.
31.
JangE. E.SawakiY. (2025). Advancing language assessment for teaching and learning in the era of the artificial intelligence revolution: Promises and challenges. Language Testing, 42(4), 361–368. https://doi.org/10.1177/02655322251348685
32.
JeonE. H.In’namiY.KoizumiR. (2022). L2 speaking and its external correlates: A meta-analysis. In JeonE. H.In’namiY. (Eds.), Understanding L2 proficiency: Theoretical and meta-analytic investigations (pp. 339–367). John Benjamins.
33.
KaneM. (2004). Certification testing as an illustration of argument-based validation. Measurement: Interdisciplinary Research and Perspectives, 2(3), 135–170. https://doi.org/10.1207/s15366359mea0203_1
34.
KoizumiR. (2021). The telephone standard speaking test: An outside evaluator’s investigation of a rebuttal to the generalization inference. In ChapelleC. A.VossE. (Eds.), Validity argument in language testing: Case studies of validation research (pp. 154–175). Cambridge University Press.
35.
KurataF.SaekiM.FujieS.MatsuyamaY. (2023). Multimodal turn-taking model using visual cues for end-of-utterance prediction in spoken dialogue systems. Proceedings of Interspeech, 2023, 2658–2662. https://doi.org/10.21437/Interspeech.2023-578
36.
LiuM. (2007). Language anxiety in EFL testing situations. ITL–International Journal of Applied Linguistics, 153(1), 53–75. https://doi.org/10.2143/ITL.153.0.2022821
37.
MacIntyreP. D. (1999). Language anxiety: A review of the research for language teachers. In YoungD. J. (Ed.), Affect in foreign language and second language learning (pp. 24–45). McGraw-Hill.
38.
MacIntyreP. D. (2012). The idiodynamic method: A closer look at the dynamics of communication traits. Communication Research Reports, 29(4), 361–367. https://doi.org/10.1080/08824096.2012.723274
39.
MacIntyreP. D. (2017). An overview of language anxiety research and trends in its development. In GkonouC.DaubneyM.DewaeleJ.-M. (Eds.), New insights into language anxiety: Theory, research and educational implications (pp. 11–30). Multilingual Matters.
40.
MacIntyreP. D.DuckerN. (2022). The idiodynamic method: A practical guide for researchers. Research Methods in Applied Linguistics, 1(2), 100007. https://doi.org/10.1016/j.rmal.2022.100007
41.
MacIntyreP. D.GregersenT. (2022). The idiodynamic method: Willingness to communicate and anxiety processes interacting in real time. International Review of Applied Linguistics in Language Teaching, 60(1), 67–84. https://doi.org/10.1515/iral-2021-0024
42.
MacIntyreP. D.LegattoJ. J. (2011). A dynamic system approach to willingness to communicate: Developing an idiodynamic method to capture rapidly changing affect. Applied Linguistics, 32(2), 149–171. https://doi.org/10.1093/applin/amq037
43.
MacIntyreP. D.McGillivrayM. F. (2023). The inner workings of anxiety in second language learning. Annual Review of Applied Linguistics, 43, 88–104. https://doi.org/10.1017/S0267190523000065
44.
MacIntyreP. D.NoelsK. A.ClémentR. (1997). Biases in self-ratings of second language proficiency: The role of language anxiety. Language Learning, 47(2), 265–287. https://doi.org/10.1111/0023-8333.81997008
45.
MacIntyreP. D.WangL. (2021). Willingness to communicate in the L2 about meaningful photos: Application of the pyramid model of WTC. Language Teaching Research, 25(6), 878–898. https://doi.org/10.1177/13621688211004645
46.
MacIntyreP. D.WangL. (2022). Anxiety. In LiS.HiverP.PapiH. (Eds.), The Routledge handbook of second language acquisition and individual differences (pp. 175–189). Routledge.
47.
MalterudK.SiersmaV. D.GuassoraA. D. (2016). Sample size in qualitative interview studies: Guided by information power. Qualitative Health Research, 26(13), 1753–1760. https://doi.org/10.1177/1049732315617444
48.
MatsuyamaY.SuzukiS.SaekiM.TakatsuH.MatsuuraR.AraiY. (2023, June8). Towards an explainable automated scoring of spoken interaction with a conversational AI agent. In JangE.SawakiY. (Convenors), Technology in diagnostic and formative language assessment: Friend or foe? [Colloquium]. Language Testing Research Colloquium.
49.
MessickS. (1995). Validity of psychological assessment. Validation of inferences from persons’ responses and performance as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
50.
OckeyG.Chukharev-HudilainenE. (2021). Human versus computer partner in the paired oral discussion test. Applied Linguistics, 42(5), 924–944. https://doi.org/10.1093/applin/amaa067
51.
O’SullivanB.GreenA. (2011). Test taker characteristics. In TaylorL. (Ed.), Examining speaking: Research and practice in assessing second language speaking (pp. 36–64). Cambridge University Press.
52.
PolatB.KimY. (2014). Dynamics of complexity and accuracy: A longitudinal case study of advanced untutored development. Applied Linguistics, 35(2), 184–207. https://doi.org/10.1093/applin/amt013
53.
SaekiM.TakatsuH.KurataF.SuzukiS.EguchiM.MatsuuraR.TakizawaK.YoshikawaS.MatsuyamaY. (2024). InteLLA: Intelligent language learning assistant for assessing language proficiency through interviews and roleplays. Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue. https://aclanthology.org/2024.sigdial-1.34
54.
SaghafiK.Elahi ShirvanM. (2020). Rapid changes in foreign language learning anxiety caused by a multiplicity of topics: An idiodynamic approach. Journal of Language and Education, 6(1), 83–102. https://doi.org/10.17323/jle.2020.9684
55.
SawakiY.AraiY.EguchiM.SuzukiS.MatsuyamaY. (2024, July4). Building an argument for test score interpretation and use for a fully automated online assessment of L2 spoken interaction [Conference presentation]. Language Testing Research Colloquium, Innsbruck, Austria.
SwainM. (1985). Large-scale communicative language testing: A case study. In LeeY. P. A.FokC. Y. Y.LordR.LowG. (Eds.), New directions in language testing (pp. 35–46). Pergamon.
TerryG.HayfieldN. (2021). Essentials of thematic analysis. American Psychological Association.
62.
Timpe-LaughlinV.SydorenkoT.DombiJ. (2024). Human versus machine: Investigating L2 learner output in face-to-face versus fully automated role-plays. Computer Assisted Language Learning, 37(1–2), 149–178. https://doi.org/10.1080/09588221.2022.2032184
63.
ToulminS. E. (2003). The uses of argument. Cambridge University Press.
64.
VossE. (2024). Language assessment and artificial intelligence. In KunnanA. J. (Ed.), The concise companion of language assessment (pp. 112–125). Wiley.
ZhaiC.WibowoS. (2023). A systematic review on artificial intelligence dialogue systems for enhancing English as foreign language students’ interactional competence in the university. Computers and Education: Artificial Intelligence, 4, 100134. https://doi.org/10.1016/j.caeai.2023.100134
67.
ZhangC.MengY.MaX. (2024). Artificial intelligence in EFL speaking: Impact on enjoyment, anxiety, and willingness to communicate. System, 121, 103259. https://doi.org/10.1016/j.system.2024.103259
