Anxiety in an artificial intelligence (AI)-based English conversation test: An idiodynamic approach to explore potential construct-irrelevant variance

Abstract

The present study explored anxiety and its fluctuation during an artificial intelligence (AI)-based English conversation test as a test validation study. Applying an idiodynamic approach, 13 Japanese university students learning English as a foreign language (EFL) took the test and participated in pre- and post-test interviews, which were analyzed by conducting reflexive thematic analysis. Individual differences in the pattern of anxiety experience fluctuations were observed, while factors affecting their anxiety experiences were also explored. These were classified into either authentic factors observed in both testing and non-testing contexts (e.g., perceived English proficiency, perceived communication skills, a two-way relationship between cognition and anxiety, and features of conversation) or factors unique to the test (evaluative situation and AI’s unnatural behavior). From the perspective of test validation, this study emphasized the importance of paying attention to construct-irrelevant variance, such as anxiety unnecessarily arising from test-specific factors.

Keywords

AI-based conversation test anxiety idiodynamic method test-taker characteristics test validation

Get full access to this article

View all access options for this article.

References

Arai

(2025a, September 26). Anxiety in an artificial intelligence (AI)-based English conversation test: An idiodynamic approach to explore potential construct-irrelevant variance [Supplementary Materials]. osf.io/qr7j9

Arai

(2025b). Exploring flow as the consequences of placement decisions in an EFL speaking course. Language Testing in Asia, 15, 13. https://doi.org/10.1186/s40468-025-00347-y

Aubrey

(2022). The relationship between anxiety, enjoyment, and breakdown fluency during second language speaking tasks: An idiodynamic investigation. Frontiers in Psychology, 13, Article 968946. https://doi.org/10.3389/fpsyg.2022.968946

Bachman

Palmer

(2010). Language assessment in practice. Oxford University Press.

Bachman

L. F.

(2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1–34. https://doi.org/10.1207/s15434311laq0201_1

Bielak

Mystkowska-Wiertelak

(2024). Emotions and emotion regulation in L2 classroom speaking tasks: A mixed-methods study combining the idiodynamic and quantitative perspectives. The Modern Language Journal, 108(3), 688–718. https://doi.org/10.1111/modl.12950

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Braun

Clarke

(2022). Thematic analysis: A practical guide. Sage.

Chapelle

C. A.

Enright

M. K.

Jamieson

(2008). Building a validity argument for the test of English as a foreign language. Routledge.

10.

Chukharev-Hudilainen

Ockey

G. J.

(2021). The development and evaluation of interactional competence elicitor for oral language assessments (ETS Research Report Series, RR-21-06). https://doi.org/10.1002/ets2.12319

11.

Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. https://rm.coe.int/16802fc1bf

12.

Creswell

J. W.

(2015). A concise introduction to mixed methods research. Sage.

13.

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods research (3rd ed.). Sage.

14.

Dai

D. W.

Suzuki

Chen

(2025). Generative AI for professional communication training in intercultural contexts: Where are we now and where are we heading? Applied Linguistics Review, 16(2), 763–774. https://doi.org/10.1515/applirev-2024-0184

15.

Daniel

B. K.

(2024). Transforming language education: A systematic review of AI-powered chatbots for English as a foreign language speaking practice. Computers and Education: Artificial Intelligence, 6, 100230. https://doi.org/10.1016/j.caeai.2024.100230

16.

Ducker

(2021). Protecting and enhancing willingness to communicate with idiodynamic peer-peer strategy sharing. System, 103, 102634. https://doi.org/10.1016/j.system.2021.102634

17.

Elahi Shirvan

Talebzadeh

. (2017). English as a foreign language learners’ anxiety and interlocutors’ status and familiarity: An idiodynamic perspective. Polish Psychological Bulletin, 48(4), 489–503. https://doi.org/10.1515/ppb-2017-0056

18.

El Shazly

. (2021). Effects of artificial intelligence on English speaking anxiety and speaking performance: A case study. Expert Systems, 38(3), e12667. https://doi.org/10.1111/exsy.12667

19.

Ericsson

Hashemi

S. S.

Lundin

(2023). Fun and frustrating: Students’ perspectives on practising speaking English with virtual humans. Cogent Education, 10(1), 2170088. https://doi.org/10.1080/2331186X.2023.2170088

20.

Forsyth

C. M.

Luce

Zapata-Rivera

Jackson

G. T.

Evanini

(2019). Evaluating English language learners’ conversations: Man vs. machine. Computer Assisted Language Learning, 32(4), 398–417. https://doi.org/10.1080/09588221.2018.1517126

21.

Garcia

B. C.

Appel

(2021). Online speaking interaction in foreign languages: How and why do students experience anxiety? In Simons

Smits

(Eds.), Language education and emotions: Research into emotions and language learners, language teachers and educational processes (pp. 55–76). Routledge.

22.

Gokturk

Chukharev

(2024). Exploring the potential of a spoken dialog system-delivered paired discussion task for assessing interactional competence. Language Assessment Quarterly, 21(1), 60–99. https://doi.org/10.1080/15434303.2023.2289173

23.

Gregersen

MacIntyre

P. D.

Meza

M. D.

(2014). The motion of emotion: Idiodynamic case studies of learners’ foreign language anxiety. The Modern Language Journal, 98(2), 574–588. https://doi.org/10.1111/modl.12084

24.

Haladyna

T. M.

Downing

S. M.

(2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17–27. https://doi.org/10.1111/j.1745-3992.2004.tb00149.x

25.

Horwitz

(2001). Language anxiety and achievement. Annual Review of Applied Linguistics, 21, 112–126. https://doi.org/10.1017/S0267190501000071

26.

Horwitz

E. K.

Horwitz

M. B.

Cope

(1986). Foreign language classroom anxiety. The Modern Language Journal, 70(2), 125–132. https://doi.org/10.1111/j.1540-4781.1986.tb05256.x

27.

Huang

H.-T. D.

(2018). Modeling the relationships between anxieties and performance in second/foreign language speaking assessment. Learning and Individual Differences, 63, 44–56. https://doi.org/10.1016/j.lindif.2018.03.002

28.

Huang

H.-T. D.

Hung

S.-T. A.

Hong

H.-T. V.

(2016). Test-taker characteristics and integrated speaking test performance: A path-analytic study. Language Assessment Quarterly, 13(4), 283–301. https://doi.org/10.1080/15434303.2016.1236111

29.

In’nami

Koizumi

Jeon

E. H.

Arai

(2022). L2 listening and its correlates: A meta-analysis. In Jeon

E. H.

In’nami

(Eds.), Understanding L2 proficiency: Theoretical and meta-analytic investigations (pp. 235–283). John Benjamins.

30.

Isaacs

(2016). Assessing speaking. In Tsagari

Banerjee

(Eds.), Handbook of second language assessment (pp. 131–146). DeGruyter Mouton.

31.

Jang

E. E.

Sawaki

(2025). Advancing language assessment for teaching and learning in the era of the artificial intelligence revolution: Promises and challenges. Language Testing, 42(4), 361–368. https://doi.org/10.1177/02655322251348685

32.

Jeon

E. H.

In’nami

Koizumi

(2022). L2 speaking and its external correlates: A meta-analysis. In Jeon

E. H.

In’nami

(Eds.), Understanding L2 proficiency: Theoretical and meta-analytic investigations (pp. 339–367). John Benjamins.

33.

Kane

(2004). Certification testing as an illustration of argument-based validation. Measurement: Interdisciplinary Research and Perspectives, 2(3), 135–170. https://doi.org/10.1207/s15366359mea0203_1

34.

Koizumi

(2021). The telephone standard speaking test: An outside evaluator’s investigation of a rebuttal to the generalization inference. In Chapelle

C. A.

Voss

(Eds.), Validity argument in language testing: Case studies of validation research (pp. 154–175). Cambridge University Press.

35.

Kurata

Saeki

Fujie

Matsuyama

(2023). Multimodal turn-taking model using visual cues for end-of-utterance prediction in spoken dialogue systems. Proceedings of Interspeech, 2023, 2658–2662. https://doi.org/10.21437/Interspeech.2023-578

36.

Liu

(2007). Language anxiety in EFL testing situations. ITL–International Journal of Applied Linguistics, 153(1), 53–75. https://doi.org/10.2143/ITL.153.0.2022821

37.

MacIntyre

P. D.

(1999). Language anxiety: A review of the research for language teachers. In Young

D. J.

(Ed.), Affect in foreign language and second language learning (pp. 24–45). McGraw-Hill.

38.

MacIntyre

P. D.

(2012). The idiodynamic method: A closer look at the dynamics of communication traits. Communication Research Reports, 29(4), 361–367. https://doi.org/10.1080/08824096.2012.723274

39.

MacIntyre

P. D.

(2017). An overview of language anxiety research and trends in its development. In Gkonou

Daubney

Dewaele

J.-M.

(Eds.), New insights into language anxiety: Theory, research and educational implications (pp. 11–30). Multilingual Matters.

40.

MacIntyre

P. D.

Ducker

(2022). The idiodynamic method: A practical guide for researchers. Research Methods in Applied Linguistics, 1(2), 100007. https://doi.org/10.1016/j.rmal.2022.100007

41.

MacIntyre

P. D.

Gregersen

(2022). The idiodynamic method: Willingness to communicate and anxiety processes interacting in real time. International Review of Applied Linguistics in Language Teaching, 60(1), 67–84. https://doi.org/10.1515/iral-2021-0024

42.

MacIntyre

P. D.

Legatto

J. J.

(2011). A dynamic system approach to willingness to communicate: Developing an idiodynamic method to capture rapidly changing affect. Applied Linguistics, 32(2), 149–171. https://doi.org/10.1093/applin/amq037

43.

MacIntyre

P. D.

McGillivray

M. F.

(2023). The inner workings of anxiety in second language learning. Annual Review of Applied Linguistics, 43, 88–104. https://doi.org/10.1017/S0267190523000065

44.

MacIntyre

P. D.

Noels

K. A.

Clément

(1997). Biases in self-ratings of second language proficiency: The role of language anxiety. Language Learning, 47(2), 265–287. https://doi.org/10.1111/0023-8333.81997008

45.

MacIntyre

P. D.

Wang

(2021). Willingness to communicate in the L2 about meaningful photos: Application of the pyramid model of WTC. Language Teaching Research, 25(6), 878–898. https://doi.org/10.1177/13621688211004645

46.

MacIntyre

P. D.

Wang

(2022). Anxiety. In Li

Hiver

Papi

(Eds.), The Routledge handbook of second language acquisition and individual differences (pp. 175–189). Routledge.

47.

Malterud

Siersma

V. D.

Guassora

A. D.

(2016). Sample size in qualitative interview studies: Guided by information power. Qualitative Health Research, 26(13), 1753–1760. https://doi.org/10.1177/1049732315617444

48.

Matsuyama

Suzuki

Saeki

Takatsu

Matsuura

Arai

(2023, June 8). Towards an explainable automated scoring of spoken interaction with a conversational AI agent. In Jang

Sawaki

(Convenors), Technology in diagnostic and formative language assessment: Friend or foe? [Colloquium]. Language Testing Research Colloquium.

49.

Messick

(1995). Validity of psychological assessment. Validation of inferences from persons’ responses and performance as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

50.

Ockey

Chukharev-Hudilainen

(2021). Human versus computer partner in the paired oral discussion test. Applied Linguistics, 42(5), 924–944. https://doi.org/10.1093/applin/amaa067

51.

O’Sullivan

Green

(2011). Test taker characteristics. In Taylor

(Ed.), Examining speaking: Research and practice in assessing second language speaking (pp. 36–64). Cambridge University Press.

52.

Polat

Kim

(2014). Dynamics of complexity and accuracy: A longitudinal case study of advanced untutored development. Applied Linguistics, 35(2), 184–207. https://doi.org/10.1093/applin/amt013

53.

Saeki

Takatsu

Kurata

Suzuki

Eguchi

Matsuura

Takizawa

Yoshikawa

Matsuyama

(2024). InteLLA: Intelligent language learning assistant for assessing language proficiency through interviews and roleplays. Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue. https://aclanthology.org/2024.sigdial-1.34

54.

Saghafi

Elahi Shirvan

(2020). Rapid changes in foreign language learning anxiety caused by a multiplicity of topics: An idiodynamic approach. Journal of Language and Education, 6(1), 83–102. https://doi.org/10.17323/jle.2020.9684

55.

Sawaki

Arai

Eguchi

Suzuki

Matsuyama

(2024, July 4). Building an argument for test score interpretation and use for a fully automated online assessment of L2 spoken interaction [Conference presentation]. Language Testing Research Colloquium, Innsbruck, Austria.

56.

Scovel

(1978). The effect of affect: A review of the anxiety literature. Language Learning, 28(1), 129–142. https://doi.org/10.1111/j.1467-1770.1978.tb00309.x

57.

Skehan

(2018). Second language task-based performance: Theory, research, and assessment. Routledge.

58.

Storbeck

Clore

G. L.

(2007). On the interdependence of cognition and emotion. Cognition and Emotion, 21(6), 1212–1237. https://doi.org/10.1080/02699930701438020

59.

Swain

(1985). Large-scale communicative language testing: A case study. In Lee

Y. P. A.

Fok

C. Y. Y.

Lord

Low

(Eds.), New directions in language testing (pp. 35–46). Pergamon.

60.

Takatsu

Suzuki

Eguchi

Matsuura

Saeki

Matsuyama

(2026). Gnowsis: Multimodal multitask learning for oral proficiency assessments. Computer Speech & Language, 95, 101860. https://doi.org/10.1016/j.csl.2025.101860

61.

Terry

Hayfield

(2021). Essentials of thematic analysis. American Psychological Association.

62.

Timpe-Laughlin

Sydorenko

Dombi

(2024). Human versus machine: Investigating L2 learner output in face-to-face versus fully automated role-plays. Computer Assisted Language Learning, 37(1–2), 149–178. https://doi.org/10.1080/09588221.2022.2032184

63.

Toulmin

S. E.

(2003). The uses of argument. Cambridge University Press.

64.

Voss

(2024). Language assessment and artificial intelligence. In Kunnan

A. J.

(Ed.), The concise companion of language assessment (pp. 112–125). Wiley.

65.

Woodrow

(2006). Anxiety and speaking English as a second language. RELC Journal, 37(3), 308–328. https://doi.org/10.1177/0033688206071315

66.

Zhai

Wibowo

(2023). A systematic review on artificial intelligence dialogue systems for enhancing English as foreign language students’ interactional competence in the university. Computers and Education: Artificial Intelligence, 4, 100134. https://doi.org/10.1016/j.caeai.2023.100134

67.

Zhang

Meng

(2024). Artificial intelligence in EFL speaking: Impact on enjoyment, anxiety, and willingness to communicate. System, 121, 103259. https://doi.org/10.1016/j.system.2024.103259

68.

Zhang

Liu

(2013). Evaluating the impact of oral test anxiety and speaking strategy use on oral English performance. The Journal of Asia TEFL, 10(2), 115–148. https://www.asiatefl.org/main/download_pdf.php?i=71&c=1419297878&fn=10_2_05.pdf