Restricted accessReview articleFirst published online 2026
Exploring the Impact of Generative AI-Powered Conversational Agents on Student Learning: A Systematic Review and Meta-Analysis Grounded in Activity Theory
Grounded in activity theory, this study conducted a systematic review and meta-analysis, following PRISMA guidelines, to evaluate the impact of generative AI-powered conversational agents on students’ learning outcomes in both cognitive and non-cognitive skills. Twenty-seven empirical studies published between 2022 and 2025 were analyzed. The results revealed that generative AI-powered conversational agents had moderately positive effects on both cognitive (g = 0.462, p < .001) and non-cognitive skills (g = 0.519, p < .001). Moderator analyses identified several contextual variables—educational levels, knowledge types, intervention durations, agent roles, and instructional modes—as significant or marginally significant. These findings offer practical implications for educators, developers, and researchers, aiming to enhance the effectiveness of generative AI-powered conversational agents in improving learning outcomes in educational settings.
Articles marked with an asterisk (*) were analyzed in the meta-analysis.
2.
AllisonJ. (2025). RAISE the standard: A framework for transparent reporting of artificial intelligence studies in education. Journal of Educational Computing Research, 64(1), 3–15. https://doi.org/10.1177/07356331251377430
3.
*AlneyadiS.WardatY. (2023). ChatGPT: Revolutionizing student achievement in the electronic magnetism unit for eleventh-grade students in Emirates schools. Contemporary Educational Technology, 15(4), ep448. https://doi.org/10.30935/cedtech/13417
4.
AndersonL. W.KrathwohlD. R. (Eds.), (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Complete ed.). Longman.
5.
Atman UsluN.YavuzG. Ö.Koçak UsluelY. (2023). A systematic review study on educational robotics and robots. Interactive Learning Environments, 31(9), 5874–5898. https://doi.org/10.1080/10494820.2021.2023890
6.
*BašićŽ.BanovacA.KružićI.JerkovićI. (2023). ChatGPT-3.5 as writing assistance in students’ essays. Humanities and Social Sciences Communications, 10(1), 750. https://doi.org/10.1057/s41599-023-02269-7
7.
BaylorA.KimY. (2005). Simulating instructional roles through pedagogical agents. International Journal of Artificial Intelligence in Education, 15(2), 95–115. https://doi.org/10.3233/IRG-2005-15
8.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta‐analysis. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470743386
9.
BozkurtA. (2023). Unleashing the potential of generative AI, conversational agents and chatbots in educational praxis: A systematic review and bibliometric analysis of GenAI in education. Open Praxis, 15(4), 261–270. https://doi.org/10.55982/openpraxis.15.4.609
10.
*ChenA.WeiY.LeH.ZhangY. (2024). Learning-by-teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2412.15226
*CubillosC.MelladoR.Cabrera-PaniaguaD.UrraE. (2025). Generative artificial intelligence in computer programming: Does it enhance learning, motivation, and the learning environment?IEEE Access, 13, 40438–40455. https://doi.org/10.1109/ACCESS.2025.3532883
13.
*DasariD.HendriyantoA.SaharaS.SuryadiD.MuhaiminL. H.ChaoT.FitrianaL. (2024). ChatGPT in didactical tetrahedron, does it make an exception? A case study in mathematics teaching and learning. Frontiers in Education, 8, Article 1295413. https://doi.org/10.3389/feduc.2023.1295413
14.
DavisR. O.ParkT.VincentJ. (2023). A meta-analytic review on embodied pedagogical agent design and testing formats. Journal of Educational Computing Research, 61(1), 30–67. https://doi.org/10.1177/07356331221100556
15.
de KockT. A. J. (2025). The impact of GenAI-disclosure on the evaluation of chatbot conversations: Its effects on user perception in successful conversations and conversational failures. Tilburg University. Tilburg University thesis repository (ARNO). [Master’s thesis]. https://arno.uvt.nl/show.cgi?fid=183049
16.
DeciE. L.RyanR. M. (2000). The ‘What’ and ‘Why’ of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268. https://doi.org/10.1207/S15327965PLI1104_01
17.
EggerM.SmithG. D.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
18.
*El FathiT.SaadA.LarhzilH.LamriD.Al IbrahmiE. M. (2025). Integrating generative AI into STEM education: Enhancing conceptual understanding, addressing misconceptions, and assessing student acceptance. Disciplinary and Interdisciplinary Science Education Research, 7(1), 6. https://doi.org/10.1186/s43031-025-00125-z
19.
EngeströmY. (2001). Expansive learning at work: Toward an activity theoretical reconceptualization. Journal of Education and Work, 14(1), 133–156. https://doi.org/10.1080/13639080020028747
20.
EngwallO.LopesJ. (2022). Interaction and collaboration in robot-assisted language learning for adults. Computer Assisted Language Learning, 35(5–6), 1273–1309. https://doi.org/10.1080/09588221.2020.1799821
21.
*EscalanteJ.PackA.BarrettA. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20(1), 57. https://doi.org/10.1186/s41239-023-00425-2
22.
*EsselH. B.VlachopoulosD.Tachie-MensonA.JohnsonE. E.BaahP. K. (2022). The impact of a virtual teaching assistant (chatbot) on students’ learning in Ghanaian higher education. International Journal of Educational Technology in Higher Education, 19(1), 57. https://doi.org/10.1186/s41239-022-00362-6
23.
*FanY.TangL.LeH.ShenK.TanS.ZhaoY.ShenY.LiX.GaševićD. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544
24.
*FarahJ. C.IngramS.SpaenlehauerB.LasneF. K.-L.GilletD. (2023). Prompting large language models to power educational chatbots. In XieH.LaiC.-L.ChenW.XuG.PopescuE. (Eds.), Advances in web-based learning – ICWL 2023 (pp. 169–188): Springer Nature. https://doi.org/10.1007/978-981-99-8385-8_14
25.
FengS. (2025). Group interaction patterns in generative AI ‐supported collaborative problem solving: Network analysis of the interactions among students and a GAI chatbot. British Journal of Educational Technology, 56(5), 2125–2145. https://doi.org/10.1111/bjet.13611
26.
*Gutiérrez-FerréA.Hernández-LeoD.Sánchez-ReinaJ. R. (2024). Generative AI chatbot in PyramidApp: Students’ behaviors and design principles. In SantosP.ÁlvarezC.Hernández-LeoD.KobayashiM.ZuritaG. (Eds.), Collaboration technologies and social computing (14890, pp. 248–256): Springer. https://doi.org/10.1007/978-3-031-67998-8_19
27.
HigginsJ. P. T.ThompsonS. G.DeeksJ. J.AltmanD. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557
28.
HigginsK.Huscroft-D’AngeloJ.CrawfordL. (2019). Effects of technology in mathematics on achievement, motivation, and attitude: A meta-analysis. Journal of Educational Computing Research, 57(2), 283–319. https://doi.org/10.1177/0735633117748416
*KhuibutW.PremthaisongS.ChaipidechP. (2023). Integrating ChatGPT into synectics model to improve high school student’s creative writing skill. In proceedings of ICCE2023 (pp. 44–51). APSCE. https://library.apsce.net/index.php/ICCE/article/view/4737
31.
*KhuibutW.PremthaisongS.ChaipidechP. (2024). Integrating ChatGPT into flipped learning: Enhancing students’ creative writing skills and perception. International Conference on Computersin Education. https://doi.org/10.58459/icce.2024.4967
32.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
33.
LaunM.WolffF. (2025). Chatbots in education: Hype or help? A meta-analysis. Learning and Individual Differences, 119, Article 102646. https://doi.org/10.1016/j.lindif.2025.102646
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Sage Publications, Inc.
36.
*MageiraK.PittouD.PapasalourosA.KotisK.ZangogianniP.DaradoumisA. (2022). Educational AI chatbots for content and language integrated learning. Applied Sciences, 12(7), 3239. https://doi.org/10.3390/app12073239
37.
MerrillM. D. (2013). First principles of instruction: Identifying and designing effective, efficient, and engaging instruction. Pfeiffer.
MohammedI. A.BelloA.AyubaB. (2025). Effect of large language models artificial intelligence ChatGPT chatbot on achievement of computer education students. Education and Information Technologies, 30(9), 11863–11888. https://doi.org/10.1007/s10639-02413293-8
40.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G.The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-Analyses: The PRISMA statement. PLoS Medicine, 6(7), Article e1000097. https://doi.org/10.1371/journal.pmed.1000097
41.
OuyangF.XuW. (2024). The effects of educational robotics in STEM education: A multilevel meta-analysis. International Journal of STEM Education, 11(1), 7. https://doi.org/10.1186/s40594-024-00469-4
42.
ParkM. S.UpamaP. B.AnikA. A.AhamedS. I.LuoJ.TianS.RabbaniM.OhH. (2023). A Survey of conversational agents and their applications for self-management of chronic conditions. In Proceedings: Annual International Computer Software and Applications Conference. COMPSAC, 2023 (pp. 1064–1075). https://doi.org/10.1109/COMPSAC57700.2023.00162
43.
ParkS. J.HanJ. H.KangB. H.ShinK. C. (2011). Teaching assistant robot, ROBOSEM, in English class and practical issues for its diffusion. In Advanced robotics and its social impacts (pp. 8–11). https://doi.org/10.1109/ARSO.2011.6301971
44.
*QureshiB. (2023). Exploring the use of ChatGPT as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges [Preprint]. arXiv. https://doi.org/10.48550/ARXIV.2304.11214
45.
RandallN. (2020). A survey of robot-assisted language learning (RALL). ACM Transactions on Human-Robot Interaction, 9(1), 1–36. https://doi.org/10.1145/3345506
46.
RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), (2005). Publication bias in meta‐analysis: Prevention, assessment and adjustments (1st ed.). Wiley. https://doi.org/10.1002/0470870168
47.
SivulaM. (2011). Hybrid graduate education: Assessing student comfort with technology interventions. Ubiquitous Learning: An International Journal, 3(1), 35–42. https://doi.org/10.18848/1835-9795/CGP/v03i01/40485
48.
*SongC.SongY. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, Article 1260843. https://doi.org/10.3389/fpsyg.2023.1260843
49.
*SunD.BoudouaiaA.ZhuC.LiY. (2024). Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study. International Journal of Educational Technology in Higher Education, 21(1), 14. https://doi.org/10.1186/s41239-024-00446-5
50.
SwellerJ.Van MerriënboerJ. J. G.PaasF. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261–292. https://doi.org/10.1007/s10648-019-09465-5
51.
TiptonE. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. https://doi.org/10.1037/met0000011
52.
TliliA.LinV.ChenN.-S.HuangR. (2020). A systematic review on robot-assisted special education from the activity theory perspective. Educational Technology & Society, 23(3), 95–109. https://doi.org/10.30191/ETS.202007_23(3).0008
53.
TuC.-H.McIsaacM. (2002). The relationship of social presence and interaction in online classes. American Journal of Distance Education, 16(3), 131–150. https://doi.org/10.1207/S15389286AJDE1603_2
54.
*UrbanM.DěchtěrenkoF.LukavskýJ.HrabalováV.SvachaF.BromC.UrbanK. (2024). ChatGPT improves creative problem-solving performance in university students: An experimental study. Computers & Education, 215(1), Article 105031. https://doi.org/10.1016/j.compedu.2024.105031
55.
VittadiniG.SturaroC.FolloniG. (2022). Non-cognitive skills and cognitive skills to measure school efficiency. Socio-Economic Planning Sciences, 81(4), Article 101058. https://doi.org/10.1016/j.seps.2021.101058
56.
VygotskyL. S. (1980). In ColeM.Jolm-SteinerV.ScribnerS.SoubermanE. (Eds.), Mind in society: Development of higher psychological processes. Harvard University Press. https://doi.org/10.2307/j.ctvjf9vz4
57.
WangD. (2024). Teacher- versus AI-Generated (Poe application) corrective feedback and language learners’ writing anxiety, complexity, fluency, and accuracy. The International Review of Research in Open and Distributed Learning, 25(3), 37–56. https://doi.org/10.19173/irrodl.v25i3.7646
58.
WangY.GongS.CaoY.LangY.XuX. (2023). The effects of affective pedagogical agent in multimedia learning environments: A meta-analysis. Educational Research Review, 38(Article), 100506. https://doi.org/10.1016/j.edurev.2022.100506
59.
*WangZ.MaY.YangX.LiK. (2023). The impact of ChatGPT-type reading platforms on graduate students’ academic reading ability [in Chinese]. Open Education Research, 29(6), 60–68. https://doi.org/10.13966/j.cnki.kfjyyj.2023.06.007
60.
*WangH.WangC.ChenZ.LiuF.BaoC.XuX. (2025). Impact of AI-agent-supported collaborative learning on the learning outcomes of University programming courses. Education and Information Technologies, 30(12), 17717–17749. https://doi.org/10.1007/s10639-025-13487-8
61.
*WardatY.AlneyadiS. (2024). Integrating ChatGPT in grade 12 quantum theory education: An exploratory study at emirate school (UAE). International Journal of Information and Education Technology, 14(3), 389. https://doi.org/10.18178/ijiet.2024.14.3.2061
62.
*WeiX.WangL.LeeL.-K.LiuR. (2025). Multiple generative AI pedagogical agents in augmented reality environments: A study on implementing the 5E model in science education. Journal of Educational Computing Research, 63(2), 336–371. https://doi.org/10.1177/07356331241305519
63.
What Works Clearinghouse. (2022). What works Clearinghouse procedures and standards handbook, version 5.0 (version 5.0). U.S. Department of Education, Institute of Education Sciences. https://ies.ed.gov/ncee/wwc/Handbooks
64.
WuX.LiR. (2024). Effects of robot-assisted language learning on English-as-a-Foreign-Language skill development. Journal of Educational Computing Research, 62(4), 790–814. https://doi.org/10.1177/07356331231226171
65.
*WuC.HuS.JiangY.MouY.PiZ. (2024). GenAI-driven educational digital humans: Architecture design and an empirical study [in Chinese]. Modern Educational Technology, 34(9), 26–36. https://doi.org/10.3969/j.issn.1009-8097.2024.09.003
66.
*WuJ.ZhouW.CaoC. (2024). An Empirical Study on Empowering Oral Teaching with AIGC [in Chinese]. China Educational Technology, (4),105–111.
67.
*YilmazR.Karaoglan YilmazF. G. (2023). The effect of generative artificial intelligence (AI)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Computers and Education: Artificial Intelligence, 4(2), Article 100147. https://doi.org/10.1016/j.caeai.2023.100147
68.
YusufH.MoneyA.Daylamani-ZadD. (2025). Pedagogical AI conversational agents in higher education: A conceptual framework and survey of the state of the art. Educational Technology Research and Development, 73(2), 815–874. https://doi.org/10.1007/s11423-025-10447-4
69.
*ZhangZ.HuangX. (2024). The impact of chatbots based on large language models on second language vocabulary acquisition. Heliyon, 10(3), Article e25370. https://doi.org/10.1016/j.heliyon.2024.e25370
70.
ZhaoF.MayerR. E.Adamo-VillaniN.MousasC.ChoiM.HauserK. (2025). Role of race and gender of pedagogical agents in multimedia learning. Journal of Educational Computing Research, 63(4), 801–840. https://doi.org/10.1177/07356331251322469
71.
*ZhengS. (2024). The effects of chatbot use on foreign language reading anxiety and reading performance among Chinese secondary school students. Computers and Education: Artificial Intelligence, 7, Article 100271. https://doi.org/10.1016/j.caeai.2024.100271
72.
ZimmermanB. J. (2000). Attaining self-regulation: A social cognitive perspective. In BoekaertsM.PintrichP. R.ZeidnerM. (Eds.), Handbook of self-regulation (pp. 13–39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7
