Despite the increasing integration of generative Artificial Intelligence (GenAI) in education, a significant research gap exists in its comparative effectiveness against traditional instructor-led guidance in specialized humanities like philology. This quasi-experimental study compares an experimental group (EG, n = 26), using GenAI chatbots for conceptual understanding and GenAI platforms to create interactive materials from classical texts, with a control group (CG, n = 26) receiving standard instruction. This study investigates the impact of GenAI-assisted instructional design on students’ learning achievements and empowerment in using GenAI within real-world contexts. It specifically examines if there is a statistically significant difference (a) in philological content mastery—defined as understanding classical or modern texts/themes—and (b) in perceived empowerment in using GenAI (self-efficacy, impact, and meaningfulness). The findings indicate that the EG significantly outperformed the CG in mastering philological content, with a notably higher adjusted post-test mean score and a large effect size. The same group also reported substantially greater perceived empowerment in using GenAI within real-world contexts, demonstrating strong increases in self-efficacy, impact, and meaningfulness compared to their counterparts in the CG. Implications suggest integrating GenAI into philology curricula foster engagement, autonomy, and confidence in creative tasks, provided educators mitigate risks to analytical rigor through intentional design.
BabuM. A.YusufK. M.EniL. N.JamanS. M. S.SharminM. R. (2024). ChatGPT and generation ‘Z’: A study on the usage rates of ChatGPT. Social Sciences & Humanities Open, 10(1), Article 101163. https://doi.org/10.1016/j.ssaho.2024.101163
3.
Baidoo-AnuD.Owusu AnsahL. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52–66. https://doi.org/10.61969/jai.1337500
BereiterC.ScardamaliaM. (2003). Learning to work creatively with knowledge. In De CorteE.VerschaffelL.EntwistleN.van MerriënboerJ. (Eds.), Powerful learning environments: Unravelling basic components and dimensions (pp. 55–68). Pergamon.
6.
BertiS. (2018). Digital methods in classical philology: Image scanning, automated transcription, and network analysis of medieval codices. Routledge.
7.
BlackP.WiliamD. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 92(1), 81–90. (Note: Original publication was 1998, the 2010 Phi Delta Kappan reference is a reprint/reflection on the original.). https://doi.org/10.1177/003172171009200119
CampbellJ. L.QuincyC.OssermanJ.PedersenO. K. (2013). Coding in-depth semi-structured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research, 42(3), 294–320. https://doi.org/10.1177/004912411350047
11.
ChenA.ZhangY.JiaJ.LiangM.ChaY.LimC. P. (2025). A systematic review and meta-analysis of AI-enabled assessment in language learning: Design, implementation, and effectiveness. Journal of Computer Assisted Learning, 41(1), Article e13064. https://doi.org/10.1111/jcal.13064
12.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203771587
13.
ColeR. (2023). Inter-rater reliability methods in qualitative case study research. Sociological Methods & Research, 53(4), 1944–1975. https://doi.org/10.1177/00491241231156971
14.
CortinaJ. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98
15.
CreswellJ. W.PothC. N. (2023). Qualitative inquiry and research design: Choosing among five approaches (5th ed.). Sage Publications. (ISBN 978-1544398396).
16.
CurranP. J.WestS. G.FinchJ. F. (1996). The robustness of test statistics to non-normality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16. https://doi.org/10.1037/1082-989X.1.1.16
DengR.JiangM.YuX.LuY.LiuS. (2024). Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers & Education, 227, Article 105224. https://doi.org/10.1016/j.compedu.2024.105224
19.
EbelR. L.FrisbieD. A. (1991). Essentials of educational measurement (5th ed.). Prentice Hall.
20.
FarberS. (2025). Comparing human and AI expertise in the academic peer review process: Towards a hybrid approach. Higher Education Research and Development, 44(2), 1–15. https://doi.org/10.1080/07294360.2024.2445575
21.
FieldA. (2009). Discovering statistics using SPSS (3rd ed.). Sage Publications.
22.
FraserJ. (2025). Kafkaesque algorithms: Kafka’s writing in the age of artificial intelligence. Humanities, 14(1), 13. https://doi.org/10.3390/h14010013
23.
FredricksJ. A.BlumenfeldP. C.ParisA. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. https://doi.org/10.3102/00346543074001059
24.
FryerL.CarpenterR. (2006). Bots as language learning tools. Language, Learning and Technology, 10(3), 8–14. https://doi.org/10.64152/10125/44068
25.
GuanL.LiS.GuM. M. (2024). AI in informal digital English learning: A meta-analysis of its effectiveness on proficiency, motivation, and self-regulation. Computers and Education: Artificial Intelligence, 7, Article 100323. https://doi.org/10.1016/j.caeai.2024.100323
26.
Gültekin TalayhanÖ.BabayiğitM.ÖğrenciY. Z. Y. A. (2023). The influence of AI writing tools on the content and organization of students’ writing: A focus on EFL instructors’ perceptions. CUDES Current Debates in Social Sciences, 6(2), 83–93. https://doi.org/10.29228/cudes.71701
HuS. (2024). The effect of artificial intelligence-assisted personalized learning on student learning outcomes: A meta-analysis based on 31 empirical research papers. Science Insights Education Frontiers, 24(1), 3873–3894. https://doi.org/10.15354/sief.24.re395
29.
HurbanskaS.BotvynT.BurmakinaN.ShovkopliasY.HurbanskaA. (2022). On the problems of modern philology and the creative methodology of teaching foreign languages in the European educational system. Amazonia Investiga, 11(55), 124–131. https://doi.org/10.34069/AI/2022.55.07.13
30.
JänickeS.FranziniG.CheemaM. F.ScheuermannG. (2015). On close and distant reading in digital humanities: A survey and future challenges. In BorgoR.GanovelliF.ViolaI. (Eds.), Proceedings of the Eurographics Conference on Visualization (EuroVis) (pp. 83-103). The Eurographics Association. https://doi.org/10.2312/eurovisstar.20151113
31.
JonssonA.SvingbyG. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
32.
KasneciG.SeßlerK.KüchemannS.BannertM.DementievaD.FischerF.KasneciE. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274
33.
KazanidisI.PellasN. (2024). Harnessing generative artificial intelligence for digital literacy Innovation: A comparative study between early childhood education and computer science undergraduates. AI, 5(3), 1427–1445. https://doi.org/10.3390/ai5030068
34.
KhalilH.Al-MaawaliW.El-GhoolR. (2022). Impact of quiz-based interactive videos into personal learning environment on regulating E-Portfolio design and learning engagement: An experimental study. Universal Journal of Educational Research, 10(4), 281–288. https://doi.org/10.13189/ujer.2022.100403
35.
KimH.-Y. (2013). Statistical notes for clinical researchers: Assessing normal distribution (2) using skewness and kurtosis. Restorative Dentistry & Endodontics, 38(1), 52–54. https://doi.org/10.5395/rde.2013.38.1.52
36.
KohnkeL.ZouD.SuF. (2025). Exploring the potential of GenAI for personalized English teaching: Learners' experiences and perceptions. Computers and Education: Artificial Intelligence, 8, Article 100371. https://doi.org/10.1016/j.caeai.2025.100371
37.
KongS. C.ZhuJ.YangY. N. (2025). Developing and validating a scale of empowerment in using artificial intelligence for problem-solving for senior secondary and university students. Computers and Education Artificial Intelligence, 8, Article 100359. https://doi.org/10.1016/j.caeai.2024.100359
38.
KramschC. (2006). From communicative competence to symbolic competence. The Modern Language Journal, 90(2), 249–252. https://www.jstor.org/stable/3876875
39.
LabadzeL.GrigoliaM.MachaidzeL. (2023). Role of GenAI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1
40.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
41.
LavidasK.VoulgariI.PapadakisS.AthanassopoulosS.AnastasiouA.FilippidiA.KomisV.KaracapilidisN. (2024). Determinants of Humanities and Social Sciences students’ intentions to use artificial intelligence applications for academic purposes. Information, 15(6), 314. https://doi.org/10.3390/info15060314
42.
LeeG.KimH. (2023). Human vs. AI: The battle for authenticity in fashion design and consumer response. Journal of Retailing and Consumer Services, 77, 103690. https://doi.org/10.1016/j.jretconser.2023.103690
43.
LiuC.HouJ.TuY.WangY.HwangG. (2021). Incorporating a reflective thinking promoting mechanism into artificial intelligence-supported English writing environments. Interactive Learning Environments, 31(9), 5614–5632. https://doi.org/10.1080/10494820.2021.2012812
44.
LiuW.WangY. (2024). The effects of using AI tools on critical thinking in English literature classes among EFL learners: An intervention study. European Journal of Education, 59(4), 12804. https://doi.org/10.1111/ejed.12804
45.
LombardM.Snyder-DuchJ.BrackenC. C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28(4), 587–604. https://doi.org/10.1111/j.1468-2958.2002.tb00826.x
46.
MacQueenK. M.McLellanE.KayK.MilsteinB. (1998). Codebook development for team-based qualitative analysis. CAM Journal, 10(2), 31–36. https://doi.org/10.1177/1525822X980100020301
47.
McMillanJ. H.SchumacherS. (2010). Research in education: Evidence-based inquiry (7th ed.). Pearson.
48.
NazaretskyT.Mejia-DomenzainP.SwamyV.FrejJ.KäserT. (2025). The critical role of trust in adopting AI-Powered educational technology for learning: An instrument for measuring student perceptions. Computers and Education Artificial Intelligence, 8, Article 100368. https://doi.org/10.1016/j.caeai.2025.100368
49.
NicolD. J.Macfarlane-DickD. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090
50.
OcenS.ElasuJ.AarakitS. M.OlupotC. (2025). Artificial intelligence in higher education institutions: Review of innovations, opportunities, and challenges. Frontiers in Education, 10, 1530247. https://doi.org/10.3389/feduc.2025.1530247
51.
O’ConnorC.JoffeH. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19, Article 160940691989922. https://doi.org/10.1177/1609406919899220
52.
PellasN. (2023a). The influence of sociodemographic factors on students' attitudes toward AI-generated video content creation. Smart Learning Environments, 10(1), 57. https://doi.org/10.1186/s40561-023-00276-4
53.
PellasN. (2023b). The effects of generative AI platforms on undergraduates’ narrative intelligence and writing self-efficacy. Special Issue “Trustworthy artificial intelligence in education”. Education Sciences, 13(11), 1155. https://doi.org/10.3390/educsci13111155
54.
PellasN. (2025a). The role of students’ higher-order thinking skills in the relationship between academic achievements and machine learning using generative AI chatbots. In Research and practice in technology enhanced learning (20, p. 36). Springer. https://doi.org/10.58459/rptel.2025.20036
55.
PellasN. (2025b). The impact of AI-generated instructional videos on problem-based learning in science teacher education. Education Sciences, 15(1), 102. https://doi.org/10.3390/educsci15010102
56.
PellasN. (2025c). AI chatbots in philology: A user experience case study of conversational interfaces for content creation and instruction. Multimodal Technologies and Interaction, 9(7), 65. https://doi.org/10.3390/mti9070065
57.
RezigallaA. A.EleragiA. M. E. S. A.ElhusseinA. B.AlfaifiJ.AlghamdiM. A.Al AmeerA. Y.YahiaA. I. O.MohammedO. A.AdamM. I. E. (2024). Item analysis: The impact of distractor efficiency on the difficulty index and discrimination power of multiple-choice items. BMC Medical Education, 24(1), 445. https://doi.org/10.1186/s12909-024-05433-y
58.
SchleppegrellM. J. (2004). The language of schooling: A functional linguistics perspective. Lawrence Erlbaum Associates.
59.
SchreibmanS.SiemensR.UnsworthJ. (Eds.), (2021). A new companion to digital humanities (2nd ed.). Wiley-Blackwell.
60.
SchulzK.DeichslerF. (2024). November 16). SEFLAG: Systematic evaluation framework for NLP models and datasets in Latin and Ancient Greek. Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, (NLP4DH),247–258. https://doi.org/10.18653/v1/2024.nlp4dh-1.24
61.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference (2nd ed.). Houghton Mifflin.
62.
ShataA.HartleyK. (2025). Artificial intelligence and communication technologies in academia: Faculty perceptions and the adoption of generative AI. International Journal of Educational Technology in Higher Education, 22(1), 14. https://doi.org/10.1186/s41239-025-00511-7
63.
SongC.SongY. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, 1260843. https://doi.org/10.3389/fpsyg.2023.1260843
SongY.HuangL.ZhengL.FanM.LiuZ. (2025). Interactions with generative AI chatbots: Unveiling dialogic dynamics, students’ perceptions, and practical competencies in creative problem-solving. International Journal of Educational Technology in Higher Education, 22(1), 12. https://doi.org/10.1186/s41239-025-00508-2
66.
SvenssonP. (2011). The digital humanities as a humanities project. Arts and Humanities in Higher Education, 11(1–2), 42–60. https://doi.org/10.1177/1474022211427367
67.
WeiX.WangL.LeeL.LiuR. (2025). The effects of generative AI on collaborative problem-solving and team creativity performance in digital story creation: An experimental study. International Journal of Educational Technology in Higher Education, 22(1), 23. https://doi.org/10.1186/s41239-025-00526-0
68.
WiboolyasarinW.WiboolyasarinK.TiranantP.BoonyakitanontP.JinowatN. (2024). Designing chatbots in language classrooms: An empirical investigation from user learning experience. Smart Learning Environments, 11(1), 32. https://doi.org/10.1186/s40561-024-00319-4
69.
WineburgS. S. (1998). Reading abraham Lincoln: An expert/expert study in the interpretation of historical texts. Cognitive Science, 22(3), 319–346. https://doi.org/10.1016/S0364-0213(99)80043-3
70.
WuR.YuZ. (2023). Do GenAI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55(1), 10–33. https://doi.org/10.1111/bjet.13334
YeJ. (2022). The application of artificial intelligence technologies in digital humanities: Applying to dunhuang culture inheritance, development, and innovation. Journal of Computer Science and Technology Studies, 4(2), 31–38. https://doi.org/10.32996/jcsts.2022.4.2.5
73.
YetiŞensoyO.KaradumanH. (2024). The effect of AI-powered chatbots in social studies education. Education and Information Technologies, 29(13), 17035–17069. https://doi.org/10.1007/s10639-024-12485-6
74.
Zawacki-RichterO.MarínV. I.BondM.GouverneurF. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators?International Journal of Educational Technology in Higher Education, 16(1), 1–27. https://doi.org/10.1186/s41239-019-0171-0
75.
ZhangR.ZouD.ChengG. (2023). Chatbot-based training on logical fallacy in EFL argumentative writing. Innovation in Language Learning and Teaching, 17(5), 932–945. https://doi.org/10.1080/17501229.2023.2197417
76.
ZhengS. (2024). The effects of chatbot use on foreign language reading anxiety and reading performance among Chinese secondary school students. Computers and Education Artificial Intelligence, 7(2024), Article 100271. https://doi.org/10.1016/j.caeai.2024.100271
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
