Explainable Artificial Intelligence (XAI) refers to systems that make AI models more transparent, helping users understand how outputs are generated. XAI algorithms are considered valuable in educational research, supporting outcomes like student success, trust, and motivation. Their potential to enhance transparency and reliability in online education systems is particularly emphasized. This study systematically analyzed educational research using XAI systems from 2019 to 2024, following the PICOS framework, and reviewed 35 studies. Methods like SHapley Additive exPlanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME), used in these studies, explain model decisions, enabling users to better understand AI models. This transparency is believed to increase trust in AI-based tools, facilitating their adoption by teachers and students.
AdadiA.BerradaM. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138–52160. https://doi.org/10.1109/access.2018.2870052
2.
AfzaalM.NouriJ.ZiaA.PapapetrouP.ForsU.WuY.LiX.WeegarR. (2021). Explainable AI for data-driven feedback and intelligent action recommendations to support students self-regulation. Frontiers in artificial intelligence, 4. https://doi.org/10.3389/frai.2021.723447
3.
AlonsoJ. M. (2020). Teaching explainable artificial intelligence to high school students. International Journal of Computational Intelligence Systems, 13(1), 974. https://doi.org/10.2991/ijcis.d.200715.003
4.
AlrajhiL.PereiraF. D.CristeaA. I.AljohaniT. (2022). A good classifier is not enough: A XAI approach for urgent instructor-intervention models in MOOCs. In Lecture notes in computer science (pp. 424–427). Springer. https://doi.org/10.1007/978-3-031-11647-6_84
5.
AlwarthanS.AslamN.KhanI. U. (2022). An explainable model for identifying at-risk student at higher education. IEEE Access, 10, 107649–107668. https://doi.org/10.1109/access.2022.3211070
6.
Al-ZyoudH. M. M. (2020). The role of artificial intelligence in teacher professional development. Universal journal of educational research, 8(11B), 6263–6272. https://doi.org/10.13189/ujer.2020.082265
BoulangerD.KumarV. (2020). SHAPed Automated Essay Scoring: Explaining Writing Features’ Contributions to English Writing Organization. In Lecture notes in computer science (pp. 68–78). Springer. https://doi.org/10.1007/978-3-030-49663-0_10
11.
BulathwelaS.Pérez-OrtizM.HollowayC.CukurovaM.Shawe-TaylorJ. (2024). Artificial intelligence alone will not democratise education: On educational inequality, techno-solutionism and inclusive tools. Sustainability, 16(2), 781. https://doi.org/10.3390/su16020781
12.
ČernevičienėJ.KabašinskasA. (2024). Explainable artificial intelligence (XAI) in finance: A systematic literature review. Artificial Intelligence Review, 57(8), 11. https://doi.org/10.1007/s10462-024-10854-8
13.
ChenX.ZouD.ChengG.XieH. (2023). Deep neural networks for the automatic understanding of the semantic content of online course reviews. Education and Information Technologies, 29(4), 3953–3991. https://doi.org/10.1007/s10639-023-11980-6
14.
ClanceyW. J.HoffmanR. R. (2021). Methods and standards for research on explainable artificial intelligence: Lessons from intelligent tutoring systems. Applied AI letters, 2(4). https://doi.org/10.1002/ail2.53
15.
ClementT.KemmerzellN.AbdelaalM.AmbergM. (2023). Xair: A systematic metareview of explainable AI (XAI) aligned to the software development process. Machine learning and knowledge extraction, 5(1), 78–108. https://doi.org/10.3390/make5010006
16.
CohauszL. (2022). Towards real interpretability of student success prediction combining methods of XAI and social science. https://eric.ed.gov/?id=ED624149
17.
ConijnR.KahrP.SnijdersC. (2023). The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation. Journal of Learning Analytics, 10(1), 37–53. https://doi.org/10.18608/jla.2023.7801
DehoO. B.JoksimovicS.LiJ.ZhanC.LiuJ.LiuL. (2023). Should Learning Analytics Models Include Sensitive Attributes? Explaining the Why. IEEE Transactions on Learning Technologies, 16(4), 560–572. https://doi.org/10.1109/tlt.2022.3226474
20.
DuanX.PeiB.AmbroseG. A.HershkovitzA.ChengY.WangC. (2024). Towards transparent and trustworthy prediction of student learning achievement by including instructors as co-designers: a case study. Education and Information Technologies, 29(3), 3075–3096. https://doi.org/10.1007/s10639-023-11954-8
21.
EricksonJ. A.BotelhoA. F.McAteerS.VaratharajA.HeffernanN. T. (2020). The automated grading of student open responses in mathematics. https://doi.org/10.1145/3375462.3375523
22.
FarrowR. (2023). The possibilities and limits of XAI in education: A socio-technical perspective. Learning, Media and Technology, 48(2), 266–279. https://doi.org/10.1080/17439884.2023.2185630
23.
Feldman-MaggorY.NazaretskyT.AlexandronG. (2024). Explainable AI for unsupervised machine learning: A proposed scheme applied to a case study with science teachers. In 16th International conference on computer supported education. Angers, France. May, 2024. https://doi.org/10.5220/0012687000003693
24.
FlanaganB.LiangC.MajumdarR.OgataH. (2022). Towards explainable group formation by knowledge map based genetic algorithm. 2022 International Conference on Advanced Learning Technologies (ICALT). https://doi.org/10.1109/icalt52272.2021.00119
25.
FlanaganB.TakamiK.TakiiK.DaiY.MajumdarR.OgataH. (2021). EXAIT: A symbiotic explanation education system. In RodrigoM. M. T., et al. (Eds.), Proceedings of the 29th International Conference on Computers in Education(pp. 404–409). Asia-Pacific Society for Computers in Education.
26.
GaoY.GuS.JiangJ.HongS. R.YuD.ZhaoL. (2024). Going beyond XAI: A systematic survey for explanation-guided learning. ACM Computing Surveys, 56(7), 1–39. https://doi.org/10.1145/3644073
27.
GhaiB.LiaoQ. V.ZhangY.BellamyR.MuellerK. (2021). Explainable active learning (XAL). Proceedings of the ACM on human-computer interaction, 4(CSCW3), 1–28. https://doi.org/10.1145/3432934
28.
GhoshS.KamalM. S.ChowdhuryL.NeogiB.DeyN.SherrattR. S. (2024). Explainable AI to understand study interest of engineering students. Education and Information Technologies, 29(4), 4657–4672. https://doi.org/10.1007/s10639-023-11943-x
29.
GinB. C.CateO. T.O’SullivanP. S.HauerK. E.BoscardinC. (2022). Exploring how feedback reflects entrustment decisions using artificial intelligence. Medical Education, 56(3), 303–311. https://doi.org/10.1111/medu.14696
30.
GligoreaI.CiocaM.OanceaR.GorskiA.-T.GorskiH.TudoracheP. (2023). Adaptive learning using artificial intelligence in e-learning: A literature review. Education Sciences, 13(12), 1216. https://doi.org/10.3390/educsci13121216
31.
GuleriaP.SoodM. (2022). Explainable AI and machine learning: Performance evaluation and explainability of classifiers on educational data mining inspired career counseling. Education and Information Technologies, 28(1), 1081–1116. https://doi.org/10.1007/s10639-022-11221-2
32.
HaT.KimS. (2023). Improving trust in AI with mitigating confirmation bias: Effects of explanation type and debiasing strategy for decision-making with explainable AI. International journal of human–computer interaction, 1–12. https://doi.org/10.1080/10447318.2023.2285640
33.
HuangA. Y.LuO. H.YangS. J. (2023). Effects of artificial intelligence enabled personalized recommendations on learners’ learning engagement, motivation, and outcomes in a flipped classroom. Computers & Education, 194, 104684. https://doi.org/10.1016/j.compedu.2022.104684
34.
HurP.LeeH.BhatS.BoschN. (2022). Using machine learning explainability methods to personalize interventions for students. International Educational Data Mining Society. https://doi.org/10.5281/zenodo.6853181
IslamM. R.AhmedM. U.BaruaS.BegumS. (2022). A systematic review of explainable artificial intelligence in terms of different application domains and tasks. Applied sciences, 12(3), 1353. https://doi.org/10.3390/app12031353
37.
JangY.ChoiS.JungH.KimH. (2022). Practical early prediction of students’ performance using machine learning and eXplainable AI. Education and Information Technologies, 27(9), 12855–12889. https://doi.org/10.1007/s10639-022-11120-6
38.
KarS. P.DasA. K.ChatterjeeR.MandalJ. K. (2023). Assessment of learning parameters for students’ adaptability in online education using machine learning and explainable AI. Education and Information Technologies, 29(6), 7553–7568. https://doi.org/10.1007/s10639-023-12111-x
39.
KarpouzisK. (2024). Explainable AI for intelligent tutoring systems. In FarmanbarM.TzamtziM.VermaA.K.ChakravortyA. (Eds.), Frontiers of Artificial Intelligence, Ethics, and Multidisciplinary Applications. FAIEMA 2023. Frontiers of Artificial Intelligence, Ethics and Multidisciplinary Applications. Singapore: Springer. https://doi.org/10.1007/978-981-99-9836-4_6
40.
KhosraviH.DemartiniG.SadiqS.GasevicD. (2021). Charting the design and analytics agenda of learnersourcing systems. LAK21: 11th international learning analytics and knowledge conference, Association for Computing Machinery, New York, NY, USA (2021), pp. 32-42, https://doi.org/10.1145/3448139.3448143
41.
KhosraviH.ShumS. B.ChenG.ConatiC.TsaiY. S.KayJ.KnightS.Martinez-MaldonadoR.SadiqS.GaševićD. (2022). Explainable artificial intelligence in education. Computers and education: Artificial Intelligence, 3, 100074. https://doi.org/10.1016/j.caeai.2022.100074
42.
KohamaH.BanY.HirakawaT.YamashitaT.FujiyoshiH.ItaiA.UsamiH. (2023). Recommending learning actions using neural network. Proceedings of the 31st International Conference on Computers in Education. Asia-Pacific Society for Computers in Education.https://doi.org/10.58459/icce.2023.946
43.
KumarR. (2023). Faculty members’ use of artificial intelligence to grade student papers: A case of implications. International journal for educational integrity, 19(1). https://doi.org/10.1007/s40979-023-00130-7
44.
KumarV.BoulangerD. (2020). Explainable automated essay scoring: Deep learning really has pedagogical value. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.572367
45.
LeeC.-A.TzengJ.-W.HuangN. F.SuY. S. (2021). Prediction of student performance in massive open online courses using deep learning system based on learning behaviors. Educational Technology & Society, 24(3), 130–146. https://www.jstor.org/stable/27032861
46.
MajumdarR.TakamiK.OgataH. (2023). Learning with explainable AI-recommendations at school: Extracting patterns of self-directed learning from learning logs. 2023 IEEE International Conference on Advanced Learning Technologies (ICALT),Orem, UT, USA, pp. 245-249. https://doi.org/10.1109/icalt58122.2023.00078
47.
MakhambetovaA.ZhiyenbayevaN.ErgeshevaE. (2021). Personalized learning strategy as a tool to improve academic performance and motivation of students. International Journal of Web-Based Learning and Teaching Technologies, 16(6), 1–17. https://doi.org/10.4018/ijwltt.286743
48.
MeloE.SilvaI.CostaD. G.ViegasC. M. D.BarrosT. M. (2022). On the Use of eXplainable Artificial Intelligence to Evaluate School Dropout. Education Sciences, 12(12), 845. https://doi.org/10.3390/educsci12120845
49.
MgonjaT. (2024). Using interpretable machine learning approaches to predict and provide explanations for student completion of remedial mathematics. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12647-6
50.
MinhD.WangH. X.LiY. F.NguyenT. N. (2021). Explainable artificial intelligence: A comprehensive review. Artificial Intelligence Review, 55(5), 3503–3568. https://doi.org/10.1007/s10462-021-10088-y
51.
NaenM. F.AdnanM. H. M.YaziN. A.NeeC. K. (2021). Development of attendance monitoring system with artificial intelligence optimization in cloud. International Journal of Artificial Intelligence, 8(2), 88–98. https://doi.org/10.36079/lamintang.ijai-0802.315
52.
NagyM.MolontayR. (2023). Interpretable dropout prediction: Towards XAI-based personalized intervention. International Journal of Artificial Intelligence in Education, 34(1), 202. https://doi.org/10.1007/s40593-023-00331-8
53.
NazaretskyT.BarC.WalterM.AlexandronG. (2022). Empowering teachers with AI: Co-designing a learning analytics tool for personalized instruction in the science classroom. In LAK22: 12th International Learning Analytics and Knowledge Conference.https://doi.org/10.1145/3506860.3506861
54.
NewmanM.GoughD. (2019). Systematic reviews in educational research: Methodology, perspectives and application (pp. 3–22). Springer eBooks. https://doi.org/10.1007/978-3-658-27602-7_1
55.
NiemeijerK.FeskensR.KremplG.KoopsJ.BrinkhuisM. J. S. (2020). Constructing and predicting school advice for academic achievement. In LAK ’20: 10th International Conference on Learning Analytics and Knowledge.https://doi.org/10.1145/3375462.3375486
56.
PadmasiriP.KasthuriarachchiS. (2024). Interpretable prediction of student dropout using explainable AI models. In 2024 International Research Conference on Smart Computing and Systems Engineering (SCSE), Colombo, Sri Lanka, 2024, pp. 1-7, https://doi.org/10.1109/SCSE61872.2024.10550525
57.
PageM. J.McKenzieJ. E.BossuytP. M.BoutronI.HoffmannT. C.MulrowC. D.ShamseerL.TetzlaffJ. M.AklE. A.BrennanS. E.ChouR.GlanvilleJ.GrimshawJ. M.HróbjartssonA.LaluM. M.LiT.LoderE. W.Mayo-WilsonE.McDonaldS.MoherD. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, n71. https://doi.org/10.1136/bmj.n71
58.
PlumleyR. D.BernackiM. L.GreeneJ. A.KuhlmannS.RakovićM.UrbanC. J.HoganK. A.LeeC.PanterA. T.GatesK. M. (2024). Co‐designing enduring learning analytics prediction and support tools in undergraduate biology courses. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13472
59.
RajabiE.EtminaniK. (2022). Knowledge-graph-based explainable AI: A systematic review. Journal of information science, 50(4), 1019–1029. https://doi.org/10.1177/01655515221112844
60.
RakovićM.LiY.FoumaniN. M.SalehiM.KuhlmannL.MackellarG.Martinez-MaldonadoR.HaffariG.SwieckiZ.LiX.ChenG.GaševićD. (2024). Measuring Affective and Motivational States as Conditions for Cognitive and Metacognitive Processing in Self-Regulated Learning. https://doi.org/10.1145/3636555.3636934
61.
SaksK.LeijenÄ. (2014). Distinguishing self-directed and self-regulated learning and measuring them in the e-learning context. Procedia - Social and Behavioral Sciences, 112, 190–198. https://doi.org/10.1016/j.sbspro.2014.01.1155
62.
SaranyaA.SubhashiniR. (2023). A systematic review of Explainable Artificial Intelligence models and applications: Recent developments and future trends. Decision analytics journal, 7, 100230. https://doi.org/10.1016/j.dajour.2023.100230
63.
ScheersH.De LaetT. (2021). Interactive and Explainable Advising Dashboard Opens the Black Box of Student Success Prediction. In Lecture notes in computer science (pp. 52–66). Springer. https://doi.org/10.1007/978-3-030-86436-1_5
SpeithT. (2022). A review of taxonomies of explainable artificial intelligence (XAI) methods. In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22), June 21–24, 2022, Seoul, Republic of Korea. New York, NY, USA: ACM. https://doi.org/10.1145/3531146.3534639
66.
SwamyV.DuS.MarrasM.KaserT. (2023). Trusting the explainers: Teacher validation of explainable artificial intelligence for course design. https://doi.org/10.1145/3576050.3576147
67.
SzymanskiM.OogeJ.De CroonR.Vanden AbeeleV.VerbertK. (2024). Feedback, control, or explanations? Supporting teachers with steerable distractor-generating AI. In The 14th Learning Analytics and Knowledge Conference(LAK ’24), March 18–22, 2024, Kyoto, Japan, New York, NY, USA. ACM. https://doi.org/10.1145/3636555.3636933
68.
TangL.SuY. S. (2024). Ethical implications and principles of using artificial intelligence models in the classroom: A systematic literature review. International journal of interactive multimedia and artificial intelligence, 1. In press. https://doi.org/10.9781/ijimai.2024.02.010
69.
TulliS. (2020). Explainability in Autonomous Pedagogical Agents. Proceedings of the AAAI Conference on Artificial Intelligence, 34(10), 13738–13739. https://doi.org/10.1609/aaai.v34i10.7141
70.
UnalZ.UnalA. (2024). Enhancing preservice teacher education through ai tools: A comparative study in classroom management & course satisfaction. In CohenJ.SolanoG. (Eds.), Proceedings of society for information technology & teacher education international conference (pp. 892-897). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/224057/
WangY. (2023). Artificial Intelligence Technologies in College English Translation Teaching. Journal of Psycholinguistic Research, 52(5), 1525–1544. https://doi.org/10.1007/s10936-023-09960-5
73.
XuW.MengJ.RajaS. K. S.PriyaM. P.DeviM. K. (2021). Artificial intelligence in constructing personalized and accurate feedback systems for students. International journal of modeling, simulation and scientific computing, 14(1). https://doi.org/10.1142/s1793962323410015
74.
YangS. J.OgataH.MatsuiT.ChenN. (2021). Human-centered artificial intelligence in education: Seeing the invisible through the visible. Computers and education: Artificial Intelligence, 2, 100008. https://doi.org/10.1016/j.caeai.2021.100008
75.
ZhaoY.WangY.DerrT. (2023). Fairness and explainability: Bridging the gap towards fair model explanations. Proceedings of the AAAI Conference on Artificial Intelligence, 37(9), 11363–11371. https://doi.org/10.1609/aaai.v37i9.26344
76.
ZhouY.YeX.LiuY. (2022). The influence of personalized learning intervention system on student learning a study of junior middle school. Interactive Technology and Smart Education, 19(4), 441–459. https://doi.org/10.1108/itse-10-2021-0192
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
