Researchers and software users benefit from the rapid growth of artificial intelligence (AI) to an unprecedented extent in various domains where automated intelligent action is required. However, as they continue to engage with AI, they also begin to understand the limitations and risks associated with ceding control and decision-making to not always transparent artificial computer agents. Understanding of “what is happening in the black box” becomes feasible with explainable AI (XAI) methods designed to mitigate these risks and introduce trust into human-AI interactions. Our study reviews the essential capabilities, limitations, and desiderata of XAI tools developed over recent years and reviews the history of XAI and AI in education (AIED). We present different approaches to AI and XAI from the viewpoint of researchers focused on AIED in comparison with researchers focused on AI and machine learning (ML). We conclude that both groups of interest desire increased efforts to obtain improved XAI tools; however, these groups formulate different target user groups and expectations regarding XAI features and provide different examples of possible achievements. We summarize these viewpoints and provide guidelines for scientists looking to incorporate XAI into their own work.
DošilovićFKBrčićMHlupićN. Explainable artificial intelligence: A survey. In: 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), 2018, pp.210–215.
4.
HolzingerABiemannCPattichisCS, et al. What do we need to build explainable AI systems for the medical domain?arXiv e-prints. December2017; arXiv:1712.09923.
5.
SamekWWiegandTMüllerK-R. Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. arXiv e-prints. August2017; arXiv:1708.08296.
6.
RiberaMLapedrizaA. Can we do better explanations? A proposal of user-centered explainable AI. In: IUI Workshops, Marina del Ray, CA, 2019.
7.
Doshi-VelezFKimB. Towards a rigorous science of interpretable machine learning. arXiv e-prints. February2017; arXiv:1702.08608.
8.
ArrietaABDíaz-RodríguezNDel SerJ, et al. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inform Fusion2020; 58: 82–115.
9.
LapuschkinSWäldchenSBinderA, et al. Unmasking Clever Hans predictors and assessing what machines really learn. Nat Commun2019; 10: 1–8.
10.
LiptonZC. The mythos of model interpretability. ACM Queue2018; 61: 36–43.
11.
Jiménez-LunaJGrisoniFSchneiderG. Drug discovery with explainable artificial intelligence. Nat Mach Intell2020; 2: 573–584.
12.
MillerT. Explanation in artificial intelligence: Insights from the social sciences. Artif Intell2019; 267: 1–38.
13.
UstunBRudinC. Supersparse linear integer models for optimized medical scoring systems. Mach Learn2016; 102: 349–91.
FéraudRClérotF. A methodology to explain neural network classification. Neural Networks2002; 15: 237–46.
16.
RibeiroMTSinghSGuestrinC. “Why should i trust you?” Explaining the predictions of any classifier. In: KDD ’16: Proceedings of the 22nd ACM SIGKDD, 2016, pp.1135–1144.
17.
LundbergSMNairBVavilalaMS, et al. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nat Biomed Eng2018; 2: 749–760.
18.
ShapleyLS. Notes on the n-Person Game—II: The Value of an n-Person Game. Report, RAND Corporation, 1951.
19.
ZeilerMDFergusR. Visualizing and understanding convolutional networks. In: European conference on computer vision (eds FleetDPajdlaTSchieleBTuytelaarsT), Zurich, Switzerland, 2014, pp.818–833. Springer International Publishing.
20.
ZhouBKhoslaALapedrizaA, et al. Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, 2016, pp.2921–2929.
21.
SelvarajuRRCogswellMDasA, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, 2017, pp.618–626. Available at: http://arxiv.org/abs/1610.02391
22.
ZhaoGZhouBWangK, et al. Respond-CAM: Analyzing Deep Models for 3D Imaging Data by Visualizations. In: Medical Image Computing and Computer Assisted Intervention – MICCAI2018 2018, pp.485–492. Cham: Springer International Publishing.
23.
YangCRangarajanARankaS. Visual explanations from deep 3D convolutional neural networks for Alzheimer’s disease classification. AMIA Annu Symp Proc2018: pp.1571–1580.
24.
SundararajanMTalyAYanQ. Axiomatic Attribution for Deep Networks. In: Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 2017, pp.3319–3328.
25.
BachSBinderAMontavonG, et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One2015; 10: 1–46.
PennebakerJWFrancisMEBoothRJ. Linguistic inquiry and word count. Mahw: Lawrence Erlbaum Associates, 2001.
28.
CrossleySAKyleKMcNamaraDS. Sentiment analysis and social cognition engine (SEANCE): An automatic tool for sentiment, social cognition, and social-order analysis. Behav Res Methods2017; 49: 803–821.
29.
FiokKKarwowskiWGutierrez-FrancoE, et al. Analysis of sentiment in tweets addressed to a single Twitter account: comparison of model performance and explainability of predictions. Symmetry2020. Available at: https://github.com/krzysztoffiok/twitter_sentiment.
30.
HuttoCGilbertE. VADER: A parsimonious rule-based model for sentiment analysis of social media text. In: Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media, Ann Arbor, 2014. Available at: https://ojs.aaai.org/index.php/ICWSM/article/view/14550.
31.
HuMLiuB. Mining and summarizing customer reviews. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, 2004, pp.168–177.
32.
HochreiterSSchmidhuberJ. Long short-term memory. Neural Comput1997; 9(8): 1735–80.
33.
LiJChenXHovyE, et al. Visualizing and understanding neural models in NLP. arXiv e-prints. June2015; arXiv:1506.01066.
34.
ArrasLMontavonGMüllerK-R, et al. Explaining Recurrent Neural Network Predictions in Sentiment Analysis. arXiv e-prints. June2017; arXiv:1706.07206.
VigJ. A multiscale visualization of attention in the transformer model. arXiv e-prints. June2019; arXiv:1906.05714.
38.
LiuYOttMGoyalN, et al. RoBERTa: a robustly optimized BERT pretraining approach. arXiv e-prints. July2019; arXiv:1907.11692.
39.
Alvarez-MelisDJaakkolaTS. On the robustness of interpretability methods. arXiv e-prints. June2018; arXiv:1806.08049.
40.
SimonyanKVedaldiAZissermanA. Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv e-prints. December2013; arXiv:1312.6034.
41.
KindermansP-JHookerSAdebayoJ, et al. The (Un)reliability of Saliency Methods. In: SamekWMontavonGVedaldiA, et al. (eds) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Cham: Springer International Publishing, 2019, pp.267–280.
42.
MontavonGLapuschkinSBinderA, et al. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit2017; 65: 211–222.
43.
SpringenbergJTDosovitskiyABroxT, et al. Striving for simplicity: the all convolutional net. arXiv e-prints. December2014; arXiv:1412.6806.
44.
AdebayoJGilmerJMuellyM, et al. Sanity checks for saliency maps. Adv Neural Inf Process Syst2018: 9505–9515.
45.
ShrikumarAGreensidePKundajeA. Learning important features through propagating activation differences. In: Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 2017, pp.3145–3153.
46.
SelvarajuRRDasAVedantamR, et al. Grad-CAM: Why did you say that? arXiv e-prints. November2016; arXiv:1611.07450.
47.
SmilkovDThoratNKimB, et al. SmoothGrad: removing noise by adding noise. arXiv e-prints. June2017; arXiv:1706.03825.
48.
ErhanDBengioYCourvilleA, et al. Visualizing higher-layer features of a deep network. Bernoulli2009; 1341: 1–13.
49.
DouglasPKFarahaniFV. On the similarity of deep learning representations across didactic and adversarial examples. arXiv e-prints. February2020; arXiv:2002.06816.
MuellerSTHoffmanRRClanceyW, et al. Explanation in human-AI systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. arXiv e-prints. February2019; arXiv:1902.01876.
52.
Du BoulayB. Artificial intelligence as an effective classroom assistant. IEEE Intell Syst2016; 31(6): 76–81.
BullS. Supporting learning with open learner models. Planning2004; 29: 1.
55.
LaneHCCahillCFoutzS, et al. The effects of a pedagogical agent for informal science education on learner behaviors and self-efficacy. In: LaneHCYacefKMostowJPavlikP (eds) Artificial intelligence in education. Berlin: Springer Berlin Heidelberg, 2013, pp.309–318.
56.
VanLehnK. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ Psychol2011; 46: 197–221.
57.
KoedingerKRAlevenV. An interview reflection on “intelligent tutoring goes to school in the big city.”Int J Artif Intell Educ2016; 26: 13–24.
58.
ConatiCPorayska-PomstaKMavrikisM. AI in education needs interpretable machine learning: lessons from open learner modelling. arXiv e-prints. June2018; arXiv:1807. 00154.
59.
CooperA. The inmates are running the asylum:[Why high-tech products drive us crazy and how to restore the sanity]. Indianapolis: Sams, 2004.
60.
ShehRK. Different XAI for different HRI. In: 2017 AAAI Fall Symposium Series, Arlington, Virginia, 2017.
61.
SilverDHubertTSchrittwieserJ, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv e-prints. December2017; arXiv:1712. 01815.
62.
Van LentMFisherWMancusoM. An explainable artificial intelligence system for small-unit tactical behavior. In: Proceedings of the National Conference on Artificial Intelligence, San Jose, California, 2004.
63.
CoreMGLaneHCVan LentM, et al. Toward question answering for simulations. In: Proceedings of the IJCAI 2005 Workshop on Knowledge and Reasoning for Answering Questions (KRAQ05), Edinburgh, Scotland, 2005.
64.
CoreMTraumDLaneHC, et al. Teaching negotiation skills through practice and reflection with virtual humans. Simulation2006; 82: 685–701.
65.
HarbersMVan Den BoschKMeyerJ-J. Self-explaining agents in virtual training. In: EC-TEL PROLEAN, Maastricht, The Netherlands, 2008.
66.
SchneebergerTGebhardPBaurT, et al. PARLEY: A transparent virtual social agent training interface. In: Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, 2019, pp.35–36.
67.
WagnerJBaurTSchillerD, et al. Show me what you’ve learned: applying cooperative machine learning for the semi-automated annotation of social signals. In: IJCAI/ECAI, Workshop on XAI 2018, Stockholm, Sweden, 2018.
68.
PutnamVConatiC. Exploring the need for explainable artificial intelligence (XAI) in intelligent tutoring systems (ITS). In: IUI Workshops, Marina del Ray, CA, 2019.
69.
CowlingPIBuroMBidaM, et al. Search in real-time video games. In: Dagstuhl Seminar 12191, Wadern, Germany, 2013.
70.
AllenBD. Serious gaming for building a basis of certification via trust and trustworthiness of autonomous systems. In: 2018 Aviation Technology, Integration and Operations Conference, Atlanta, Georgia, 2018, pp.1–6.
71.
GilpinLHBauDYuanBZ, et al. Explaining explanations: an overview of interpretability of machine learning. In: 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy, 2018, pp.80–89.
72.
PáezA. The pragmatic turn in explainable artificial intelligence (XAI). Mind Mach2019; 29: 441–59.
73.
EhsanUTambwekarPChanL, et al. Automated rationale generation: A technique for explainable AI and its effects on human perceptions. In: International Conference on Intelligent User Interfaces, Proc IUI, Marina del Ray, CA, 2019, Part F1476, pp.263–74.
74.
HolzingerADehmerMJurisicaI. Knowledge discovery and interactive data mining in bioinformatics - State-of-the-Art, future challenges and research directions. BMC Bioinf2014; 15: I1.
