On the role of knowledge graphs in explainable AI

Abstract

The current hype of Artificial Intelligence (AI) mostly refers to the success of machine learning and its sub-domain of deep learning. However, AI is also about other areas, such as Knowledge Representation and Reasoning, or Distributed AI, i.e., areas that need to be combined to reach the level of intelligence initially envisioned in the 1950s. Explainable AI (XAI) now refers to the core backup for industry to apply AI in products at scale, particularly for industries operating with critical systems. This paper reviews XAI not only from a Machine Learning perspective, but also from the other AI research areas, such as AI Planning or Constraint Satisfaction and Search. We expose the XAI challenges of AI fields, their existing approaches, limitations and opportunities for Knowledge Graphs and their underlying technologies.

1. Introduction

Artificial Intelligence (AI), as a discipline aiming at building intelligent machines mimicking “cognitive” functions that humans associate with other human minds, such as “learning”, “problem solving” [67], and addresses intelligence for systems from a large variety of facets. From Machine Learning (ML) to Knowledge Representation and Reasoning (KRR), Game Theory, Uncertainty in AI (UAI), Robotics, Multi-Agent Systems, Constraint Satisfaction and Search (CSS), Planning and Scheduling, Computer Vision, Natural Language Processing, all are foundational pillars of AI as we know it today. All latter sub-fields of AI have matured, specialized, and sometimes converged together with the aim of accessing to General Artificial Intelligence, i.e., the holy grail of AI.

Many research questions have been vertical to all sub-fields of AI, such as decidability and complexity from a theoretical perspective or scalability from a more applied dimension. However, one is remaining current, even getting more traction than others in the new world of industrialized AI: explainability. Obtaining explainable AI systems consists in addressing the following question: “how to build intelligent systems able to expose explanation in a human-comprehensible way” for any of its AI decision. We will use the well-adopted XAI term, standing for eXplainable AI, when referencing to the explanation problem in AI. Answering this XAI question is far from trivial, and has been studied for years in all subfields of AI, with no exception. Such problem has been tackled under different names, concepts, definitions, with various requirements and objectives. For instance interpretation and justification are terms coined in KRR, diagnostics in UAI, debugging in robotics, constraints relaxation in CSS, feature importance in ML, or feature attribution for Neural Networks [73,79].

Despite a surge of innovation focusing on ML-based AI systems such question of explainability has not been deeply studied as much as in the other AI subfields, such as KRR. However, answers to this question of explainability and questions related to the responsibility, validity (e.g., robustness), privacy-preserving and more broadly trust of AI systems (Fig. 1) will be intrinsically connected to the adoption of AI in industry at scale, particularly in industries operating with critical systems. Indeed explanation, which could be used for debugging intelligent systems or deciding to follow a recommendation in real-time, will increase acceptance and user trust.

Fig. 1.

On the combination of valid, responsible, privacy-preserving and explainable AI towards trustable AI.

Unsurprisingly, the exact same research community, from which the most successful ML-based AI systems [35,74] emerged, is now trying to fill the gap between black-box ML systems [46] to more white-box ML systems. Some approaches are more successful than others, but still the AI community is far from having self-explainable AI systems which automatically adapt to any (i) data, (ii) ML algorithm, (iii) model, (iv) user, or (v) application and (v) context. Even more surprisingly, only works in KRR and its subfields of Web and AI, i.e., Semantic Web [11], Linked Data [12], and more recently Knowledge Graphs [13], engaged in the endeavour of explaining the broader family of ML-based systems. However, KRR, the Semantic Web together with Knowledge Graphs, aiming at representing and reasoning over structured information [25], should be designed and armed to move XAI closer to human comprehension. In the following we will refer to Knowledge Graphs any graph structured knowledge bases that store factual information in form of relationships between entities [58] e.g., YAGO [78], DBpedia [7], NELL [16], Freebase [13], and the Google Knowledge Graph [77].

This paper reviews XAI in the various fields of AI, i.e., by first describing the main research question, its XAI challenge, existing approaches, their limitations and opportunities for Knowledge Graphs and their underlying technologies.

Fig. 2.

XAI challenges in major AI fields (DAI: distributed AI, UAI: uncertainty in AI, KRR: knowledge representation and reasoning, NLP: natural language processing).

2. Knowledge graphs for XAI methods

This section highlights the main research question in major AI fields, their associated XAI challenge (Fig. 2), together with existing approaches, their limitations and opportunities for Semantic Web and Knowledge Graphs technologies. AI areas are broken down following the AAAI taxonomy for research paper submission [75]. Although such a taxonomy has some limitations e.g., questionable limit, natural intersection of AI domains, at least it benefits from a well-accepted list of fields in AI, which are well-represented in major generalist AI conferences, such as IJCAI [50] and ECAI [42].

2.1. Machine learning (except neural netwok)

∙ Research Question : ML algorithms [68] aim at elaborating a mathematical model based on sample data, known as“training data”, in order to make predictions or decisions on unseen data, known as “test data” without being explicitly programmed to perform the task. Five main tasks of learning are studied: (i) supervised learning if data contains both input and labeled data, (ii) unsupervised learning to derive some structures in data if labels are not exposed, (iii) semi-supervised learning if labelled data is small compared to unlabelled data, (iv) distant learning [34] which exploits relational data of unlabelled data from existing knowledge bases, and (v) reinforcement learning if further information could be captured through interaction with the environment.

∙ XAI Challenge : All tasks of ML expose mathematical models through an appropriate, but somehow abstract representation of data. XAI in ML [30] is about explanation of (i) models, known as global explanation, and (ii) a prediction, known as local explanation.

∙ Approaches : Some models are naturally designed to explicit their rationale e.g., linear regression, decision trees, generalized linear (or additive), naive Bayes models. In case of more complex models, some of their representative elements, such as feature importance, partial dependency plot or individual conditional expectation can be used for capturing high level representation of the ML model for global explanation. State-of-the-art approaches [52,64] go further by revisiting feature importance for local explanation.

∙ Limitations : Most approaches limit explanation to features involved in the data and model, or at best to examples, prototypes [44] or counterfactuals [55]. Explanation should go beyond correlation (which is what feature importance is about) and numerical similarity (which is what local explanation is about).

∙ Opportunity : Knowledge Graphs do encode contexts, do expose connections and relations, and support inference and causation natively. Existing XAI approaches in ML consider a flat representation of data, and context is out of the loop of the explanation process. Knowledge Graphs could be used for encoding better representations of data, structuring an ML model in a more interpretable way, adopt semantic similarity for local explanation. For instance we could envision linking knowledge graphs extracts to input data of a Machine Learning task to solve some distant learning tasks [34]. In addition we could envision approaches relying on Knowledge Graphs to compact large trees in decisions trees or even random forest. For instance combinations of nodes could be captured as a unique (probabilistic) concept or property in Knowledge Graphs. Machine Learning and Knowledge Graphs have great potential to be combined, and benefit from each other strength [26].

2.2. Artificial (deep) neural networks

∙ Research Question : Similarly to other ML approaches, Artificial Neural Networks (ANNs) aim at learning representation. The main differentiator with other approaches is its scalability and performance with a high number of features and instances, which better fit images and texts.

∙ XAI Challenge : Both local and global explanations are a strong focus of the ANN community.

∙ Approaches : Contrary to other ML approaches, there is no easy way around explanation of ANN models or predictions. Existing techniques either encode feature importance through attribution [73,79], attention mechanism [62], or obtain a more interpretable approximation through surrogate models [24], such as decision tree.

Fig. 3.

On the role of knowledge graphs for explainable artificial (deep) neural networks. (What is the causal relationship between the input/output/training data?) – Extension of Fig. 8 in [83] and https://fortune.com/longform/ai-artificial-intelligence-deep-machine-learning/.

∙ Limitations : Explanations are artificially built, for instance by forcing the network to focus on some group of features or correlations at best. In addition they do not represent any logic of the learning task, making explanation a very difficult task to achieve. The latter is due to the foundational theory of ANN, which consists in deriving a mathematical model through local optimizations.

∙ Opportunity : Novel ANN architectures need to be designed to natively encode explanation. Some recent approaches which aim at capturing better model hierarchical relationships [36], or causality mechanism [10] are promising. However, they could be polished further by (i) adding logic representation layers in ANN, such as [38] using network dissection approaches [8], (ii) encoding the semantics of inputs, outputs and their properties cf. Fig. 3. Knowledge Graphs could play a central role in such a new design, particularly as novel architectures should embed causation and feature reasoning. This is the case of [53] which introduced a layered graph model representation of (RDF-type) graphs in the ANN architectures for reasoning purpose. The layer is representing the semantics of predicates in Knowledge Graphs, and is captured as $3\mathit{D}$ adjacency matrices. Other approaches from the neural-symbolic reasoning community [37] are worth investigating as they combine ANNs with probabilistic logic [54] or first order fuzzy logic [27]. Knowledge graph embeddings [33,72] are also Machine Learning artifacts where explanations could be elaborated their a latent representations. Such design could advance ANN further by supporting integration, discovery, fragmentation, composition and even reasoning.

2.3. Computer vision

∙ Research Question : Computer Vision relies on ANN architectures due to the nature and size of its data. Tasks range from semantic segmentation, object detection, scene reconstruction to visual question answering.

∙ XAI Challenge : The main XAI task in Computer Vision is identification of pixels, or group of pixels responsible for triggering a shape detection, an uncertainty or an error. Explanation is often referred to as visual inspection due to the nature of data processed.

∙ Approaches : Saliency maps [2] are classic methodologies in Computer Vision. They include many variants of gradient modification for capturing representative features. Network dissection [8] is another approach segmenting ANN to derive interpretable units and layers.

∙ Limitations : Although saliency maps expose interesting visualization artifacts, they do not capture any semantics. At best those artifacts capture a disentangled representation, which remain subject to human interpretation. Knowledge Graphs could expose the semantics of such disentangled representation. However, integrating semantics in ANN, hidden units of feature space remain open challenges.

∙ Opportunity : Adding semantics through context and Knowledge Graphs could help answering open questions, such as: What is a disentangled representation, and how can its factors be quantified and detected? Do interpretable hidden units reflect a special alignment of feature space, or are interpretations a chimera? All are open questions discussed in [8], and not yet resolved. Other open questions are: What conditions in state-of-the-art training lead to representations with greater or lesser entanglement? What is the semantics of a group of hidden units in neural networks? Interesting avenues aim at combining detection with reasoning to improve, and potentially explain semantic segmentation [3].

2.4. Constraint satisfaction and search

∙ Research Question : Constraint Satisfaction and Search aims at finding a solution to a set of constraints that impose conditions that the variables must satisfy. A solution is a set of values for the variables that satisfies all constraints. Constraints are defined on a finite domain.

∙ XAI Challenge : The main challenge is to identify which constraints to relax for conflict resolutions. Explanations are usually a subset of variables which satisfies a set of constraints.

∙ Approaches : Constraint Satisfaction and Search problems on finite domains are typically solved using a form of search. Backtracking, constraint propagation, local search are examples of such approaches. Even though the problem is known to be an NP complete problem with respect to the domain size, research has shown a number of tractable sub-cases with promising approaches [40,59].

∙ Limitations : Even though optimal structures and search spaces have been largely introduced in the community, complexity remains one of the main limitations.

∙ Opportunity : It has been demonstrated that any structure in problem representation has largely benefited search [49]. We could envision more knowledge-driven structure, inspired from Knowledge Graphs, which could dynamically adapt to variables, constraints, search space. Knowledge Graphs could even drive search through semantic and logical relations among constraints, which could be modelled as entities in a graph. In such cases constraints will be augmented with distant data from Knowledge Graphs.

2.5. Game theory

∙ Research Question : Game Theory [70] is the study of mathematical models of strategic interaction between rationale decision-makers. Examples of games include zero-sum games [56], in which one person’s gains result in losses for the other participants.

∙ XAI Challenge : Game Theory has been dealing with XAI from its inception as one of its main challenge is to identify and to understand the underlying mathematical model as well as its properties. Game theory is applied to a wide range of behavioural relations, and is now an umbrella term for the science of logical decision making in humans, animals, and computers, in which explanation is the core question driving the modelling.

∙ Approaches : The Shapley value [69] is a solution concept in game theory, which inspired recent research in Machine Learning to address the problem of explanation [52]. The Shapley value is characterized by a collection of desirable properties, and is used to capture the influence of a player in a game settings (or a feature in a machine learning setting). Such properties characterize the explanation.

∙ Limitations : Similarly to the domain of Constraint Satisfaction and Search, complexity is a challenge for explainability in game theory. Only an approximate solution is feasible, usually identified through some randomization of coalition in feature values.

∙ Opportunity : As recently explored, structured representation of the models as its features [18] has shown better scalability, while not necessarily improving explainability. Knowledge Graphs could be considered to better structure models, organize features, then reducing the search space and potentially improve understanding and readability of explanation, particularly when embedded in a structured set of connected entities. Recent examples [19] have demonstrated that graph structures do reduce the complexity of search.

2.6. Uncertainty in AI

∙ Research Question : The field of Uncertainty in AI is at the frontier of various AI fields, namely knowledge representation, learning and reasoning. Bayesian probability is one of the core fundamental, and Probabilistic Graphical Models (PGMs) [47] are usually central for representing and reasoning with uncertainty as they encode probability distributions.

∙ XAI Challenge : Graphical models are often used to model multivariate data, since they allow to represent high-dimensional distributions compactly. The explanations draw their attention on the compact distributions and their underlying data. Explanation is then naturally embedded through those relationships, usually through interdependencies and decomposition in data.

∙ Approaches : Some approaches [9] are formulating PGMs as weighted logical formulas [43] to tightly decouple the constraints and dependencies from the probabilistic parameters. Reasoning can then be performed on the logic representations. Other approaches analyzes latent spaces and its direct connections with the underlying data [81]. The strength of existing approaches is the underlying reasoning capabilities that PGMs and other probabilistic and logic systems offer.

∙ Limitations : Even though PGMs are appropriate representations to connect inter-dependable data, dependencies remains probabilistic. Therefore humans are required to remain in the loop to interpret any dependencies. Even embedded in logical formulas there is little gained as we are still embedded in the framework of standard probability theory.

∙ Opportunity : Semantic representations and connections through Knowledge Graphs could be used to disambiguate and force latent variables to represent interpretable content. This is particularly relevant as PGMs fit naturally in graph representations, in contextual information such as knowledge graphs could extend reasoning functionalities. Interesting avenues are Probabilistic Knowledge Graphs [57] or knowledge expansion over probabilistic knowledge bases [20].

2.7. Robotics

∙ Research Question : Robotics is an interdisciplinary branch of engineering and AI science, which deals with the design, construction, operation, and use of robots, as well as computer systems for their control, sensory feedback, and information processing. The underlying technologies are used to develop machines that can replicate human actions. They usually combine and integrate many of the technologies in the AI field.

∙ XAI Challenge : XAI is required in Robotics mainly for debugging and resolving discrepancy between a solution and an expected answer. Some of the XAI challenges are (1) the rationale of coordination in multi-robots Systems and swarms, (2) the fusion of explanation coming from many underlying AI systems, such as Planning and Scheduling, Computer Vision, or Knowledge Representation and Reasoning. They are unique challenges for robotics with many interesting opportunities as explanation is multi-modal, could be complementary but also conflicting, is spatial and temporal, is driven by goals but also initial conditions.

∙ Approaches : Narration of autonomous robot experience [66] together with approaches of summarization [15] have been recently introduced as a succinct way of presenting the decision process of robots. Various levels of granularity in the decision process are provided. [61] combine a robotics ontology with linguistic elements to expose the rational of robots’ actions.

∙ Limitations : Although the latter models extract information from a large poll of data, such systems do not explain their actions and justify their decisions [71]. Explanation is usually too fine-grained to be properly integrated by humans. Seamless integration of multi-modal explanation is also not addressed in the literature.

∙ Opportunity : The level of abstraction in explanation together with its multi-modal fusion are net opportunities for Knowledge Graphs. Some semantics could deeply support in exposing appropriate and personalized representations of explanations while fusing explanation content in a compact and comprehensible representation [60]. Knowledge Graphs have been designed to capture knowledge from heterogenous domains, making them a great candidate to achieve explanation per se in robotics.

2.8. Distributed AI

∙ Research Question : Distributed AI is the field of AI dedicated to the development of distributed solutions for problems. It is related to Multi-Agent Systems but also to any representation, structure, system which could make AI scalable.

∙ XAI Challenge : Main XAI challenges are focusing on explaining and resolving agent conflicts, based on their intentions and beliefs [80]. State-of-the-art approaches aim at identifying the best strategy, through explanation, to achieve a goal. More recent works focus on human comprehension of agent behaviour, its strategy, and its convergence in case of conflicting intentions and beliefs of agents [4,5].

∙ Approaches : Approaches, such as [39] determines the motivation for a decision by recalling the situation in which the decision was made, and replaying the decision under variants of the original situation. In such scenario they are able to discover what factors led to the decisions, and what alternatives might have been chosen had the situation been slightly different. Approaches tend to be very close to counterfactual [14] and case-based reasoning [1].

∙ Limitations : Even though ontology is a core representation layer for agents to communicate and negotiate, it is rarely used for explaining agent behaviour, its strategy and success. Lighter knowledge representations might be envisioned.

∙ Opportunity : The dynamics of agents interaction should be captured more formally, and embedded with broader common sense knowledge to identify human interpretable explanation. Formalization does not need to be complex. For instance some dedicated Knowledge Graphs could be used to contextualize the agents environment. Some recent works are going towards this direction of formalizing agent interactions [21].

2.9. Automated planning and scheduling

∙ Research Question : Automated Planning and Scheduling [29] is a branch of Artificial Intelligence that is about the realization of strategies or action sequences, typically for execution by intelligent agents, autonomous robots and unmanned vehicles. Unlike classical control and classification problems, the solutions are complex and must be discovered and optimized in multi-dimensional space. It could be done in real-time, i.e., on-line, or at design-time, i.e., off-line. Solutions usually resort to iterative trial and error processes.

∙ XAI Challenge : XAI challenges in AI planning [28] are as follows: explaining (i) causal relationships of actions, (ii) why some actions are chosen in particular situations, (iii) why plans are better than some, (iv) why plans could not be computed, (v) why replanning might be required.

∙ Approaches : Past work on explanations primarily involved the AI system explaining the correctness of its plan and the rationale for its decision in terms of its own model [17].

∙ Limitations : Existing approaches fail in exposing human-understandable explanation, as it is usually limited to the planner’s domain e.g., in term of actions and initial situation. This strongly limits the comprehension to experts in the given tasks.

∙ Opportunity : Knowledge Graphs could be a way forward to better contextualize complex terms, and even better summarize complex actions in more succinct and meaningful way.

2.10. Natural language processing

∙ Research Question : Natural Language Processing is concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data. Research questions includes (visual [6], multi-turn [51]) question answering [48], conversational agents with broader questions related to Speech Recognition, Natural Language Understanding and Generation.

∙ XAI Challenge : Similarly to machine learning, identifying importance of feature or entity is critical, as it aims at identifying which part of speech is driving the most relevant information. Other core XAI tasks include: explaining the rationale of questions sequencing in dialogue, debugging a plan-based dialogue system [45] or explaining the utterances which were intended to achieve [23]

∙ Approaches : The problem of identifying the most representative entities in a text classification task is addressed by [64] with many variants. Some works [22] extract plan-based model to understand intention and explain rationale of the discourse.

∙ Limitations : On the one hand ML-based approaches, which focus on important entities in text, suffer from having statistics-based explanation only, i.e., mainly based on co-occurrence and correlation. Pioneering work [76], relying on tree like structure in form of dependency trees, have been first steps towards structuring text processing tasks. On the other hand plan-based models have not been deeply explored, and many research questions related to their representation, rationale in questions sequencing remain open.

∙ Opportunity : Semantic descriptions, exposing meaningful representations, have demonstrated to have a positive impact on tasks such as relation extraction [32,41], event extraction [63] or text classification [82]. Similar representations, inspired from Knowledge Graphs could provide the semantic layer missing from brute-force machine learning approaches on text, aiming at exposing explanation [65]. They could also drive or at least guide sequencing of questions by refining, abstracting or instantiating obscure terms in questions. Challenges and approaches from neural language models for the semantic web are also interesting avenues of exploration [31].

3. Conclusion

Despite a surge of innovation focusing on ML-based AI systems, industry is facing the dilemma of applying in products at scale, particularly for industries operating with critical systems. Trust, and trust in AI has been revelled as the one term coining industry needs to move to the next step. Trustable AI is about responsibility validity, privacy-preserving modelling and also explainability. Explanation, which could be used for debugging intelligent systems or deciding to follow a recommendation in real-time, will increase acceptance and user trust. Explanation in AI has different open questions, meaning, definitions and approaches, depending on which AI fields is touching the question. Although various solutions have been introduced, the question remain open in all areas of AI. We presented their challenges in more details, some of their existing approaches, their limitations and opportunities for Knowledge Graphs to bring explainable AI to the right level of semantics and interpretability. Indeed significant progress in complex AI tasks, such as explainable AI could only be achieved through combinations with semantic layers, empowering explanation of complex AI systems.