Sage Journals: Discover world-class research

Abstract

Symbolic Systems in Artificial Intelligence which are based on formal logic and deductive reasoning are fundamentally different from Artificial Intelligence systems based on artificial neural networks, such as deep learning approaches. The difference is not only in their inner workings and general approach, but also with respect to capabilities. Neural-symbolic Integration, as a field of study, aims to bridge between the two paradigms. In this paper, we will discuss neural-symbolic integration in its relation to the Semantic Web field, with a focus on promises and possible benefits for both, and report on some current research on the topic.

Keywords

Neural-symbolic integration deductive reasoning artificial neural networks deep learning

Approaches in Artificial Intelligence (AI) based on machine learning, and in particular those employing artificial neural networks, differ fundamentally from approaches that leverage knowledge bases to perform logical deduction and reasoning.1

¹
We focus herein on deductive reasoning. Logical inductive and abductive reasoning have also been looked at in the Semantic Web context, e.g. [14,22], but to keep the discussion concise, we have not included them in this treatise.

The former are connectionist or subsymbolic AI systems able to solve complex tasks over unstructured data using supervised or unsupervised learning, including problems which cannot reasonably be hand-coded by humans. Subsymbolic methods are generally robust against noise in training or input data and have recently, in the wake of deep learning, been shown to exceed human performance in tasks involving video, audio, and text processing. The latter are symbolic systems that thrive under the presence of large amounts of structured data, including for agent planning, constraint solving, data management, integration and querying, and other traditional application areas of knowledge-based systems and formal semantics. Classical rule-based systems, ontologies, and knowledge graphs that power search and information retrieval across the Web are also types of symbolic AI systems.

Symbolic and subsymbolic systems are rather complementary to each other. For example, the key strengths of subsymbolic systems are weaknesses of symbolic ones, and vice versa. Symbolic systems are brittle, i.e., susceptible to data noise or minor flaws in the logial encoding of a problem, which stands in contrast to the robustness of connectionist approaches. But subsymbolic systems are generally black boxes in the sense that the systems cannot be inspected in ways that provide insight into their decisions (despite some recent progress on this in the wake of the explainable AI effort) while symbolic knowledge bases can in principle be inspected to interpret how a decision follows from input. Most importantly, symbolic and subsymbolic systems contrast in the types of problems and data they excel at. Scene recognition from images appears to be a problem which in general lies outside the capabilities of symbolic systems, for example, while complex planning scenarios appear to be outside the scope of current deep learning approaches.2

The topic is being investigated, of course, and some recent progress is made. E.g., [1] reports on an application of deep learning to planning, and explicitly frames it as work towards bridging the “subsymbolic-symbolic boundary.”

On a more technical level, symbolic and subsymbolic systems differ fundamentally in how they represent data, information, or knowledge. Symbolic systems typically utilize structured representation languages, e.g. stemming from formal logic and the subfield of AI known as knowledge representation and reasoning. Trainable artificial neural networks, on the other hand, typically use representations based on high-dimensional Euclidean space, i.e. real-valued vectors, matrices, etc., and it is by no means obvious how reconciliations between these representation forms can be designed.3

It is possible to establish a formal, mathematical bridge in some cases, as e.g. laid out in [31], but so far with limited applicability [3].

The complementary nature of these methods has drawn a divide in the rich field of AI. The divide is technical in nature, as symbol manipulation as captured by logical, deductive reasoning, which lies at the core of symbolic approaches, cannot be sufficiently performed using current subsymbolic systems. Moreover, the training to study subsymbolic systems (involving probability theory, statistics, linear algebra, and optimization) differs from symbolic systems (involving logic and propositional calculus, set and recursion theory, and advanced computability reasoning) so strongly that AI researchers tend to find a side of the divide based on their intellectual interests and background. The divide is also cultural in nature, one of mindsets and prior believes, that in the past could sometimes split the academic AI research community by provoking (heated) fundamental discussions. The divide is even geographical, where the European Union holds a much higher prevalence of researchers working on symbolic approaches than in the United States.

Neural-Symbolic Integration [2,4,16,28],4

⁴

See also http://www.neural-symbolic.org/.

as a field of research, addresses fundamental problems related to building a technical bridge between the symbolic and subsymbolic sides of the divide. The promises from a successful bridging of the divide are plenty. In the abstract, one could hope for best-of-both-worlds systems, which combine the transparency and reasoning-ability of symbolic systems with the robustness and learning-capabilities of subsymbolic ones. As such, integrated symbolic-subsymbolic systems may be able to address the knowledge acquisition bottleneck faced by symbolic systems, learn to perform advanced logical or symbolic reasoning tasks even in the presence of noisy or uncertain facts, and even yield self-explanatory subsymbolic models. This is the promise of added value of neural-symbolic integration research for Computer Science. And also more fundamentally, a bridging may shed insights into how natural (human) neural networks can perform tasks as witnessed by homo sapiens pursuing mathematics, formal logic, and other pursuits that we, introspectively, see as symbolic in nature; this is a basic research problem for Cognitive Science as a discipline.

In the following, we will first lay out, in more detail, promises and possible benefits of neural-symbolic integration research for the Semantic Web. Then we will look at potential benefits of Semantic Web and neural-symbolic integration research for deep learning. Finally, we will also provide brief pointers to some current research going on in relation to this theme.

1. Benefits of neural-symbolic integration for the Semantic Web

One of the issues that plagues the Semantic Web (as well as many other fields in Computer Science and its applications) is the knowledge acquisition bottleneck. It refers to the difficult issue of encoding or otherwise storing knowledge, as structured information, for use in Computer Science applications. The manual encoding of such information, e.g. from human experts’ knowledge, is a very slow and time-consuming, thus costly, process involving both topic experts and knowledge engineers. At the same time, automated methods are a far cry from producing artifacts (e.g., from textbooks, technical documentations, and other written sources) which would be of sufficient quality for use in intelligent systems applications based on logical inference, such as expert systems, or for data curation and integration.

The underpinnings of key Semantic Web standards, such as RDF [9] and OWL [29], are explicitly logical, which reflects that Semantic Web applications often rely on high data (and schema/ontology) quality, similar to knowledge bases used primarily for deductive reasoning. The knowledge acquisition bottleneck in the Semantic Web field is very noticeable, e.g., given that the creation of ontologies as well as the creation of high-quality knowledge graphs involves high amounts of human export labour and is correspondingly expensive.

The promise of integrated neural-symbolic systems is that they would be capable of both learning and (deductive) reasoning, and thus that they would be able to acquire, through machine learning, knowledge which is of sufficiently high quality to perform deductive reasoning. This anticipated capability directly addresses the knowledge acquisition bottleneck. There is, thus, a promise in this line of work that integrated neural-symbolic systems will lead to

better methods for automated ontology construction,

better methods for ontology population (and, thus, knowledge graph construction),

better methods for ontology alignment,

better methods for assessing the quality of knowledge graph content,

and similar major lines of research central to the Semantic Web field.

At the same time, integrated neural-symbolic systems carry the promise of being able to perform deductive reasoning – after training – using a (highly parallel) artificial neural network architecture. Consequently, reasoning using such systems can be expected to be extremely fast. This contrasts with traditional deductive reasoning methods, which are usually designed to be provably sound and complete but suffer from long algorithm runtimes. While there has been significant progress on developing highly efficient deductive reasoning engines for Semantic Web content, this remains an issue given ever-increasing availability of data. In fact, the underlying problem is fundamental, as sound and complete reasoning over Semantic Web data necessarily suffers from high computational complexity [30].

Integrated neural-symbolic systems would perform reasoning after training, and presumably this form of reasoning would not be provably sound and complete, but would trade correctness guarantees with higher runtime efficiency, in the spirit of approximate reasoning – see e.g. [32] for an exhibition of the underlying rationale. As such, integrated neural-symbolic systems carry a promise to elevate deductive Semantic Web reasoning to much larger amounts of data.

With integrated neural-symbolic systems capable of approximate deductive reasoning, this would furthermore open up possible investigations into combining deductive and inductive reasoning, as well as common-sense reasoning based e.g. on natural language, within a single (artificial neural network based) system.

Side products of such approaches would also be, e.g., entity encodings in formats suitable for artificial neural networks, such as vector or matrix representations. These in turn could be utilized to assess entity similarity with potential applications in data integration. Such encodings could furthermore be used as a sort of compression for data transfer and storage.

2. Benefits of Semantic Web technologies and neural-symbolic integration for deep learning

Semantic Web Technologies are designed for enabling better and more efficient data sharing, discovery, integration and reuse. These data management core capabilities of Semantic Web Technologies are designed to ease the data curation and preparation burden for the training of deep learning systems. Semantic Web data, provided in large amounts and freely available on the Web [51], furthermore provides a rich resource for training data, and deductive reasoning methods over such data can further extend it.

Integrated neural-symbolic systems will furthermore make it possible to utilize background knowledge, given as knowledge graphs or ontologies, as part of deep learning applications. Promises of this include the leveraging of background knowledge and deductive reasoning aspects for improved trainability, but also for interpreting trained deep learning systems by means of background knowledge. The former aspect attempts to reinforce the usefulness of deep learning models through injection of knowledge and has been successfully used in task-oriented conversational AI systems [23] and question answering [44]. The latter aspect touches on the Explainable AI theme currently being discussed, which aims at addressing the black-box nature of deep learning systems by making them more transparent, understandable, verifiable, and trustworthy. Most of the current work on this topic attempts to explain system behavior by means of input or output features; however explanations by way of background knowledge carry the promise of being much closer to human conceptualizations, and thus more useful in applications.

Integrated neural-symbolic systems which incorporate deductive reasoning capabilities could furthermore naturally combine these with inferences based on statistics or similarities, including natural-language common-sense reasoning as demonstrated by some deep learning approaches. Such combinations should naturally lead to stronger deep learning systems.

Neural-symbolic systems have already been used on linked datasets like Freebase and DBpedia for different tasks like link prediction [62] and noise tolerant RDFS reasoning [41]. The links between linked datasets could further allow neural-symbolic systems to both integrate and reason over information coming from different sources. The advantage of this is twofold: firstly, the combined information can be used to extend the amount of training data for neural-symbolic systems; secondly, a neural-symbolic system can be used to learn to reason over a single knowledge graph and then links can be used as entry points to reason over a different one. This could be useful in contexts in which it is costly to learn to reason over a large dataset; one could thus use neural-symbolic methods over a smaller one (or a part of the large one) and then use the learned capabilities over the large one.

Recent years have also seen some progress in zero/few-shot relation learning over knowlegdge graphs, utilizing deep learning [12]. Zero/few-shot relation learning refers to the ability of the deep learning model to infer new relations of pairs of entities where that relation has not been seen or has only occurred a few times before in the training set [7]. This generalization capability is still quite limited and fundamentally different from the efforts that have been done under transfer learning and the domain adaptation paradigm in other machine learning tasks.

3. Selection of recent related work

3.1. Deductive Semantic Web reasoning using deep learning

Deductive reasoning over RDF(S) and OWL data has become a part of the standard toolbox for knowledge graphs, and the use of neural-symbolic systems for this purpose has begun to be investigated.

[41] has proposed a noise-tolerant algorithm for deep-learning-based reasoning designed specifically for RDF(S) knowledge graphs. They have introduced a layered graph model representation of RDF graphs based on their predicates, in the form of 3D adjacency matrices where each layer layout forms a graph word. Each input graph and its corresponding entailments then have been represented as a sequence of graph words and have been fed to a neural machine translation model. Their results show noise-tolerant capabilities of their deep model, compared to their symbolic counterpart. However evaluation and training are done on a dataset that uses only one ontology for the inference, i.e., there is no learning of the general logical deduction calculus, and consequently no transfer thereof to new data.

[33] applies Recursive Reasoning Networks (RNN) to OWL RL reasoning where recursive update layers are used to update the individual embeddings using the relations and class memberships in the knowledge base. Their results show the potential of neural-symbolic methods to attain accuracy similar to symbolic methods. However, as for the above mentioned [41], re-training is required for new ontologies to learn the embeddings for the new vocabularies in the ontology, i.e., the approach does not natively support transfer to new data.

[20] addresses the transferability issue by adapting end-to-end memory networks for emulating deductive RDFS reasoning. Transfer was achieved primarily by utilizing a preprocessing step consisting of a normalization. It was demonstrated that the resulting approach can perform reasoning over previously unseen RDFS knowledge graphs.

3.2. Knowledge graph embeddings

With the recent revival of interest in artificial neural networks, neural link prediction models have been applied extensively for the completion of knowledge graphs, understood in the sense of link-prediction.5

⁵
Traditionally, “completion” in the context of RDF(S) referred only to materialization of logical consequences; more recently, the term has also been used to refer to the adding of new relationships (graph edges) based on statistical or NLP methods.

These methods [6 ,10 ,39 ,47 ,50 ,62 –64,66,69] heavily rely on the subsymbolic representations, called embeddings, of entities and relations learned through maximization of a scoring objective function over valid factual triples. Thus, the current success of such deep models hinges primarily on the power of those subsymbolic continuous real-valued representations in encoding the similarity/relatedness of entities and relations. For example, TransE [6] for a knowledge base triple s, p, o learns an embedding function by minimizing the distance based on the respective vector representations

d (s + p, o)

where d is a distance function. These methods have been often tested over link prediction tasks.

The use of additional information, such has text, can increase the quality of the representation [65,67,68]. Moreover, embedded representations of knowledge graphs can be extended by considering the logical axioms that appear in a knowledge base, for example, complex logical formulas can be aggregated using fuzzy logic [27].

A recent trend in knowledge graph embedding concerns approaches that use hyperbolic geometry in place of euclidean geometry [46,54]; hyperbolic geometry generally appears to be more suited to represent hierarchical structures like terminologies and ontologies.

Node2vec [24] is instead a widely adopted approach that combines random walks and natural language techniques [43] to efficiently generate vector representations of networks nodes that has also been used to support knowledge graph embeddings [48]. In the same line of works, RDF2Vec [52] embeds RDF-based entities in a vector space by applying word embedding techniques [43] over a virtual document that contains lexicalized rdf-graph walks; thus the generated representations are based on token-token co-occurrences.

While most knowledge graph embedding approaches rely on a single encoding of triples, there is a recent line of work that tries to leverage the information that can be found in longer paths using recurrent neural networks [15,70]

Also recently, a number of works have been done on the problem of generalizing neural networks to work on arbitrarily structured graphs [17,34] opening promising directions for future research on reasoning on structured data.

3.3. Explainable deep learning

While deep learning is highly successful [36] and even surpasses human capabilities [59,60] in many fields, it also lacks transparency or interpretability [26,40] of how a decision is being produced from these systems. In safety-critical applications, e.g., in medical, legal or military contexts, this is deemed insufficient. Consequently, researchers are investigating how to produce explanations for the behavior of deep learning systems [25].

Explanations [55,71,72] produced from deep learning systems are mostly statistical and helpful to understand how it produces the output, and the additional use of domain information helps to enhance [19] the explanation. [49] used an ontology-based deep learning model which predicts human behaviour via Restrictricted Boltzman Machines [61] and produces explanations of the output using domain ontologies. In the domain of transfer learning to explain which features are beneficial and which are not for the transfer, [11] used domain knowledge to enhance the explanation.

[73] shows the use of semantic annotations to label objects in the hidden layers of popular CNN architectures. Labels ranging from colors, materials, textures, parts, objects and scenes help to get a better understanding of hidden parts of the deep network. Although the labels are not semantically structured, this shows that background knowledge can help to improve explainability.

[57] provides a feasibility study on how domain ontologies together with description logic based concept induction [37,56] can be used to explain input output behaviour of trained deep neural networks.

Although the explanation produced solely using statistical techniques is beneficial, it is far from being a trustable explanation [35,40]. The main limitation of statistical methods is that it does not take domain knowledge or general background knowledge into account when making the output. A combined effort to use statistical techniques with semantic web technologies should be helpful to provide trustable explanations. An overview of using knowledge graphs to enhance explanation, and possible limitations of this, is described in [35].

3.4. Other systems for deductive reasoning using deep learning

The Neural Theorem Prover (NTP) [53] is an extension of the Prolog language in which strict atom unification is replaced with similarity of atoms in an embedded space; while originally NTP suffered from scalability issues, due to the complexity of the approach, there is evidence that proof-path selection strategies can reduce the complexity impact [45].

DeepProbLog [42] is a programming language that combines a probabilistic logic with neural networks, thus offering a framework that combines the strengths from both approaches.

Logic Tensor Networks [58] (LTNs) combine deep neural networks and first order fuzzy logic. Elements of the logic language are embedded in a vector space (e.g., constants are represented as vectors while predicates are neural tensor networks [62] that have been used on simple reasoning tasks [8,62]). LTNs can be trained over both facts and rules and after training they can be used to make novel logical inferences over data. LTNs have been applied to semantic image interpretation tasks [18] but they have also been shown to have some computational limitations [5].

PossibleWorldNet [21] is a variant of Tree Neural Networks (TreeNN) which has been successfully used for conducting entailment over propositional logic formulas. To evaluate whether A entails B, the PossibleWorldNet generates a set of “possible worlds,” and then evaluates A and B in each of those worlds. Their results show the clear advantage of using this model compared to sequence-to-sequence models which would capture the structure implicitly.

Neural multi-hop reasoners [15,70] deal with more complex reasoning on large knowledge bases where multi-hop inference is required. They combine the rich multi-hop inference of the symbolic logical reasoning paradigm with the generalization capabilities of attention-based recurrent neural networks.

4. Conclusion

In the wake of deep learning, neural-symbolic approaches are receiving renewed attention. We have laid out promises of neural-symbolic integration research for the Semantic Web field, and vice versa. It appears to be reasonable to expect that the corresponding lines of research will receive growing attention in forthcoming years. E.g., several articles in this issue point into similar directions [13,35,38].

References

Asai and

Fukunaga, Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary, in: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2–7, 2018,

S.A.

McIlraith and

K.Q.

Weinberger, eds, AAAI Press, 2018.

Bader and

Hitzler, Dimensions of neural-symbolic integration – A structured survey, in: We Will Show Them! Essays in Honour of Dov Gabbay, Volume One,

S.N.

Artëmov,

Barringer,

A.S.

d’Avila Garcez,

L.C.

Lamb and

Woods, eds, College Publications, 2005, pp. 167–194.

Bader,

Hitzler,

Hölldobler and

Witzel, A fully connectionist model generator for covered first-order logic programs, in: IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6–12,

M.M.

Veloso, ed., 2007, pp. 666–671.

T.R.

Besold,

A.S.

d’Avila Garcez,

Bader,

Bowman,

P.M.

Domingos,

Hitzler,

Kühnberger,

L.C.

Lamb,

Lowd,

P.M.V.

Lima,

de Penning,

Pinkas,

Poon and

Zaverucha, Neural-symbolic learning and reasoning: A survey and interpretation, CoRR, arXiv:1711.03902, 2017.

Bianchi and

Hitzler, On the capabilities of logic tensor networks for deductive reasoning, in: Proceedings of the AAAI 2019 Spring Symposium on Combining Machine Learning with Knowledge Engineering (AAAI-MAKE 2019), Stanford University, Palo Alto, California, USA, March 25–27, 2019,

Martin,

Hinkelmann,

Gerber,

Lenat,

van Harmelen and

Clark, eds, CEUR Workshop Proceedings, Vol. 2350, CEUR-WS.org, 2019.

Bordes,

Usunier,

García-Durán,

Weston and

Yakhnenko, Translating embeddings for modeling multi-relational data, in: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, Proceedings of a meeting held December 5–8, 2013, Lake Tahoe, Nevada, United States,

C.J.C.

Burges,

Bottou,

Ghahramani and

K.Q.

Weinberger, eds, 2013, pp. 2787–2795.

Bordes,

Weston,

Collobert and

Bengio, Learning structured embeddings of knowledge bases, in: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, California, USA, August 7–11, 2011,

Burgard and

Roth, eds, AAAI Press, 2011.

S.R.

Bowman,

Potts and

C.D.

Manning, Learning distributed word representations for natural logic reasoning, in: 2015 AAAI Spring Symposia, Stanford University, Palo Alto, California, USA, March 22–25, 2015, AAAI Press, 2015.

Brickley and

Guha (eds), RDF Schema 1.1. W3C Recommendation 25 February 2014, 2010. Available from: http://www.w3.org/TR/rdf-schema.

10.

Chang,

Yih,

Yang and

Meek, Typed tensor decomposition of knowledge bases for relation extraction, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Doha, Qatar, October 25–29, 2014, A meeting of SIGDAT, a Special Interest Group of the ACL,

Moschitti,

Pang and

Daelemans, eds, ACL, 2014, pp. 1568–1579.

11.

Chen,

Lécué,

J.Z.

Pan,

Horrocks and

Chen, Knowledge-based transfer learning explanation, in: Principles of Knowledge Representation and Reasoning: Proceedings of the Sixteenth International Conference, KR 2018, Tempe, Arizona, 30 October–2 November, 2018,

Thielscher,

Toni and

Wolter, eds, AAAI Press, 2018, pp. 349–358.

12.

Chen,

Zhang,

Chen and

Chen, Meta relational learning for few-shot link prediction in knowledge graphs, arXiv preprint, arXiv:1909.01515, 2019.

13.

d’Amato, Machine learning for the semantic web: Lessons learnt and next research directions, Semantic Web Journal (2020), to appear.

14.

d’Amato,

Fanizzi,

Fazzinga,

Gottlob and

Lukasiewicz, Ontology-based semantic search on the web and its combination with the power of inductive reasoning, Ann. Math. Artif. Intell.65(2–3) (2012), 83–121. doi:10.1007/s10472-012-9309-7.

15.

Das,

Neelakantan,

Belanger and

McCallum, Chains of reasoning over entities, relations, and text using recurrent neural networks, in: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3–7, 2017, Vol. 1: Long Papers,

Lapata,

Blunsom and

Koller, eds, Association for Computational Linguistics, 2017, pp. 132–141. doi:10.18653/v1/e17-1013.

16.

A.S.

d’Avila Garcez,

Gori,

L.C.

Lamb,

Serafini,

Spranger and

S.N.

Tran, Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning, FLAP6(4) (2019), 611–632.

17.

Defferrard,

Bresson and

Vandergheynst, Convolutional neural networks on graphs with fast localized spectral filtering, in: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, December 5–10, 2016,

D.D.

Lee,

Sugiyama,

von Luxburg,

Guyon and

Garnett, eds, 2016, pp. 3837–3845.

18.

Donadello,

Serafini and

A.S.

d’Avila Garcez, Logic tensor networks for semantic image interpretation, in: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19–25, 2017,

Sierra, ed., ijcai.org, 2017, pp. 1596–1602. doi:10.24963/ijcai.2017/221.

19.

Doran,

Schulz and

T.R.

Besold, What does explainable AI really mean? A new conceptualization of perspectives, in: Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017 Co-Located with 16th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017), Bari, Italy, November 16th and 17th, 2017,

T.R.

Besold and

Kutz, eds, CEUR Workshop Proceedings, Vol. 2071, CEUR-WS.org, 2017.

20.

Ebrahimi,

M.K.

Sarker,

Bianchi,

Xie,

Doran and

Hitzler, Reasoning over RDF knowledge bases using deep learning, arXiv preprint, arXiv:1811.04132, 2018.

21.

Evans,

Saxton,

Amos,

Kohli and

Grefenstette, Can neural networks understand logical entailment?, in: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference Track Proceedings, OpenReview.net, 2018.

22.

Fanizzi,

d’Amato and

Esposito, Inductive classification of semantically annotated resources through reduced Coulomb energy networks, in: Semantic Services, Interoperability and Web Applications – Emerging Concepts,

A.P.

Sheth, ed., CRC Press, 2011, pp. 322–342. doi:10.4018/978-1-60960-593-3.ch013.

23.

Ghazvininejad,

Brockett,

Chang,

Dolan,

Gao,

Yih and

Galley, A knowledge-grounded neural conversation model, in: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2–7, 2018,

S.A.

McIlraith and

K.Q.

Weinberger, eds, AAAI Press, 2018, pp. 5110–5117.

24.

Grover and

Leskovec, node2vec: Scalable feature learning for networks, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13–17, 2016,

Krishnapuram,

Shah,

A.J.

Smola,

C.C.

Aggarwal,

Shen and

Rastogi, eds, ACM, 2016, pp. 855–864. doi:10.1145/2939672.2939754.

25.

Guidotti,

Monreale,

Ruggieri,

Turini,

Giannotti and

Pedreschi, A survey of methods for explaining black box models, ACM Comput. Surv.51(5) (2019), 93:1–93:42. doi:10.1145/3236009.

26.

Gunning, DARPA’s explainable artificial intelligence (XAI) program, in: Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI 2019, Marina del Ray, CA, USA, March 17–20, 2019,

Fu,

Pan,

Brdiczka,

Chau and

Calvary, eds, ACM, 2019. doi:10.1145/3301275.3308446.

27.

Guo,

Wang,

Wang and

Guo, Jointly embedding knowledge graphs and logical rules, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016,

Su,

Carreras and

Duh, eds, The Association for Computational Linguistics, 2016, pp. 192–202. doi:10.18653/v1/D16-1019.

28.

Hammer and

Hitzler (eds), Perspectives of Neural-Symbolic Integration, Studies in Computational Intelligence, Vol. 77, Springer, 2007.

29.

Hitzler,

Krötzsch,

Parsia,

P.F.

Patel-Schneider and

Rudolph (eds), OWL 2 Web Ontology Language: Primer, 2nd edn, W3C Recommendation 11 December 2012, 2012. Available at: http://www.w3.org/TR/owl2-primer.

30.

Hitzler,

Krötzsch and

Rudolph, Foundations of Semantic Web Technologies, Chapman and Hall/CRC Press, 2010.

31.

Hitzler and

A.K.

Seda, A note on the relationships between logic programs and neural networks, in: 4th Irish Workshop on Formal Methods, IWFM 2000, Maynooth, Ireland, 5th–6th July 2000,

Sinclair and

J.P.

Gibson, eds, Workshops in Computing, BCS, 2000.

32.

Hitzler and

van Harmelen, A reasonable semantic web, Semantic Web1(1–2) (2010), 39–44. doi:10.3233/SW-2010-0010.

33.

Hohenecker and

Lukasiewicz, Deep learning for ontology reasoning, CoRR, arXiv:1705.10342, 2017.

34.

T.N.

Kipf and

Welling, Semi-supervised classification with graph convolutional networks, in: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017Conference Track Proceedings, OpenReview.net, 2017.

35.

Lecue, On the role of knowledge graphs in explainable AI, Semantic Web journal (2019), to appear.

36.

LeCun,

Bengio and

G.E.

Hinton, Deep learning, Nature521(7553) (2015), 436–444. doi:10.1038/nature14539.

37.

Lehmann and

Hitzler, Concept learning in description logics using refinement operators, Machine Learning78(1–2) (2010), 203–250. doi:10.1007/s10994-009-5146-2.

38.

Li,

Qi and

Ji, Hybrid reasoning in knowledge graphs: Combining symbolic reasoning and statistical reasoning, Semantic Web journal (2019), to appear.

39.

Lin,

Liu,

Sun,

Liu and

Zhu, Learning entity and relation embeddings for knowledge graph completion, in: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, Texas, USA, January 25–30, 2015, 2015, pp. 2181–2187.

40.

Z.C.

Lipton, The mythos of model interpretability, arXiv preprint, arXiv:1606.03490, 2016.

41.

Makni and

Hendler, Deep learning for noise-tolerant RDFS reasoning, Semantic Web10(5) (2019), 823–862. doi:10.3233/SW-190363.

42.

Manhaeve,

Dumancic,

Kimmig,

Demeester and

L.D.R.

Deepproblog, Neural probabilistic logic programming, in: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, 3–8 December, 2018, 2018, pp. 3753–3763.

43.

Mikolov,

Sutskever,

Chen,

G.S.

Corrado and

Dean, Distributed representations of words and phrases and their compositionality, in: Advances in Neural Information Processing Systems,

C.J.C.

Burges,

Bottou,

Welling,

Ghahramani and

K.Q.

Weinberger, eds, Vol. 26, Curran Associates, Inc., 2013, pp. 3111–3119.

44.

A.H.

Miller,

Fisch,

Dodge,

Karimi,

Bordes and

Weston, Key-value memory networks for directly reading documents, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016, 2016, pp. 1400–1409. doi:10.18653/v1/D16-1147.

45.

Minervini,

Bosnjak,

Rocktäschel and

Riedel, Towards neural theorem proving at scale, arXiv preprint, arXiv:1807.08204, 2018.

46.

Nickel and

Kiela, Poincaré embeddings for learning hierarchical representations, in: Advances in Neural Information Processing Systems,

Guyon,

U.V.

Luxburg,

Bengio,

Wallach,

Fergus,

Vishwanathan and

Garnett, eds, Vol. 30, Curran Associates, Inc., 2017, pp. 6341–6350.

47.

Nickel,

Tresp and

H.-P.

Kriegel, Factorizing YAGO, in: Proceedings of the 21st International Conference on World Wide Web – WWW 12, ACM Press, 2012. doi:10.1145/2187836.2187874.

48.

Palumbo,

Rizzo,

Troncy,

Baralis,

Osella and

Ferro, Knowledge graph embeddings with node2vec for item recommendation, in: The Semantic Web: ESWC 2018 Satellite Events,

Gangemi,

A.L.

Gentile,

A.G.

Nuzzolese,

Rudolph,

Maleshkova,

Paulheim,

J.Z.

Pan and

Alam, eds, Springer International Publishing, Cham, 2018, pp. 117–120. doi:10.1007/978-3-319-98192-5_22.

49.

Phan,

Dou,

Wang,

Kil and

Piniewski, Ontology-based deep learning for human behavior prediction with explanations in health social networks, Information Sciences384 (2017), 298–313. doi:10.1016/j.ins.2016.08.038.

50.

Riedel,

Yao,

McCallum and

B.M.

Marlin, Relation extraction with matrix factorization and universal schemas, in: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia, Association for Computational Linguistics, 2013, pp. 74–84.

51.

Rietveld,

Beek,

Hoekstra and

Schlobach, Meta-data for a lot of LOD, Semantic Web8(6) (2017), 1067–1080. doi:10.3233/SW-170256.

52.

Ristoski and

Paulheim, Rdf2vec: RDF graph embeddings for data mining, in: The Semantic Web – ISWC 2016,

Groth,

Simperl,

Gray,

Sabou,

Krötzsch,

Lecue,

Flöck and

Gil, eds, Springer International Publishing, Cham, 2016, pp. 498–514. doi:10.1007/978-3-319-46523-4_30.

53.

Rocktäschel and

Riedel, End-to-end differentiable proving, in: Advances in Neural Information Processing Systems,

Guyon,

U.V.

Luxburg,

Bengio,

Wallach,

Fergus,

Vishwanathan and

Garnett, eds, Vol. 30, Curran Associates, Inc., 2017, pp. 3788–3800.

54.

Sala,

C.D.

Sa,

Gu and

Ré, Representation tradeoffs for hyperbolic embeddings, in: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, 2018, pp. 4457–4466.

55.

Samek,

Wiegand and

K.-R.

Müller, Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models, arXiv preprint, arXiv:1708.08296, 2017.

56.

M.K.

Sarker and

Hitzler, Efficient concept induction for description logics, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 3036–3043. doi:10.1609/aaai.v33i01.33013036.

57.

M.K.

Sarker,

Xie,

Doran,

Raymer and

Hitzler, Explaining trained neural networks with semantic web technologies: First steps, in: Proceedings of the Twelfth International Workshop on Neural-Symbolic Learning and Reasoning, NeSy 2017, London, UK, July 17–18, 2017,

T.R.

Besold,

A.S.

d’Avila Garcez and

Noble, eds, CEUR Workshop Proceedings, Vol. 2003, CEUR-WS.org, 2017.

58.

Serafini and

A.S.

d’Avila Garcez, Learning and reasoning with logic tensor networks, in: AI*IA 2016: Advances in Artificial Intelligence – XVth International Conference of the Italian Association for Artificial Intelligence, Genova, Italy, November 29–December 1, 2016,

Adorni,

Cagnoni,

Gori and

Maratea, eds, Proceedings, Lecture Notes in Computer Science, Vol. 10037, Springer, 2016, pp. 334–348. doi:10.1007/978-3-319-49130-1_25.

59.

Silver,

Huang,

C.J.

Maddison,

Guez,

Sifre,

van den Driessche,

Schrittwieser,

Antonoglou,

Panneershelvam,

Lanctot,

Dieleman,

Grewe,

Nham,

Kalchbrenner,

Sutskever,

T.P.

Lillicrap,

Leach,

Kavukcuoglu,

Graepel and

Hassabis, Mastering the game of go with deep neural networks and tree search, Nature529(7587) (2016), 484–489. doi:10.1038/nature16961.

60.

Silver,

Hubert,

Schrittwieser,

Antonoglou,

Lai,

Guez,

Lanctot,

Sifre,

Kumaran,

Graepel,

T.P.

Lillicrap,

Simonyan and

Hassabis, Mastering chess and shogi by self-play with a general reinforcement learning algorithm, CoRR, arXiv:1712.01815, 2017.

61.

Smolensky, Information processing in dynamical systems: Foundations of harmony theory, Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.

62.

Socher,

Chen,

C.D.

Manning and

A.Y.

Ng, Reasoning with neural tensor networks for knowledge base completion, in: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, Proceedings of a meeting held December 5–8, 2013, Lake Tahoe, Nevada, United States,

C.J.C.

Burges,

Bottou,

Ghahramani and

K.Q.

Weinberger, eds, 2013, pp. 926–934.

63.

Toutanova,

Chen,

Pantel,

Poon,

Choudhury and

Gamon, Representing text for joint embedding of text and knowledge bases, in: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17–21, 2015,

Màrquez,

Callison-Burch,

Su,

Pighin and

Marton, eds, The Association for Computational Linguistics, 2015, pp. 1499–1509. doi:10.18653/v1/D15-1174.

64.

Trouillon,

Welbl,

Riedel,

É.

Gaussier and

Bouchard, Complex embeddings for simple link prediction, in: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19–24, 2016,

Balcan and

K.Q.

Weinberger, eds, JMLR Workshop and Conference Proceedings, Vol. 48, JMLR.org, 2016, pp. 2071–2080.

65.

Wang and

Li, Text-enhanced representation learning for knowledge graph, in: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9–15 July 2016,

Kambhampati, ed., IJCAI/AAAI Press, 2016, pp. 1293–1299.

66.

Wang,

Zhang,

Feng and

Chen, Knowledge graph embedding by translating on hyperplanes, in: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Québec City, Québec, Canada, July 27–31, 2014,

C.E.

Brodley and

Stone, eds, AAAI Press, 2014, pp. 1112–1119.

67.

Xiao,

Huang,

Meng and

Zhu, SSP: Semantic space projection for knowledge graph embedding with text descriptions, in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, California, USA, February 4–9, 2017,

S.P.

Singh and

Markovitch, eds, AAAI Press, 2017, pp. 3104–3110.

68.

Xie,

Liu,

Jia,

Luan and

Sun, Representation learning of knowledge graphs with entity descriptions, in: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, USA, February 12–17, 2016,

Schuurmans and

M.P.

Wellman, eds, AAAI Press, 2016, pp. 2659–2665.

69.

Yang,

Yih,

He,

Gao and

Deng, Embedding entities and relations for learning and inference in knowledge bases, in: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015,

Bengio and

LeCun, eds, Conference Track Proceedings, 2015, pp. 7–9.

70.

Yin,

Yaghoobzadeh and

Schütze, Recurrent one-hop predictions for reasoning over knowledge graphs, in: Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20–26, 2018,

E.M.

Bender,

Derczynski and

Isabelle, eds, Association for Computational Linguistics, 2018, pp. 2369–2378.

71.

Yosinski,

Clune,

A.M.

Nguyen,

T.J.

Fuchs and

Lipson, Understanding neural networks through deep visualization, CoRR, arXiv:1506.06579, 2015.

72.

Zhang and

Zhu, Visual interpretability for deep learning: A survey, Frontiers of IT & EE19(1) (2018), 27–39. doi:10.1631/FITEE.1700808.

73.

Zhou,

Bau,

Oliva and

Torralba, Interpreting deep visual representations via network dissection, CoRR, arXiv:1711.05611, 2017.

Neural-symbolic integration and the Semantic Web

Abstract

Keywords

1 We focus herein on deductive reasoning. Logical inductive and abductive reasoning have also been looked at in the Semantic Web context, e.g. [14,22], but to keep the discussion concise, we have not included them in this treatise.

2. Benefits of Semantic Web technologies and neural-symbolic integration for deep learning

3. Selection of recent related work

3.1. Deductive Semantic Web reasoning using deep learning

3.2. Knowledge graph embeddings

5 Traditionally, “completion” in the context of RDF(S) referred only to materialization of logical consequences; more recently, the term has also been used to refer to the adding of new relationships (graph edges) based on statistical or NLP methods.

3.4. Other systems for deductive reasoning using deep learning

4. Conclusion

References

¹
We focus herein on deductive reasoning. Logical inductive and abductive reasoning have also been looked at in the Semantic Web context, e.g. [14,22], but to keep the discussion concise, we have not included them in this treatise.

⁵
Traditionally, “completion” in the context of RDF(S) referred only to materialization of logical consequences; more recently, the term has also been used to refer to the adding of new relationships (graph edges) based on statistical or NLP methods.