A survey on knowledge graph embeddings with literals: Which model links better literal-ly?

Abstract

Knowledge Graphs (KGs) are composed of structured information about a particular domain in the form of entities and relations. In addition to the structured information KGs help in facilitating interconnectivity and interoperability between different resources represented in the Linked Data Cloud. KGs have been used in a variety of applications such as entity linking, question answering, recommender systems, etc. However, KG applications suffer from high computational and storage costs. Hence, there arises the necessity for a representation able to map the high dimensional KGs into low dimensional spaces, i.e., embedding space, preserving structural as well as relational information. This paper conducts a survey of KG embedding models which not only consider the structured information contained in the form of entities and relations in a KG but also its unstructured information represented as literals such as text, numerical values, images, etc. Along with a theoretical analysis and comparison of the methods proposed so far for generating KG embeddings with literals, an empirical evaluation of the different methods under identical settings has been performed for the general task of link prediction.

1. Introduction

Knowledge Graphs (KGs) have become quite crucial for storing structured information. There has been a sudden attention towards using KGs for various applications mainly in the area of artificial intelligence. For instance, in a more general sense, KGs can be used to support decision making process and to improve different machine learning applications such as question answering [3], recommender systems [82], and relation extraction [74]. Some of the most popular publicly available general purpose KGs are DBpedia [40], Wikidata [68], and YAGO [47]. These general purpose KGs often consist of huge amount of facts constructed using millions of entities (represented as nodes) and relations (as edges connecting these nodes).

Although KGs are effective in representing structured data, there exist some issues which hinder their efficient manipulation such as i) different KGs are usually based on different rigorous symbolic frameworks and this makes it hard to utilize their data in other applications [6] and ii) the fact that a significant number of important graph algorithms needed for the efficient manipulation and analysis of graphs have proven to be NP-complete [23]. In order to address these issues and use a KG more efficiently, it is beneficial to transform it into a low dimensional vector space while preserving its underlying semantics. To this end, various attempts have been made so far to learn vector representations (embeddings) for KGs.

As discussed in [69], a typical KG embedding approach, which uses only structured information from the KG, generally follows three steps: (i) determining the form of entity and relation representations, (ii) defining a scoring function, and (iii) learning entity and relation representations. In the first step, the forms in which entities and relations are represented in the vector space are determined. Entities can be represented as vectors or modeled as multivariate Gaussian distributions whereas relations can be encoded as operations, matrices, tensors, multivariate Gaussian distributions, or mixtures of Gaussians. Once the form of the entities are determined, in the second step, a scoring function which measures the plausibility of a triple is defined. The main goal is to enable the model to assign higher score to true triples and lower scores to false/negative/corrupted triples. Thus, in order to achieve this, the third step solves an optimization problem which maximizes the plausibility of true facts in order to learn the embeddings of entities and relations. Note that the method used to generate false/negative/corrupted triples has an impact on the performance of a model. The various negative triple generation methods and their differences are discussed in detail in [36].

Among the different embedding approaches proposed so far, TransE [5] is, to the best of our knowledge, the very first attempt to use distance-based scoring function to learn KG embedding. Given a triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ where h and t are head and tail entities respectively and r is a relation, TransE represents h, r, and t as vectors $\mathbf{\text{h}}$ , $\mathbf{\text{r}}$ , and $\mathbf{\text{t}}$ respectively by modeling the relation r as a translation vector which connects the vectors $\mathbf{\text{h}}$ and $\mathbf{\text{t}}$ . The problem with TransE is that it fails to model certain type of relations such as one-to-many or many-to-one. In order to address such limitations, different embedding techniques which are extension of TransE or are entirely new have been proposed. However, most of the existing approaches, including the current state-of-the-art models such as ConvE [15], are structure-based embeddings which do not make use of any literal information i.e., only triples consisting of entities connected via properties are usually considered. This is a major disadvantage because information encoded in the literals will be left unused when capturing the semantics of a certain entity.

Literals can bring advantages to the process of learning KG embeddings in two major ways:

1. Learning embeddings for novel entities: Novel entities are the entities which are not linked to any other entity in the KG but have literal values associated with them such as their textual description, numeric literals, and images. In most existing structure-based embedding models, it is not possible to learn embeddings for such novel entities. However, this can be addressed by utilizing the information represented in literals to learn embeddings. For example, considering the dataset FB15K-20 [78], which is a subset of Freebase, the entity ’/m/0gjd61t’ is a novel entity which does not occur in any of the training triples, but it has a description given as follows in the form $⟨\mathit{\text{subject}}\mathrm{,}\mathit{\text{relation}}\mathrm{,}\mathit{\text{object}}⟩$ .

   

   In order to learn the embedding for this particular entity (i.e., /m/0gjd61t), the model can make use of the entity’s textual description. DKRL [78] is one of those approaches which provide embeddings for novel entities using their descriptions.

   

   Fig. 1.

   A small fraction of triples taken from the KG DBpedia [40].

   
   Fig. 2.

   A small fraction of triples with literals taken from the KG DBpedia [40].

2. Improving the representation of entities in structure based embedding models: Literals play a vital role in improving the representation learning where an entity is required to appear in at least a minimum number of relational triples. For example, taking into consideration only the information provided in a sample KG presented in Fig. 1, which is extracted from DBpedia, it is not possible to tell apart the entities dbr:Gina_Torres, dbr:Patrick_J.Adams, and dbr:Sarah_Rafferty from one another. This is the case due to the fact that the only information that is available regarding these entities in this KG is that they all star in the series dbr:Suits_(season_1) and this is not enough to know which entities are similar to each other and which are not. Therefore, if some KG embedding model is trained using only this KG, it is not possible to get good representations for the entities dbr:Gina_Torres, dbr:Patrick_J.Adams, and dbr:Sarah_Rafferty.

   However, having the model trained with more triples containing literal values for these entities, as shown in Fig. 2, would improve the embeddings for the entities. For instance, based on the values of the data relation dbr:birthDate, it is possible to deduce the fact that dbr:Sarah_Rafferty and dbr:Gina_Torres are almost the same age but they are both older than dbr:Patrick_J.Adams. On the other hand, the images of the entities along with the textual descriptions (dbo:abstract) would allow us to infer the entities’ gender, i.e., dbr:Sarah_Rafferty and dbr:Gina_Torres are female and dbr:Patrick_J.Adams is male.

   The above example indicates that the use of literals along with their respective entities would add more semantics so that similar entities can be represented close to each other in the vector space while those dissimilar are further apart.

Recently, some approaches have been proposed which leverage the information present in literals to learn KG embeddings. The types of literals considered in these embedding methods are either text, numeric, images, or multi-modal literals, i.e., a combination of more than one medium of information. These methods use different techniques in order to incorporate the literals into the KG embeddings. However, data typed literals are not addressed in these KG embedding models and surveys that are conducted on KG embeddings. The main challenge with data typed literals, such as date and time, is that they require additional semantics to be represented in KG embeddings.

Table 1

KG embedding models and their categories

Categories – based on the method used	Models without literals	Models with literals
Translational distance	TransE [5] and its extensions: TransH [71] TransR [44], TransD [31], TranSparse [32], TransA [33] etc.	TransEA [75], DKRL [78], IKRL [77], Jointly(desp) [84], Jointly [80], SSP [76], KDCoE [9], EAKGAE [67]
Semantic matching	RESCAL [52] and its extensions: DistMult [81], HolE [50], ComplEx [66], and etc. Semantic matching with neural networks: SME [4], NTN [60], MLP [16], and etc.	LiteralE [37], MKBE [55], MTKGNN [63], KGlove with literals [10], Extended RESCAL [51], LiteralE with blocking [14]
Entity types	SSE [26], TKRL [79], Type constrained representation learning [39], Rules incorporated KG completion models [70], TRESCAL [8], Entity Hierarchy Embedding [29]	Extended RESCAL [51]
Relation paths	PTransE [42], Traversing KGs in Vector Space [28], RTRANSE [21], Compositional vector space [49], Reasoning using RNN [12], Context-dependent KG embedding [46]	KBLRN [22]
Logical rules	Rules incorporated KG completion models [70], Large-scale Knowledge Base Completion [73], KALE [27], Logical Background Knowledge for Relation Extraction [56], and etc.
Temporal information	Time-Aware Link Prediction [34], co-evolution of event and KGs [17], Know-evolve [65]
Graph structures	GAKE [18], Link Prediction in Multi-relational Graphs [35]	KBLRN [22]

This survey analyses different embedding approaches, which make use of literals, and highlights their advantages and drawbacks in handling different challenges such as multi-valued data properties/relations, data typed literals, and units of literals. A review of the different applications used for model evaluation by different KG embedding models is also presented. Furthermore, experiments with some of the models have been conducted specifically on the link prediction task. The contribution of this paper is summarized as follows:

1. A detailed analysis of the existing literal enriched KG embedding models and their approaches. In addition, the models are categorized into different classes based on the type of literals used.

2. An evaluation oriented comparison of the existing models on the link prediction task is performed under same experimental settings.

3. The research gaps in the area of KG embeddings in using literals are indicated which can open directions for further research.

The rest of this paper is organized as follows: Section 2 presents a brief overview of related work. In Section 3, the problem formulation including definitions, preliminaries, types of literals and research questions are provided while Section 4 analyses different KG embedding techniques with literals is discussed. Section 5 reviews different tasks used to train or evaluate the embedding models is given. Section 6 discusses the experiment conducted with the existing KG embedding models with literals on the link prediction task. Finally, concluding remarks summarize our findings on KGs with literals and are presented along with future directions in Section 7.

2. Related work

This section describes the state-of-the-art algorithms proposed for generating KG embeddings. It also gives a brief overview of the surveys already published following these lines and what is lacking in those studies.

A brief overview of the most popular KG embedding techniques, including the state-of-the-art approaches are short listed in Table 1. The categories presented in this table are inspired by a previous survey work [69] for the models without literals (column 1). These categories are created based on the methods used by the models, i.e., translation distance, semantic matching, entity types, relation paths, logical rules, temporal information, and graph structures. We have also categorized the techniques which use literals with respect to the same set of categories. Since a detailed discussion on these categories on the models without literals has already been presented in [69], in the current study, the main focus lies on analysing the models which make use of literals. The standard KG embedding techniques which are extended by the models with literals are listed in Table 2.

Table 2

KG embedding models with literals and their corresponding base models

Models with literals	The standard models they extend
Extended RESCAL [51]	RESCAL [52]
Jointly(desp) [84]	TransE [5]
DKRL [78]	TransE [5]
Jointly [80]	TransE [5]
SSP [76]	TransE [5]
KDCoE [9]	TransE [5]
KGlove with literals [10]	KGlove
LiteralE [37]	DistMult [81], ComplEx [66], ConvE [15]
TransEA [75]	TransE [5]
IKRL [77]	TransE [5]
MTKGRL [58]	TransE [5]
EAKGAE [67]	TransE [5]
MKBE [55]	DistMult [81], ConvE [15]

Few attempts have been made to conduct surveys on the techniques and applications of KG embeddings [7,25,69]. The survey [25] is conducted on factorization based, random walk based, and deep learning based network embedding approaches such as DeepWalk, Node2vec, and etc. [7,69] discuss only RESCAL [52] and KREAR [43] as methods which use attributes of entities for KG embeddings, and focus mostly on the structure-based embedding methods, i.e., methods using non-attributive triples, for example, translation based embedding models listed in Table 1. However, RESCAL is a matrix-factorization method for relational learning which encodes each object/data property as a slice of the tensor leading to an increase in the dimensionality of the tensor automatically. This method suffers from efficiency issues if literals are utilized while generating KG embeddings. Similarly, KREAR only considers those data properties which have categorical values, i.e., fixed number of values and ignores those which take any random literals as values. One of the recent surveys [53] summarizes the methods proposed so far on refining KGs. However, this survey does not confine itself to embedding techniques and also does not consider most of the approaches which are making use of literals. Another very recent related study [57], discusses different aspects of KG embedding models such as model architectures, training strategies, and hyperparameter optimization but it takes into consideration only those models without literals.

None of the surveys mentioned above include all the existing KG embedding models which make use of literals, such as the ones categorized as models incorporating information represented in literals in Table 1. To the best of our knowledge, this is the first attempt to analyse the algorithms proposed so far for generating KG embeddings using literals. In this paper, discussions on the type of literals, the embedding approaches, and the applications/tasks on which the embedding models are evaluated are given. A categorization of the models based on the type of literals they use is also provided.

This survey is an extension of an already published short survey [24]. The major difference between the two versions is that (i) this survey contains a much more detailed theoretical analysis of the KG embedding models with literals proposed so far, and (ii) it performs empirical evaluation of the discussed models under the same experimental settings under the example of link prediction.

3. Problem formulation

This section briefly introduces the fundamentals of KGs and KG embeddings followed by a formal definition of KG embeddings with literals. It also poses various research questions about why conducting this study is a stepping stone for future development.

3.1. Preliminaries

Knowledge graphs Knowledge Graphs (KGs) consist of a set of triples $\mathit{K}\subseteq \mathit{E}\times \mathit{R}\times \mathrm{(}\mathit{E}\cup \mathit{L}\mathrm{)}$ , where E is a set of resources referred to as entities, L a set of literals, and R a set of relations. An entity is identified by a URI which represents a real-world object or an abstract concept. A relation (or property) is a binary predicate and a literal is a string, date, or number eventually followed by its data type. For a triple <h, r, t>, h is a subject, r is a relation and t is an object. The subject and object are often referred to as head and tail entity respectively. The triples consisting of literals as objects are often referred to as attributive triples.

Relations (or properties) Based on the nature of the objects, relations are classified into two main categories:

Object Relation links an entity to another entity. E.g., in the triple <dbr:Albert_Einstein, dbo:field, dbr:Physics>, both dbr:Albert_Einstein and dbr:Physics are entities, the relation dbo:field is an Object Relation.

Data Type Relation links an entity to its values, i.e., literals. For example, in <dbr:Albert_Einstein, dbo:birthDate, "1879-03-14">, where "1879-03-14" is a literal value, the relation dbo:birthDate is a Data Type Relation.

3.2. Types of literals

Literals in a KG encode additional information which is not captured by the entities or relations. There are different types of literals present in the KGs:

Text Literals: A wide variety of information can be stored in KGs in the form of free text such as names, labels, titles, descriptions, comments, etc. In most of the KG embedding models with literals, text information is further categorized into Short text and Long text . The literals which are fairly short such as for relation like names, titles, labels, etc. are considered as Short text. On the other hand, for strings that are much longer such as descriptions of entities, comments, etc. are considered as Long text and are usually provided in natural language.

Numeric Literals: Information encoded as integers, float and so on such as height, date, population, etc. also provide useful information about an entity. It is worth considering the numbers as distinct entities in the embedding models, as it has its own semantics to be covered which cannot be covered by string distance metrics. For instance, 777 is more similar to 788 than 77.

Units of Measurement: Numeric literals often denote units of measurements to a definite magnitude. For example, Wikidata property wdt:P2048 (“height”) takes values in mm, cm, m, km, inch, foot and pixel. Hence, discarding the units and considering only the numeric values without normalization results in loss of semantics, especially if units are not comparable, e.g., units of length and units of weight.

Image Literals: Images also provide latent useful information for modelling the entities. For example, a person’s details such as age, gender, etc. can be deduced via visual analysis of an image depicting the person.

Other Types of Literals: Useful information encoded in the form of other literals such as external URIs which could lead to an image, text, audio or video files.

3.3. Research questions

As it can be seen from the above discussion that the information represented in the KGs is diverse, modelling these entities is a challenging task. The challenges which are further targeted in this study are given as follows:

RQ1 – How can structured (triples with object relations) and unstructured information (attributive triples) in the KGs be combined into the representation learning?

RQ2 – How can the heterogeneity of the types of literals present in the KGs be captured and combined into representation learning?

4. Knowledge graph embeddings with literals

This section investigates KG embedding models with literals divided into the following different categories based on the types of literals utilized: (i) Text, (ii) Numeric, (iii) Image, and (iv) Multi-modal. A KG embedding model which makes use of at least two types of literals providing complementary information is considered as multi-modal. In the subsequent sections, a description of the models for each of the previously described categories analyzing their similarities and differences, followed by a discussion of potential drawbacks are provided.

4.1. Models with text literals

In this section, seven KG embedding models utilizing text literals are discussed, namely, Extended RESCAL [51], Jointly(desp) [84], DKRL [78], Jointly [80], SSP [76], KDCoE [9], and KGloVe with literals [10]. A detailed description followed by a summary presenting the comparison of these models is given along with their drawback. Moreover, in order to show the differences between the models based on complexity, the number of parameters of each model is presented in Table 3.

Table 3

Complexity of the models with text literals in terms of the number of parameters. Θ is the number of parameters in the base model, H is the entity embedding size, ${\mathit{N}}_{\mathit{d}}$ is the number of data relations, L is the number of attribute-value pairs, ${\mathit{N}}_{\mathit{r}}$ is the number of relations, ${\mathit{N}}_{\mathit{w}}$ is the number of words, ${\mathit{H}}^{\prime }$ is the word embedding size, ${\mathit{N}}_0^{\mathrm{(}1\mathrm{)}}$ is the dimension of input vectors at the first layer, ${\mathit{N}}_1^{\mathrm{(}1\mathrm{)}}$ is the dimension of output vectors at layer 1, K is window size, ${\mathit{N}}_0^{\mathrm{(}2\mathrm{)}}$ is the dimension of input vectors at the second layer, ${\mathit{N}}_1^{\mathrm{(}2\mathrm{)}}$ is the dimension of output vectors at second layer, ${\mathit{N}}_{{\mathit{e}}_1}$ and ${\mathit{N}}_{{\mathit{e}}_2}$ denote the number of entities in two different languages of a multilingual KG, ${\mathit{N}}_{{\mathit{r}}_1}$ and ${\mathit{N}}_{{\mathit{r}}_2}$ denote the number of relations in two different languages of a multilingual KG, N is the total number of entities and relations, and M is the total number of entities, relations and words

Model	#Parameter
Extended Rescal	$\mathrm{\Theta }+\mathit{H}\mathit{L}$
Jointly(Desp)	$\mathrm{\Theta }+{\mathit{N}}_{\mathit{w}}{\mathit{H}}^{\prime }$
DKRL	$\mathrm{\Theta }+{\mathit{N}}_{\mathit{w}}{\mathit{H}}^{\prime }+{\mathit{N}}_0^1\mathit{K}{\mathit{N}}_1^{\mathrm{(}1\mathrm{)}}+{\mathit{N}}_0^{\mathrm{(}2\mathrm{)}}\mathit{K}{\mathit{N}}_1^{\mathrm{(}2\mathrm{)}}$
Jointly(ALSTM)	$\mathrm{\Theta }+\mathrm{(}2\mathit{H}+4\mathrm{)}\mathit{H}$
SSP	$\mathrm{\Theta }+\mathrm{(}{\mathit{N}}_{\mathit{e}}+{\mathit{N}}_{\mathit{w}}\mathrm{)}\mathit{H}$
KDCoE	$\mathrm{(}{\mathit{N}}_{{\mathit{e}}_1}+{\mathit{N}}_{{\mathit{e}}_2}+{\mathit{N}}_{{\mathit{r}}_1}+{\mathit{N}}_{{\mathit{r}}_2}+\mathit{H}\mathrm{)}\mathit{H}+\mathrm{(}5{\mathit{H}}^{\prime }+3{\mathit{N}}_{\mathit{w}}\mathrm{)}{\mathit{H}}^{\prime }$
KGlove with literals	$\mathrm{(}\mathit{N}+1\mathrm{)}\mathit{N}+{\mathit{N}}_{\mathit{w}}+\mathit{M}$

Extended RESCAL aims to improve the original RESCAL approach by extending its algorithm to process literal values more efficiently and to deal with the drawback of sparsity that accompanies tensors. In the original RESCAL approach, relational data is modeled as a three-way tensor X of size $\mathit{n}\times \mathit{n}\times \mathit{m}$ , where n is the number of entities and m is the number of relations. An entry ${\mathit{X}}_{\mathit{i}\mathit{j}\mathit{k}}=1$ denotes the existence of the triple with i-th entity as a subject, k-th relation as a predicate, and j-th entity as an object. If ${\mathit{X}}_{\mathit{i}\mathit{j}\mathit{k}}$ is set to 0, it indicates that the triple doesn’t exist. A new approach for tensor factorization is proposed which is performed on X. For further details refer to [51]. If attributive triples have to be modeled in such a way, then the literals will be taken as entities even if they cannot occur as subject in the triples. Including literals may lead to an increment in the runtime since a larger tensor has to be factorized.

In contrast to the original algorithm, the extended RESCAL algorithm handles the attributive triples in a separate matrix. The matrix factorization is performed jointly with the tensor factorization of the non-attributive triples. The attributive triples containing only text literals are encoded in an entity-attribute matrix D in such a way that the rows are entities and the columns are $⟨\mathit{data}\mathit{type}\mathit{relation}\mathrm{,}\mathit{value}⟩$ pairs. Given a triple with a textual data type such as rdfs:label or yago:hasPreferredMeaning, one or more such pairs are created by tokenizing and stemming the text in the object literal. The matrix D is then factorized into $\mathit{D}\approx \mathit{A}\mathit{V}$ with A and V being the latent-component representations of entities and attributes respectively. Despite the advantage that this approach has for handling multi-valued literals, it does not consider the sequence of words of the literal values. Note that Extended RESCAL represents RDF(S) data in such a way that there is no distinction drawn among A-Box and T-Box, i.e., both classes and instances are modeled equally as entities in a tensor. The T-Box is rather taken as soft constraints instead of letting them impose hard constraints on the model.

Jointly(Disp) is an approach which jointly learns embeddings of KGs and a text corpus of entity descriptions, i.e, it uses an alignment model to make sure the entities, relations, and words are represented in the same vector space. This approach consists of three components, namely, knowledge model, text model, and alignment model. The knowledge model is used to capture the semantics of the structured information from the KG. Given a triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ , the model defines the plausibility of the triple, same as in [72]: $$\begin{array}{}\text{(1)}& Pr\mathrm{(}\mathit{h}|\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}={\displaystyle \frac{exp\{\mathit{z}\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}\}}{{\sum }_{\tilde{\mathit{h}}\in \mathit{I}}exp\{\mathit{z}\mathrm{(}\tilde{\mathit{h}}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}\}}}\mathrm{,}\end{array}$$ where $\mathit{z}\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}=\mathit{b}-0.5·\Vert \mathit{h}+\mathit{r}-\mathit{t}{\Vert }_2^2$ , $\mathit{b}=7$ . Analogously, $Pr\mathrm{(}\mathit{r}|\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}$ and $Pr\mathrm{(}\mathit{t}|\mathit{h}\mathrm{,}\mathit{r}\mathrm{)}$ are defined.

Then, the loss function of the knowledge model is defined as follows: $$\begin{array}{}\text{(2)}& \begin{array}{cc}\hfill {\mathit{L}}_{\mathit{K}}& ={\displaystyle \sum }_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}}[logPr\mathrm{(}\mathit{h}|\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}+logPr\mathrm{(}\mathit{t}|\mathit{h}\mathrm{,}\mathit{r}\mathrm{)}\hfill \\ \hfill & +logPr\mathrm{(}\mathit{r}|\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}].\hfill \end{array}\end{array}$$

The text model adopts the same assumption made in [72] that is if two words occur in the same context then there is a relation between them. Based on this assumption, the text model defines the probability of a pair of words w and v co-occurring in a text window as follows: $$\begin{array}{}\text{(3)}& Pr\mathrm{(}\mathit{w}|\mathit{v}\mathrm{)}={\displaystyle \frac{exp\{\mathit{z}\mathrm{(}\mathit{w}\mathrm{,}\mathit{v}\mathrm{)}\}}{{\sum }_{\tilde{\mathit{w}}\in \mathit{V}}exp\{\mathit{z}\mathrm{(}\tilde{\mathit{w}}\mathrm{,}\mathit{v}\mathrm{)}\}}}\mathrm{,}\end{array}$$ where $\mathit{z}\mathrm{(}\mathit{w}\mathrm{,}\mathit{v}\mathrm{)}=\mathit{b}-0.5·\Vert \mathbf{\text{w}}-\mathbf{\text{v}}{\Vert }_2^2$ . Then, the loss function of the text model is given as: $$\begin{array}{}\text{(4)}& {\mathit{L}}_{\mathit{T}}={\displaystyle \sum }_{\mathrm{(}\mathit{w}\mathrm{,}\mathit{v}\mathrm{)}}logPr\mathrm{(}\mathit{w}|\mathit{v}\mathrm{)}.\end{array}$$ The role of the third component, the alignment model, is to put the embeddings of the entities, relations, and words into the same vector space. This submodel works by utilizing entity descriptions to align these embeddings. For every word w in the description of entity e, the conditional probability of predicting w given e is defined as: $$\begin{array}{}\text{(5)}& Pr\mathrm{(}\mathit{w}|\mathit{e}\mathrm{)}={\displaystyle \frac{exp\{\mathit{z}\mathrm{(}\mathit{e}\mathrm{,}\mathit{w}\mathrm{)}\}}{{\sum }_{\tilde{\mathit{w}}\in \mathit{V}}exp\{\mathit{z}\mathrm{(}\mathit{e}\mathrm{,}\tilde{\mathit{w}}\mathrm{)}\}}}\mathrm{,}\end{array}$$ where $\mathit{z}\mathrm{(}\mathit{e}\mathrm{,}\mathit{w}\mathrm{)}=\mathit{b}-0.5·\Vert \mathbf{\text{e}}-\mathbf{\text{w}}{\Vert }_2^2$ . The entity vector $\mathbf{\text{e}}$ in Eq. (5) is the same as the entity vector appearing in Eq. (1), i.e., an entity has a single unified representation which captures the semantics from both the structured KG and the entity descriptions. $Pr\mathrm{(}\mathit{e}|\mathit{w}\mathrm{)}$ is defined analogously. Based on the definition given in Eq. (5), the loss function of the alignment model is defined as: $$\begin{array}{}\text{(6)}& {\mathit{L}}_{\mathit{A}}={\displaystyle \sum }_{\mathit{e}\in \mathcal{E}}{\displaystyle \sum }_{\mathit{w}\in {\mathit{D}}_{\mathit{e}}}[logPr\mathrm{(}\mathit{w}|\mathit{e}\mathrm{)}+logPr\mathrm{(}\mathit{e}|\mathit{w}\mathrm{)}]\mathrm{,}\end{array}$$ where $\mathcal{E}$ and ${\mathit{D}}_{\mathit{e}}$ denote the set of entities and the description of the entity e respectively.

By adopting the joint embedding framework in [72], the main loss of Jointly(desp) is defined as follows: $$\begin{array}{}\text{(7)}& \mathit{L}\mathrm{(}\{{\mathit{e}}_{\mathit{i}}\}\mathrm{,}\{{\mathit{r}}_{\mathit{j}}\}\mathrm{,}\{{\mathit{w}}_{\mathit{l}}\}\mathrm{)}={\mathit{L}}_{\mathit{K}}+{\mathit{L}}_{\mathit{T}}+{\mathit{L}}_{\mathit{A}}.\end{array}$$ DKRL extends TransE [5] by utilizing the descriptions of entities. For each entity e, two kinds of vector representations are learned, i.e., structure-based ${\mathit{e}}_{\mathit{s}}$ and description-based ${\mathit{e}}_{\mathit{d}}$ . These two kinds of entity representations are learned simultaneously into the same vector space but not forced to be unified so that novel entities with only descriptions can be represented. In order to achieve this, given a certain triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ the energy function of the DKRL model is defined as: $$\begin{array}{}\text{(8)}& \begin{array}{cc}\hfill \mathit{E}& =\Vert {\mathit{h}}_{\mathit{s}}+\mathit{r}-{\mathit{t}}_{\mathit{s}}\Vert +\Vert {\mathit{h}}_{\mathit{d}}+\mathit{r}-{\mathit{t}}_{\mathit{d}}\Vert \hfill \\ \hfill & +\Vert {\mathit{h}}_{\mathit{s}}+\mathit{r}-{\mathit{t}}_{\mathit{d}}\Vert +\Vert {\mathit{h}}_{\mathit{d}}+\mathit{r}-{\mathit{t}}_{\mathit{s}}\Vert \mathrm{,}\hfill \end{array}\end{array}$$ where ${\mathit{h}}_{\mathit{s}}$ and ${\mathit{t}}_{\mathit{s}}$ are the structure-based representations, and ${\mathit{h}}_{\mathit{d}}$ and ${\mathit{t}}_{\mathit{d}}$ are the description-based representations of their corresponding entities.

In order to learn structure-based representations, the TransE approach is directly applied which considers the relation in a triple as the translation from the head entity to the tail entity. On the other hand, Continuous Bag of Words (CBOW) and a deep Convolutional Neural Network (CNN) model have been used to generate the description-based representations of the head and tail entities. In case of CBOW, short text is generated from the description based on keywords and their corresponding word embeddings are summed up to generate the entity embedding. In the CNN model, after preprocessing the description, pretrained word vectors from Wikipedia are provided as input. This CNN model has five layers and after every convolutional layer pooling is applied to decrease the parameter space of CNN and filter noises. Max-pooling is applied for the first pooling and mean pooling for the last one. The activation function used is either tanh or ReLU. The CNN model works better than CBOW because it preserves the sequence of words.

In order to train DKRL, the following margin-based score function is considered as an objective function and minimized using a standard back propagation using stochastic gradient descent (SGD) $$\begin{array}{}\text{(9)}& \begin{array}{cc}\hfill \mathit{L}& ={\displaystyle \sum }_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}\in \mathit{T}}{\displaystyle \sum }_{\mathrm{(}{\mathit{h}}^{\prime }\mathrm{,}{\mathit{r}}^{\prime }\mathrm{,}{\mathit{t}}^{\prime }\mathrm{)}\in {\mathit{T}}^{\prime }}max\mathrm{(}\mathit{\gamma }+\mathit{d}\mathrm{(}\mathit{h}+\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}\hfill \\ \hfill & -\mathit{d}\mathrm{(}{\mathit{h}}^{\prime }+{\mathit{r}}^{\prime }\mathrm{,}{\mathit{t}}^{\prime }\mathrm{)}\mathrm{,}0\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ where $\mathit{\gamma }\mathrm{>}0$ is a margin hyperparameter, d is a dissimilarity function and ${\mathit{T}}^{\prime }$ is the set of corrupted triples. The representation of the entities can be either structure-based or description-based.

Jointly [80] learns KG embeddings by leveraging entity descriptions. Specifically, it learns a joint embedding of an entity by combining its structure-based and description-based representations with a gating mechanism. The gate is used to find balance between the structure-based and the description-based representations. For a certain entity a representation can be encoded from its descriptions by converting the description into fixed length vector. In Jointly, different text encoders have been used such as bag-of-words, LSTM, and Attentive LSTM.

For an entity e, its joint representation e is a linear interpolation between its structure-based representation $\mathrm{(}{\mathbf{\text{e}}}_{\mathbf{\text{s}}}\mathrm{)}$ and description-based representation $\mathrm{(}{\mathbf{\text{e}}}_{\mathbf{\text{d}}}\mathrm{)}$ , which is defined as: $$\begin{array}{}\text{(10)}& \mathbf{\text{e}}={\mathit{g}}_{\mathit{e}}\odot {\mathbf{\text{e}}}_{\mathbf{\text{s}}}+\mathrm{(}1-{\mathit{g}}_{\mathit{e}}\mathrm{)}\odot {\mathbf{\text{e}}}_{\mathbf{\text{d}}}\mathrm{,}\end{array}$$ where ⊙ is an element-wise multiplication and ${\mathit{g}}_{\mathit{e}}$ is a gate to balance the two information sources (structure and text) which is computed as ${\mathit{g}}_{\mathit{e}}=\mathit{\alpha }\mathrm{(}\widetilde{{\mathit{g}}_{\mathit{e}}}\mathrm{)}$ with ${\mathit{g}}_{\mathit{e}}=\widetilde{{\mathit{g}}_{\mathit{e}}}\in {\mathcal{R}}^{\mathit{d}}$ being real-value vector stored in a lookup table.

The entity descriptions are encoded using either bag-of-words, LSTM, or Attentive LSTM (ALSTM) encoders in order to generate text-based representation for the corresponding entities. On the other hand, to better model the structure-based embedidngs, entities and relations can be pre-trained with any existing KG embedding models, such as TransE.

Jointly’s score function is inspired by TransE and defined as follows: $$\begin{array}{}\text{(11)}& \begin{array}{cc}\hfill & \mathit{f}\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t};{\mathit{d}}_{\mathit{h}}\mathrm{,}{\mathit{d}}_{\mathit{t}}\mathrm{)}\hfill \\ \hfill & =\Vert \mathrm{(}{\mathbf{\text{g}}}_{\mathit{h}}\odot {\mathbf{\text{h}}}_{\mathit{s}}+\mathrm{(}1-{\mathbf{\text{g}}}_{\mathit{h}}\mathrm{)}\odot {\mathbf{\text{h}}}_{\mathit{d}}\mathrm{)}\hfill \\ \hfill & +\mathit{r}-\mathrm{(}{\mathit{g}}_{\mathit{t}}\odot {\mathbf{\text{h}}}_{\mathit{t}}+\mathrm{(}1-{\mathbf{\text{g}}}_{\mathit{t}}\mathrm{)}\odot {\mathbf{\text{t}}}_{\mathit{d}}\mathrm{)}\Vert {}_2^2.\hfill \end{array}\end{array}$$ where ${\mathbf{\text{h}}}_{\mathit{s}}$ , ${\mathbf{\text{h}}}_{\mathit{d}}$ , and ${\mathbf{\text{g}}}_{\mathit{h}}$ are the head entity’s structure-based embedding, description-based embedding, and gate respectively whereas ${\mathbf{\text{t}}}_{\mathit{s}}$ , ${\mathbf{\text{t}}}_{\mathit{d}}$ , and ${\mathbf{\text{g}}}_{\mathit{t}}$ are the tail entity’s structure-based embedding, description-based embedding, and gate respectively.

SSP (Semantic Space Projection) [76] is a joint embedding model which learns from both structured/symbolic triples and textual descriptions. Differently from DKRL and Jointly(Desp), where first-order constraints which are weak in capturing the correlation of textual descriptions and symbolic triples are applied, SSP follows the principle that triple embedding is considered always as the main procedure and textual descriptions must interact with triples in order to learn better representation. Therefore, triple embedding is projected onto a semantic subspace such as a hyperplane to allow strong correlation by adopting quadratic constraint.

SSP applies the following scoring function: $$\begin{array}{}\text{(12)}& {\mathit{f}}_{\mathit{r}}\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}=-\mathit{\lambda }{\Vert \mathbf{\text{e}}-{\mathbf{\text{s}}}^{\mathit{T}}\mathbf{\text{es}}\Vert }_2^2+\Vert \mathbf{\text{e}}{\Vert }_2^2\mathrm{,}\end{array}$$ where $$\begin{array}{}\text{(13)}& \mathbf{\text{e}}\doteq \mathbf{\text{h}}+\mathbf{\text{r}}-\mathbf{\text{t}}\mathrm{,}\end{array}$$ and $$\begin{array}{}\text{(14)}& \mathbf{\text{s}}\doteq {\displaystyle \frac{{\mathbf{\text{s}}}_{\mathbf{\text{h}}}+{\mathbf{\text{s}}}_{\mathbf{\text{t}}}}{\Vert {\mathbf{\text{s}}}_{\mathbf{\text{h}}}+{\mathbf{\text{s}}}_{\mathbf{\text{t}}}\Vert }}.\end{array}$$ Note that λ is a suitable hyper-parameter, h and t are the structure (symbolic triples) based embedding of the head and tail entities respectively, ${\mathbf{\text{s}}}_{\mathbf{\text{h}}}$ and ${\mathbf{\text{s}}}_{\mathbf{\text{t}}}$ are the semantic vectors generated from the textual descriptions of the head and tail entities respectively. SSE adopts the Non-negative Matrix Factorization (NMF) topic model to generate description-based semantic vectors for entities ( ${\mathbf{\text{s}}}_{\mathbf{\text{h}}}$ and ${\mathbf{\text{s}}}_{\mathbf{\text{t}}}$ ), i.e., by treating each entity description as a document and taking the topic distribution of the document as the representation of the corresponding entity.

SSP provides two different settings for training which are referred to as Std and Joint. In Std, a pre-trained topic model with NMF is used to obtain description-based semantic vectors. These description-based vectors are fixed during training but the other parameters are optimized. On the other hand, in the Joint setting the topic model is also learnt simultaneously with the KG embeddings instead of using a fixed pre-trained vectors.

KDCoE focuses on the creation of an alignment between entities of multilingual KGs by creating new Inter-Lingual Links (ILLs) based on an embedding approach which exploits entity descriptions. The model uses a weakly aligned multilingual KG for semi-supervised cross-lingual learning. It performs co-training of a multilingual KG embedding Model (KGEM) and a multilingual entity Description Embedding Model (DEM) iteratively in order for each model to propose a new ILL alternately. KGEM is composed of two components, i.e., a knowledge model and an alignment model, to learn embeddings based on structured information from the KGs (the non attributive triples). Given a set of languages $\mathcal{L}$ , a separate ${\mathit{k}}_1$ -dimensional embedding space ${\mathbb{R}}_{\mathit{L}}^{{\mathit{k}}_1}$ is used for each language $\mathit{L}\in \mathcal{L}$ to represent the corresponding relations ${\mathit{R}}_{\mathit{L}}$ and entities ${\mathit{E}}_{\mathit{L}}$ . In order to learn the embeddings for ${\mathit{R}}_{\mathit{L}}$ and ${\mathit{E}}_{\mathit{L}}$ , the knowledge model adopts TransE and thus uses hinge loss as its objective function. On the other hand, a linear-transformation-based technique which has the best performance in case of cross-lingual inferences is adopted for the alignment model. This technique employs the following objective function: $$\begin{array}{}\text{(15)}& {\mathit{S}}_{\mathit{A}}={\displaystyle \sum }_{\mathrm{(}\mathit{e}\mathrm{,}{\mathit{e}}^{\prime }\mathrm{)}\in \mathit{I}\mathrm{(}{\mathit{L}}_{\mathit{i}}\mathrm{,}{\mathit{L}}_{\mathit{j}}\mathrm{)}}{\Vert {\mathit{M}}_{\mathit{i}\mathit{j}}\mathbf{\text{e}}-{\mathbf{\text{e}}}^{\prime }\Vert }_2\mathrm{,}\end{array}$$ where $\mathit{I}\mathrm{(}{\mathit{L}}_{\mathit{i}}\mathrm{,}{\mathit{L}}_{\mathit{j}}\mathrm{)}$ is ILLs between the languages ${\mathit{L}}_{\mathit{i}}$ and ${\mathit{L}}_{\mathit{j}}$ , and ${\mathit{M}}_{\mathit{i}\mathit{j}}$ is a ${\mathit{k}}_1\times {\mathit{k}}_1$ matrix used as a linear transformation on entity vectors from ${\mathit{L}}_{\mathit{i}}$ to ${\mathit{L}}_{\mathit{j}}$ .

Let ${\mathit{S}}_{\mathit{K}}$ be the hinge loss function used by the knowledge model, the KGEM model then minimizes ${\mathit{S}}_{\mathrm{KG}}={\mathit{S}}_{\mathit{K}}+\mathit{\alpha }{\mathit{S}}_{\mathit{A}}$ , where α is a positive hyperparameter. In case of DEM model, an attentive gated recurrent unit encoder (AGRU) is used to encode the multilingual entity descriptions. DEM applies multilingual word embeddings in order to capture the semantic information of multilingual entity descriptions from the word level. The two models, i.e., KGEM and DEM, are iteratively co-trained in order for each model to propose a new ILL alternately.

KGloVe with literals is an experimental attempt to incorporate entity descriptions in KGloVe KG embedding approach. The experiment is conducted on DBpedia considering the abstracts and comments of entities as their descriptions. The main goal is to extract named entities from the textual description and for every entity in the text, to replace those words representing it with the entity itself and then take its neighbouring words and entities as its context. The approach works by creating two co-occurrence matrices independently and then by merging them at the end so that a joint embedding can be performed. The first matrix is generated using the same technique as in KGloVe [11], i.e., by performing Personalized PageRank (PPR) on the (weighted) graph followed by the same optimisation used in the GloVe [54] approach.

In order to create the second matrix, the Named Entity Recognition (NER) task is performed on the entity description text using the list of entities and predicates of the KG as an input. The NER step employs a simple exact string matching technique which leads to numerous drawbacks such as missing entities due to having different keywords with the same semantics. All the English words that do not match any entity labels are added to the entity-predicate list. Then GloVe co-occurrence for text is applied to the modified text (i.e., DBpedia abstract and comments) using the entity-predicate and word list as input. Finally, the two co-occurrence matrices are summed up together to create a single unified matrix. The proposed approach has been evaluated on classification and regression tasks and the result indicates that for most of the classifiers used, except SVM, the approach does not bring significant improvement to KGloVe. However, the approach can be improved using parameter tuning with extensive experiments.

Summary The basic differences between these models lie in the methods used to exploit the information given in the text literals and combine them with structure-based representation. One major advantage of KDCoE over text literal based embedding models is that it considers descriptions present in multilingual KGs. Also, both DKRL and KDCoE embedding models are designed to perform well for the novel entities, which have only attributive triples in the KGs. Jointly(Desp) aligns KG embeddings and word embeddings on word level, which may lead to losing some semantic information on phrase or sentence level. Jointly applies a gating mechanism which allows to automatically find a balance between the structural and textual information. It also uses an LSTM encoder which enables the model to select the most related information for an entity from its text description according to different relations. Unlike DKRL and Jointly(Desp), SSP focuses on characterizing the stronger correlations between entity descriptions and structured triples by projecting triple embedding onto a semantic subspace such as a hyper-plane, as discussed above. Another common drawbacks among the presented approaches with text literals is they focus mostly on descriptions, which is long natural language text and thus, other types of text literals, such as, names, labels, titles, etc. are not widely considered. Moreover, another way to compare these approaches is by looking at their model complexity. Table 3 presents the complexity of these models in terms of their number of parameters.

4.2. Models with numeric literals

In this section, the analysis of the presented KG embedding models which use numeric literals, namely, MT-KGNN [63], KBLRN [22], LiteralE [37], and TransEA [75] are presented followed by a summary. Moreover, in order to show the differences between the models based on complexity, the number of parameters of each model is presented in Table 4.

Table 4

Complexity of the models with numerical literals in terms of the number of parameters. Θ is the number of parameters in the base model, H is the entity embedding size, ${\mathit{N}}_{\mathit{d}}$ is the number of data relations, Λ is the size of the hidden layer in the Attrnet networks of MTKGNN, ${\mathit{N}}_{\mathit{r}}$ is the number of relations, and M is attribute embedding size

Model	#Parameters
LiteralE with g	$\mathrm{\Theta }+2{\mathit{H}}^2+2{\mathit{N}}_{\mathit{d}}\mathit{H}+\mathit{H}$
LiteralE with ${\mathit{g}}_{\mathrm{lin}}$	$\mathrm{\Theta }+\mathrm{(}{\mathit{N}}_{\mathit{d}}\mathit{H}+\mathit{H}\mathrm{)}\mathit{H}$
MTKGNN	$\mathrm{\Theta }+{\mathit{N}}_{\mathit{d}}\mathit{H}+2\mathrm{(}2\mathit{H}\mathrm{\Lambda }+\mathrm{\Lambda }\mathrm{)}$
KBLN	$\mathrm{\Theta }+{\mathit{N}}_{\mathit{r}}{\mathit{N}}_{\mathit{d}}$
TransEA	$\mathrm{\Theta }+{\mathit{N}}_{\mathit{d}}\mathit{M}$

MT-KGNN is an approach for both relational learning and non-discrete attribute prediction on knowledge graphs in order to learn embeddings for entities, object properties, and data properties. It is composed of two networks, namely, the Relational Network (RelNet) and the Attribute Network (AttrNet). RelNet is a binary (pointwise) classifier for triple prediction whereas AttrNet is a regression task for attribute value prediction. Given n, m, and l as entity, relation, and literal embedding dimensions respectively, the model passes as an input $[{\mathit{e}}_{\mathit{i}}\mathrm{,}{\mathit{r}}_{\mathit{k}}\mathrm{,}{\mathit{e}}_{\mathit{j}}\mathrm{,}\mathit{t}]$ to RelNet and $[{\mathit{a}}_{\mathit{i}}\mathrm{,}{\mathit{v}}_{\mathit{i}}\mathrm{,}{\mathit{a}}_{\mathit{j}}\mathrm{,}{\mathit{v}}_{\mathit{j}}]$ to AttrNet, where ${\mathit{e}}_{\mathit{i}}$ , ${\mathit{e}}_{\mathit{j}}\in {\mathit{R}}^{\mathit{n}}$ , ${\mathit{r}}_{\mathit{k}}\in {\mathit{R}}^{\mathit{m}}$ , t is the classification target which is 0 or 1, ${\mathit{a}}_{\mathit{i}}\mathrm{,}{\mathit{a}}_{\mathit{j}}\in {\mathit{R}}^{\mathit{l}}$ , and ${\mathit{v}}_{\mathit{i}}$ and ${\mathit{v}}_{\mathit{j}}$ are normalized continuous values in the interval $[0\mathrm{,}1]$ . Note that the inputs to AttrNet, i.e., $[{\mathit{a}}_{\mathit{i}}\mathrm{,}{\mathit{v}}_{\mathit{i}}\mathrm{,}{\mathit{a}}_{\mathit{j}}\mathrm{,}{\mathit{v}}_{\mathit{j}}]$ , are taken from attributive triples with non-discrete literal values. An embedding lookup layer is used to retrieve the corresponding vector representations given these inputs as one-hot encoded indices.

In RelNet, a concatenated triple is passed through a nonlinear transform and then a sigmoid function is applied to get a linear transform: $$\begin{array}{}\text{(16)}& \begin{array}{cc}\hfill & {\mathit{g}}_{\mathrm{rel}}\mathrm{(}{\mathit{e}}_{\mathit{i}}\mathrm{,}{\mathit{r}}_{\mathit{k}}\mathrm{,}{\mathit{e}}_{\mathit{j}}\mathrm{)}\hfill \\ \hfill & =\mathit{\sigma }\mathrm{\left(}{\overrightarrow{\mathit{w}}}^{\mathit{T}}\mathit{f}\mathrm{\right(}{\mathit{W}}_{\mathit{d}}^{\mathit{T}}[\overrightarrow{{\mathit{e}}_{\mathit{i}}};\overrightarrow{{\mathit{e}}_{\mathit{j}}};\overrightarrow{{\mathit{r}}_{\mathit{k}}}]\mathrm{)}+{\mathit{b}}_{\mathrm{rel}}\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ where $\mathit{w}\in {\mathit{R}}^{\mathit{h}\times 1}$ and ${\mathit{W}}_{\mathit{d}}\in {\mathit{R}}^{3\mathit{n}\times \mathit{h}}$ are parameters of the network. σ, f, and ${\mathit{b}}_{\mathrm{rel}}$ are the sigmoid function, the hyperbolic tangent function tanh, and a scalar bias respectively. RelNet is trained by minimizing the following cross entropy loss function: $$\begin{array}{}\text{(17)}& \begin{array}{cc}\hfill {\mathit{L}}_{\mathrm{rel}}& =-{\displaystyle \sum }_{\mathit{i}=1}^{\mathit{N}}{\mathit{t}}_{\mathit{i}}log{\mathit{g}}_{\mathrm{rel}}\mathrm{(}{\mathit{\xi }}_{\mathit{i}}\mathrm{)}\hfill \\ \hfill & +\mathrm{(}1-{\mathit{t}}_{\mathit{i}}\mathrm{)}log\mathrm{(}1-{\mathit{g}}_{\mathrm{rel}}\mathrm{(}{\mathit{\xi }}_{\mathit{i}}\mathrm{)}\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ where ${\mathit{\xi }}_{\mathit{i}}$ denotes triple i in batch of size N and ${\mathit{t}}_{\mathit{i}}$ takes the value 0 or 1. In case of AttrNet, two regression tasks are performed, one for the head data properties and another for those of the tail. The following scoring functions are defined for these two tasks: $$\begin{array}{}\text{(18)}& {\mathit{g}}_{\mathit{h}}\mathrm{(}{\mathit{a}}_{\mathit{i}}\mathrm{)}=\mathit{\sigma }\mathrm{\left(}{\overrightarrow{\mathit{u}}}^{\mathit{T}}\mathit{f}\mathrm{\right(}{\mathit{B}}^{\mathit{T}}[\overrightarrow{{\mathit{a}}_{\mathit{i}}};\overrightarrow{{\mathit{e}}_{\mathit{i}}}]\mathrm{)}+{\mathit{b}}_{{\mathit{z}}_1}\mathrm{)}\mathrm{,}\end{array}$$ and $$\begin{array}{}\text{(19)}& {\mathit{g}}_{\mathit{t}}\mathrm{(}{\mathit{a}}_{\mathit{j}}\mathrm{)}=\mathit{\sigma }\mathrm{\left(}{\overrightarrow{\mathit{y}}}^{\mathit{T}}\mathit{f}\mathrm{\right(}{\mathit{C}}^{\mathit{T}}[\overrightarrow{{\mathit{a}}_{\mathit{j}}};\overrightarrow{{\mathit{e}}_{\mathit{j}}}]\mathrm{)}+{\mathit{b}}_{{\mathit{z}}_2}\mathrm{)}\mathrm{,}\end{array}$$ where $\mathit{u}\mathrm{,}\mathit{y}\in {\mathit{R}}^{{\mathit{h}}_{\mathit{a}}\times 1}$ and $\mathit{B}\mathrm{,}\mathit{C}\in {\mathit{R}}^{2\mathit{n}\times {\mathit{h}}_{\mathit{a}}}$ are parameters of AttrNet. ${\mathit{h}}_{\mathit{a}}$ is the size of the hidden layer and ${\mathit{b}}_{{\mathit{z}}_1}$ , ${\mathit{b}}_{{\mathit{z}}_2}$ are scalar biases. Each AttrNet is trained by optimizing Mean Squared Error (MSE) loss function: $$\begin{array}{}\text{(20)}& MSE\mathrm{(}\mathit{s}\mathrm{,}{\mathit{s}}^{\ast }\mathrm{)}={\displaystyle \frac{1}{\mathit{N}}}{\displaystyle \sum }_{\mathit{i}=1}^{\mathit{N}}{\mathrm{(}{\mathit{s}}_{\mathit{i}}-{\mathit{s}}_{\mathit{i}}^{\ast }\mathrm{)}}^2.\end{array}$$ where s and ${\mathit{s}}^{\ast }$ are predicted labels (scores computed by the model) and ground truth labels respectively.

The overall loss of the AttrNet is computed by adding the MSE of the head AttrNet and that of the tail AttrNet as follows: $$\begin{array}{}\text{(21)}& \begin{array}{cc}\hfill {\mathit{L}}_{\mathrm{attr}}& =MSE\mathrm{(}{\mathit{g}}_{\mathit{h}}\mathrm{(}{\mathit{a}}_{\mathit{i}}\mathrm{)}\mathrm{,}{\mathrm{(}{\mathit{a}}_{\mathit{i}}\mathrm{)}}^{\ast }\mathrm{)}\hfill \\ \hfill & +MSE\mathrm{(}{\mathit{g}}_{\mathit{t}}\mathrm{(}{\mathit{a}}_{\mathit{j}}\mathrm{)}\mathrm{,}{\mathrm{(}{\mathit{a}}_{\mathit{j}}\mathrm{)}}^{\ast }\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ where ${\mathrm{(}{\mathit{a}}_{\mathit{i}}\mathrm{)}}^{\ast }$ , ${\mathrm{(}{\mathit{a}}_{\mathit{j}}\mathrm{)}}^{\ast }$ are the ground truth labels. Finally, the two networks are trained in a multi-task fashion using a shared embedding space.

KBLRN works by combining relational (R), latent (L), and numerical (N) features together. The model is designed mainly for the purpose of KG completion. It uses a probabilistic PoE (Product of Experts) method to combine these feature types and train them jointly end to end. Each relational feature is formulated as a logical formula, by adopting the rule mining approach AMIE+ [20], to be evaluated in the KB to compute the feature’s value. The latent features are the ones that are usually generated using an embedding approach such as DistMult. Numerical features are used with the assumption that, for some relation types, the differences between the head and tail can be seen as characteristics for the relation itself. Given a triple $\mathit{d}=\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}$ , for each (relation type r, and feature type $\mathit{F}\in \{\mathit{L}\mathrm{,}\mathit{R}\mathrm{,}\mathit{N}\}$ ) pair, individual experts are defined based on linear models and DistMult embedding method as follows: $$\begin{array}{}\text{(22)}& {\mathit{f}}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{L}\mathrm{)}}\mathrm{(}\mathit{d}\mid {\mathit{\theta }}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{L}\mathrm{)}}\mathrm{)}=exp\mathrm{(}\mathrm{(}{\mathit{e}}_{\mathit{h}}\ast {\mathit{e}}_{\mathit{t}}\mathrm{)}·{\mathit{w}}^{\mathit{r}}\mathrm{)}\mathrm{,}\text{(23)}& {\mathit{f}}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{R}\mathrm{)}}\mathrm{(}\mathit{d}\mid {\mathit{\theta }}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{R}\mathrm{)}}\mathrm{)}=exp\mathrm{(}{\mathit{r}}_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}}·{\mathit{w}}_{\mathrm{rel}}^{\mathit{r}}\mathrm{)}\mathrm{,}\text{(24)}& {\mathit{f}}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{N}\mathrm{)}}\mathrm{(}\mathit{d}\mid {\mathit{\theta }}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{N}\mathrm{)}}\mathrm{)}=exp\mathrm{(}\mathit{\varphi }\mathrm{(}{\mathit{n}}_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}}\mathrm{)}·{\mathit{w}}_{\mathrm{num}}^{\mathit{r}}\mathrm{)}\mathrm{,}\end{array}$$ and $$\begin{array}{}\text{(25)}& {\mathit{f}}_{\mathrm{(}{\mathit{r}}^{\prime }\mathrm{,}\mathit{F}\mathrm{)}}\mathrm{(}\mathit{d}\mid {\mathit{\theta }}_{\mathrm{(}{\mathit{r}}^{\prime }\mathrm{,}\mathit{F}\mathrm{)}}\mathrm{)}=1\text{for all }{\mathit{r}}^{\prime }\ne \mathit{r}\end{array}$$ where ${\mathit{w}}_{\mathit{r}}$ , ${\mathit{w}}_{\mathrm{rel}}^{\mathit{r}}$ , ${\mathit{w}}_{\mathrm{num}}^{\mathit{r}}$ are the parameter vectors for the latent, relational, and numerical features corresponding to the relation r. Also, * is the element-wise product, · is the dot product, and ϕ is the radial basis function (RBF) applied element-wise to the differences of values ${\mathit{n}}_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}}$ computed as follows: $$\begin{array}{}\text{(26)}& \begin{array}{cc}\hfill \mathit{\varphi }\mathrm{(}{\mathit{n}}_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}}\mathrm{)}& =[exp\mathrm{(}{\displaystyle \frac{-\Vert {\mathit{n}}_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}}^{\mathrm{(}1\mathrm{)}}-{\mathit{c}}_1{\Vert }_2^2}{{\mathit{\sigma }}_1^2}}\mathrm{)}\hfill \\ \hfill & \dots exp\mathrm{\left(}{\displaystyle \frac{-\Vert {\mathit{n}}_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}}^{\mathrm{(}{\mathit{d}}_{\mathit{n}}\mathrm{)}}-{\mathit{c}}_{{\mathit{d}}_{\mathit{n}}}{\Vert }_2^2}{{\mathit{\sigma }}_{{\mathit{d}}_{\mathit{n}}}^2}}\mathrm{\right)}].\hfill \end{array}\end{array}$$ Here, ${\mathit{d}}_{\mathit{n}}$ corresponds to the relevant numerical features. A PoE’s probability distribution for a triple $\mathit{d}=\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}$ is defined as follows: $$\begin{array}{}\text{(27)}& \begin{array}{cc}\hfill & \mathit{p}\mathrm{(}\mathit{d}\mid {\mathit{\theta }}_1\dots {\mathit{\theta }}_{\mathit{n}}\mathrm{)}\hfill \\ \hfill & ={\displaystyle \frac{{\mathrm{\Pi }}_{\mathit{F}}{\mathit{f}}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{F}\mathrm{)}}\mathrm{(}\mathit{d}\mid {\mathit{\theta }}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{F}\mathrm{)}}\mathrm{)}}{{\sum }_{\mathit{c}}{\mathrm{\Pi }}_{\mathit{F}}{\mathit{f}}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{F}\mathrm{)}}\mathrm{(}\mathit{c}\mid {\mathit{\theta }}_{\mathrm{(}\mathit{r}\mathrm{,}\mathit{F}\mathrm{)}}\mathrm{)}}}\mathrm{,}\hfill \end{array}\end{array}$$ where c denotes all possible triples. The parameters of the entity embedding model are shared by all the experts in order to create dependencies between them. In this approach, the PoE are trained with negative sampling and a cross entropy loss to give high probability to observed triples.

LiteralE incorporates literals into existing latent feature models designed for link prediction. In this approach, without loss of generality, the focus lies on incorporating numerical literals into three state-of-the-art embedding methods: DistMult, ComplEx, and ConvE. Given a base model, for instance Distmult, LiteralE modifies the scoring function f used in Distmult by replacing the vector representations of the entities ${\mathit{e}}_{\mathit{i}}$ in f with literal enriched representations ${\mathit{e}}_{\mathit{i}}^{\mathrm{lit}}$ . In order to generate ${\mathit{e}}_{\mathit{i}}^{\mathrm{lit}}$ , LiteralE uses a learnable transformation function g which takes ${\mathit{e}}_{\mathit{i}}$ and its corresponding literal vectors ${\mathit{l}}_{\mathit{i}}$ as inputs and maps them to a new vector. The function g is defined, as shown below, based on the concept of GRU in order to make it flexible, learnable, and capable to decide, if it is beneficial to incorporate the literal information or not: $$\begin{array}{}\text{(28)}& \mathit{g}:{\mathbb{R}}^{\mathit{H}}\times {\mathbb{R}}^{{\mathit{N}}_{\mathit{d}}}\to {\mathbb{R}}^{\mathit{H}}\mathrm{,}\end{array}$$ and $$\begin{array}{}\text{(29)}& \mathbf{\text{e}}\mathrm{,}\mathbf{\text{l}}\mapsto \mathbf{\text{z}}\odot \mathbf{\text{h}}+\mathrm{(}1-\mathbf{\text{z}}\mathrm{)}\odot \mathbf{\text{e}}\mathrm{,}\end{array}$$ where $$\begin{array}{}\text{(30)}& \mathbf{\text{z}}:\mathit{\sigma }\mathrm{(}{\mathbf{\text{W}}}_{\mathit{z}\mathit{e}}^{\mathit{T}}\mathbf{\text{e}}+{\mathbf{\text{W}}}_{\mathit{z}\mathit{l}}^{\mathit{T}}\mathbf{\text{l}}+\mathbf{\text{b}}\mathrm{)}\mathrm{,}\end{array}$$ and $$\begin{array}{}\text{(31)}& \mathbf{\text{h}}=\mathit{h}\mathrm{(}{\mathbf{\text{W}}}_{\mathit{h}}^{\mathit{T}}[\mathbf{\text{e}}\mathrm{,}\mathbf{\text{l}}]\mathrm{)}.\end{array}$$ Note that ${\mathbf{\text{W}}}_{\mathit{z}\mathit{e}}\in {\mathbb{R}}^{\mathit{H}\times \mathit{H}}$ , ${\mathbf{\text{W}}}_{\mathit{z}\mathit{l}}\in {\mathbb{R}}^{{\mathit{N}}_{\mathit{d}}\times \mathit{H}}$ , $\mathbf{\text{b}}\in {\mathbb{R}}^{\mathit{H}}$ , and ${\mathbf{\text{W}}}_{\mathit{h}}\in {\mathbb{R}}^{\mathit{H}+{\mathit{N}}_{\mathit{d}}\times \mathit{H}}$ are the parameters of g, σ is the sigmoid function, ⊙ denotes the element-wise multiplication, and h is a component-wise nonlinearity. The scoring function $\mathit{f}\mathrm{(}{\mathbf{\text{e}}}_{\mathit{i}}\mathrm{,}{\mathbf{\text{e}}}_{\mathit{j}}\mathrm{,}{\mathbf{\text{r}}}_{\mathit{k}}\mathrm{)}$ has been replaced with $\mathit{f}\mathrm{(}\mathit{g}\mathrm{(}{\mathbf{\text{e}}}_{\mathit{i}}\mathrm{,}{\mathbf{\text{l}}}_{\mathit{i}}\mathrm{)}\mathrm{,}\mathit{g}\mathrm{(}{\mathbf{\text{e}}}_{\mathit{j}}\mathrm{,}{\mathbf{\text{l}}}_{\mathit{j}}\mathrm{)}\mathrm{,}{\mathbf{\text{r}}}_{\mathit{k}}\mathrm{)}$ and trained following the same procedure as in the base model.

TransEA has two component models; a directly adopted translation-based structure embedding model (i.e., TransE) and a newly proposed attribute embedding model. In the former, the scoring function of a given triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ , is defined as follows: $$\begin{array}{}\text{(32)}& {\mathit{f}}_{\mathit{r}}\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}=-\Vert \mathit{h}+\mathit{r}-\mathit{t}{\Vert }_{1\mathrm{/}2}\mathrm{,}\end{array}$$ where $\Vert \mathit{x}{\Vert }_{1\mathrm{/}2}$ denotes either the L1 or L2 norm. The loss function of the structure embedding, for all the relational triples in the KG, is defined as: $$\begin{array}{}\text{(33)}& \begin{array}{cc}\hfill {\mathit{L}}_{\mathit{R}}& ={\displaystyle \sum }_{⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩\in \mathit{T}}{\displaystyle \sum }_{⟨{\mathit{h}}^{\prime }\mathrm{,}\mathit{r}\mathrm{,}{\mathit{t}}^{\prime }⟩\in {\mathit{T}}^{\prime }}max\mathrm{(}\mathit{\gamma }+{\mathit{f}}_{\mathit{r}}\mathrm{(}\mathit{h}\mathrm{,}\mathit{t}\mathrm{)}\hfill \\ \hfill & -{\mathit{f}}_{\mathit{r}}\mathrm{(}{\mathit{h}}^{\prime }\mathrm{,}{\mathit{t}}^{\prime }\mathrm{)}\mathrm{,}0\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ where ${\mathit{T}}^{\prime }$ denotes the set of negative triples constructed by corrupting either the head or the tail entity and $\mathit{\gamma }\mathrm{>}0$ is a margin hyperparameter.

For the attribute embedding, it uses all attributive triples containing numeric values as input and applies a linear regression model to learn embeddings of entities and attributes. Given an attributive triple $⟨\mathit{e}\mathrm{,}\mathit{a}\mathrm{,}\mathit{v}⟩$ , the scoring function is defined as: $$\begin{array}{}\text{(34)}& {\mathit{f}}_{\mathit{a}}\mathrm{(}\mathit{e}\mathrm{,}\mathit{v}\mathrm{)}=-{\Vert {\mathbf{\text{a}}}^{\mathit{T}}·\mathbf{\text{e}}+{\mathit{b}}_{\mathit{a}}-\mathit{v}\Vert }_{1\mathrm{/}2}\mathrm{,}\end{array}$$ where $\mathbf{\text{a}}$ and $\mathbf{\text{e}}$ are vectors of attribute a and entity e, ${\mathit{b}}_{\mathit{a}}$ is a bias for attribute a. On the other hand, given all the attributive triples with numeric values S, the loss function for the attributive embedding is computed as: $$\begin{array}{}\text{(35)}& {\mathit{L}}_{\mathit{A}}={\displaystyle \sum }_{⟨\mathit{e}\mathrm{,}\mathit{a}\mathrm{,}\mathit{v}⟩\in \mathit{S}}{\mathit{f}}_{\mathit{a}}\mathrm{(}\mathit{e}\mathrm{,}\mathit{v}\mathrm{)}\mathrm{,}\end{array}$$

The main loss function for TransEA (i.e., $\mathit{L}=\mathrm{(}1-\mathit{\alpha }\mathrm{)}·{\mathit{L}}_{\mathit{R}}+\mathit{\alpha }·{\mathit{L}}_{\mathit{A}}$ ) is defined by taking the sum of the respective loss functions of the component models with a hyperparameter to assign a weight for each of the models. Finally, the two models are jointly optimized in the training process by sharing the embeddings of entities.

Summary Despite their support for numerical literals, all the embedding methods discussed fail to interpret the semantics behind units/data typed literals. For instance, given the following two triples taken from DBpedia,

the literal value “110.0” from the first triple and the literal value “110.0” from the second triple could be considered exactly the same if the semantics of the types kilogram and pound are ignored. Moreover, most of the models do not have a proper mechanism to handle multi-valued literals.

Regarding model complexity, the number of parameters used in each model is presented in Table 4 to show the complexity in terms of the parameters. It is noted that the complexity of the models depend on the size of the dataset and TransEA has lower complexity as compared to the other models.

4.3. Models with image literals

In this section, KG embedding models utilizing images of entities, namely, IKRL [77] and MTKGRL [58] are discussed. First, a detailed analysis of the models is presented followed by a summary. Moreover, in order to show the differences between the models based on complexity, the number of parameters of each model is presented in Table 5.

Table 5

Complexity of the models with text literals in terms of the number of parameters. Θ is the number of parameters in the base model, H is the entity embedding size, ${\mathit{H}}_{\mathit{i}}$ represents the dimension of image features, ${\mathit{\theta }}_{\mathrm{AlexNet}}$ is the number of parameters in AlexNet [38], ${\mathit{N}}_{\mathit{e}}$ represents the number of entities, and ${\mathit{N}}_{\mathit{i}}$ is the number of images

Model	#Parameter
IKRL	$\mathrm{\Theta }+{\mathit{H}}_{\mathit{i}}\mathit{H}+{\mathrm{\Theta }}_{\mathrm{AlexNet}}$
MTKGRL	$\mathrm{\Theta }+{\mathit{N}}_{\mathit{e}}\mathit{H}+{\mathit{N}}_{\mathit{i}}{\mathit{H}}_{\mathit{i}}$

IKRL [77] learns embeddings for KGs by jointly training a structure-based representation with an image-based representation. The structure-based representation of an entity is learned by adapting a conventional embedding model like TransE. For the image-based representation, given the fact that an entity may have multiple image instances, an image encoder is applied to generate an embedding for each instance of a multi-valued image relation. The image encoder consists of a neural representation module and a projection module to extract discriminative features from images and to project these representations from image space to entity space respectively.

For the i-th image, its image-based representation ${\mathit{p}}_{\mathit{i}}$ in entity space is computed as: $$\begin{array}{}\text{(36)}& {\mathbf{\text{p}}}_{\mathit{i}}=\mathbf{\text{M}}·\mathit{f}\mathrm{(}{\mathit{img}}_{\mathit{i}}\mathrm{)}\mathrm{,}\end{array}$$ where $\mathit{M}\in {\mathbb{R}}^{{\mathit{d}}_{\mathit{i}}\times {\mathit{d}}_{\mathit{s}}}$ is the projection matrix with ${\mathit{d}}_{\mathit{i}}$ and ${\mathit{d}}_{\mathit{s}}$ representing the dimension of image features and the dimension of entities respectively. $\mathit{f}\mathrm{(}{\mathit{img}}_{\mathit{i}}\mathrm{)}$ is the i-th image feature representation in image space.

Attention-based multi-instance learning is used to integrate the representations learned for each image instance by automatically calculating the attention that should be given to each instance. The attention for the i-th image representation ${\mathit{p}}_{\mathit{i}}^{\mathrm{(}\mathit{k}\mathrm{)}}$ of the k-th entity is given as: $$\begin{array}{}\text{(37)}& att\mathrm{(}{\mathbf{\text{p}}}_{\mathit{i}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{,}{\mathbf{\text{e}}}_{\mathit{S}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{)}={\displaystyle \frac{exp\mathrm{(}{\mathbf{\text{p}}}_{\mathit{i}}^{\mathrm{(}\mathit{k}\mathrm{)}}·{\mathbf{\text{e}}}_{\mathit{S}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{)}}{{\sum }_{\mathit{j}=1}^{\mathit{n}}exp\mathrm{(}{\mathbf{\text{p}}}_{\mathit{j}}^{\mathrm{(}\mathit{k}\mathrm{)}}·{\mathbf{\text{e}}}_{\mathit{S}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{)}}}\mathrm{,}\end{array}$$ where ${\mathbf{\text{e}}}_{\mathit{S}}^{\mathrm{(}\mathit{k}\mathrm{)}}$ denotes the structure-based representation of the k-th entity. The higher the attention the more similar the image-based representation is to its corresponding structure-based representation which indicates that it should be given more importance when aggregating the image-based representations. The aggregated image-based representation for the k-th entity is defined as follows: $$\begin{array}{}\text{(38)}& {\mathbf{\text{e}}}_{\mathit{I}}^{\mathrm{(}\mathit{k}\mathrm{)}}={\displaystyle \sum }_{\mathit{i}=1}^{\mathit{n}}{\displaystyle \frac{att\mathrm{(}{\mathbf{\text{p}}}_{\mathit{i}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{,}{\mathbf{\text{e}}}_{\mathit{S}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{)}·{\mathbf{\text{p}}}_{\mathit{i}}^{\mathrm{(}\mathit{k}\mathrm{)}}}{{\sum }_{\mathit{j}=1}^{\mathit{n}}att\mathrm{(}{\mathbf{\text{p}}}_{\mathit{j}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{,}{\mathbf{\text{e}}}_{\mathit{S}}^{\mathrm{(}\mathit{k}\mathrm{)}}\mathrm{)}}}.\end{array}$$

Given a triple, the overall energy function is defined by combining four energy functions (i.e., $\mathit{E}\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}={\mathit{E}}_{\mathit{S}\mathit{S}}+{\mathit{E}}_{\mathit{I}\mathit{I}}+{\mathit{E}}_{\mathit{S}\mathit{I}}+{\mathit{E}}_{\mathit{I}\mathit{S}}$ . These energy functions are based on two kinds of entity representations (i.e, structure-based and image-based representations). The first energy function (i.e., ${\mathit{E}}_{\mathit{S}\mathit{S}}=\Vert {\mathit{h}}_{\mathit{S}}+\mathit{r}-{\mathit{t}}_{\mathit{S}}\Vert$ ) is same as TransE and the second function (i.e., ${\mathit{E}}_{\mathit{I}\mathit{I}}=\Vert {\mathit{h}}_{\mathit{I}}+\mathit{r}-{\mathit{t}}_{\mathit{I}}\Vert$ ) uses their corresponding image-based representations for both head and tail entities. The third function (i.e., ${\mathit{E}}_{\mathit{S}\mathit{I}}=\Vert {\mathit{h}}_{\mathit{S}}+\mathit{r}-{\mathit{t}}_{\mathit{I}}\Vert$ ) is based on the structure-based representation of the head entity and the image-based representation of the tail entity whereas the fourth function (i.e., ${\mathit{E}}_{\mathit{I}\mathit{S}}=\Vert {\mathit{h}}_{\mathit{I}}+\mathit{r}-{\mathit{t}}_{\mathit{S}}\Vert$ ) is the exact opposite. These third and forth functions ensure that both structure-based representation and image-based representations are learned into the same vector space.

Given the energy function $\mathit{E}\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}$ , a margin-based scoring function is defined as follows: $$\begin{array}{}\text{(39)}& \begin{array}{cc}\hfill \mathit{L}& ={\displaystyle \sum }_{\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}\in \mathit{T}}{\displaystyle \sum }_{\mathrm{(}{\mathit{h}}^{\prime }\mathrm{,}{\mathit{r}}^{\prime }\mathrm{,}{\mathit{t}}^{\prime }\mathrm{)}\in {\mathit{T}}^{\prime }}max\mathrm{(}\mathit{\gamma }+\mathit{E}\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}\hfill \\ \hfill & -\mathit{E}\mathrm{(}{\mathit{h}}^{\prime }\mathrm{,}{\mathit{r}}^{\prime }\mathrm{,}{\mathit{t}}^{\prime }\mathrm{)}\mathrm{,}0\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ where γ is a margin hyperparameter and ${\mathit{T}}^{\prime }$ is the negative sample set of T generated by replacing the head entity, tail entity or the relation for each triple in T. Note that triples which are already in T are removed from ${\mathit{T}}^{\prime }$ .

MTKGRL [58] is a KG embedding approach which combines structural (symbolic), visual, and linguistic KG representations. The structural representations are created by adopting TransE embedding technique whereas visual embeddings are obtained from the feature layers of deep networks for image classification on the images that are associated with entities. For linguistic representations, pre-trained word embedding technique, specifically the skipgram model, is used. However, the information source for the linguistic representation are not literals from the KG but an external source, i.e., the word embedding model, trained on Google 100B token news dataset. Due to this fact, the model MTKGRL is not considered as multi-modal KG embedding model in the context of this survey and thus, it is not categorized under ‘Models with Multi-modal Literals’ (Section 4.4).

MTKGRL defines an energy function for each kind of representation and also their combinations, i.e., structural energy, multimodal energies, and structural-multimodal energies. Structural energy is adopted from TransE, which is defined as ${\mathit{E}}_{\mathit{S}}=\Vert {\mathit{\text{h}}}_{\mathit{\text{s}}}+{\mathit{\text{r}}}_{\mathit{\text{s}}}-{\mathit{\text{t}}}_{\mathit{\text{s}}}\Vert$ . The multimodal representations for the head and tail entities are computed as ${\mathit{\text{h}}}_{\mathit{\text{m}}}={\mathit{\text{h}}}_{\mathit{\text{w}}}\oplus {\mathit{\text{h}}}_{\mathit{\text{i}}}$ and ${\mathit{\text{t}}}_{\mathit{\text{m}}}={\mathit{\text{t}}}_{\mathit{\text{w}}}\oplus {\mathit{\text{t}}}_{\mathit{\text{i}}}$ respectively, where the operator ⊕ can be a concatenation operator or a mapping function.

The multimodal energy function under the translational assumption is given as: $$\begin{array}{}\text{(40)}& {\mathit{E}}_{\mathit{M}1}=\Vert {\mathbf{\text{h}}}_{\mathbf{\text{m}}}+{\mathbf{\text{r}}}_{\mathbf{\text{s}}}-{\mathbf{\text{t}}}_{\mathbf{\text{m}}}\Vert .\end{array}$$ ${\mathit{E}}_{\mathit{M}1}$ can be extended by considering the structural embeddings in addition to the multimodal embeddings as follows: $$\begin{array}{}\text{(41)}& {\mathit{E}}_{\mathit{M}2}=\Vert \mathrm{(}{\mathbf{\text{h}}}_{\mathbf{\text{m}}}+{\mathbf{\text{h}}}_{\mathbf{\text{s}}}\mathrm{)}{\mathbf{\text{r}}}_{\mathbf{\text{s}}}-\mathrm{(}{\mathbf{\text{t}}}_{\mathbf{\text{m}}}+{\mathbf{\text{t}}}_{\mathbf{\text{s}}}\mathrm{)}\Vert .\end{array}$$ On the other hand, in order to allow the structural and multimodal embeddings to be learned in the same vector space, the following structural-multimodal energies are defined as shown below: $$\begin{array}{}\text{(42a)}& {\mathit{E}}_{\mathit{S}\mathit{M}}=\Vert {\mathbf{\text{h}}}_{\mathbf{\text{s}}}+{\mathbf{\text{r}}}_{\mathbf{\text{s}}}-{\mathbf{\text{t}}}_{\mathbf{\text{m}}}\Vert \text{(42b)}& {\mathit{E}}_{\mathit{M}\mathit{S}}=\Vert {\mathbf{\text{h}}}_{\mathbf{\text{m}}}+{\mathbf{\text{r}}}_{\mathbf{\text{s}}}-{\mathbf{\text{t}}}_{\mathbf{\text{s}}}\Vert \end{array}$$

The overall energy function, shown in Equation (43), is defined by combining the aforementioned energy functions, i.e., ${\mathit{E}}_{\mathit{S}}$ , ${\mathit{E}}_{\mathit{M}1}$ , ${\mathit{E}}_{\mathit{M}2}$ , ${\mathit{E}}_{\mathit{S}\mathit{M}}$ , ${\mathit{E}}_{\mathit{M}\mathit{S}}$ . $$\begin{array}{}\text{(43)}& \begin{array}{cc}\hfill \mathit{E}\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}\mathrm{)}& ={\mathit{E}}_{\mathit{S}}+{\mathit{E}}_{\mathit{M}1}+{\mathit{E}}_{\mathit{M}2}+{\mathit{E}}_{\mathit{S}\mathit{M}}\hfill \\ \hfill & +{\mathit{E}}_{\mathit{M}\mathit{S}}\hfill \end{array}\end{array}$$ Finally, a margin-based ranking loss function is minimized in order to train the model.

Summary IKRL makes use of the images of entities for KG representation learning by combining structure-based representation with image-based representation. However, given a triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ , in order to achieve very good representations for the entities h and t, both entities are required to have images associated with them. The other issue with this model is that an image is considered as an attribute of only those entities it is associated with. For example, if there is an image of two entities ${\mathit{e}}_1$ and ${\mathit{e}}_2$ but the image is associated with only ${\mathit{e}}_1$ , then it will be taken as one image instance of ${\mathit{e}}_1$ but not of ${\mathit{e}}_2$ . However, it would be more beneficial to explicitly associate images with all the entities they represent before using them for learning KG embedding. Some of the main points which make MTKGRL differ from IKRL are: i) in addition to images, MTKGRL uses linguistic embeddings for entities, ii) MTKGRL introduces an additional energy function that considers both linguistic and visual representations of entities as discussed above. These differences allow MTKGRL to learn better representation for KGs as compared to IKRL.

4.4. Models with multi-modal literals

This section presents an analysis of the embedding models making use of at least two types of literals providing complementary information. First, the category with numeric and text literals is discussed followed by the category with numeric, text, and image. Moreover, in order to show the differences between the models based on complexity, the number of parameters of each model is presented in Table 6.

Table 6

Complexity of the models with multimodal literals in terms of the number of parameters. Θ is the number of parameters in the base model, H is the entity embedding size, ${\mathit{N}}_{\mathit{d}}$ is the number of data relations, ${\mathit{N}}_{\mathrm{char}}$ is the number of characters, and ${\mathit{N}}_{\mathit{i}}$ is the number of images, ${\mathrm{\Theta }}_{\mathrm{CNN}}$ is the number of parameters in the CNN model used in [19], ${\mathrm{\Theta }}_{\mathrm{ARAE}}$ is the number of parameters in ARAE [83] where instead of using the random noise vector z, the generator is conditioned on the entity embeddings, ${\mathrm{\Theta }}_{\mathrm{GAN}}$ denotes the sum of the number of parameters in BE-GAN [1] and in pix2pix-GAN [30]

Model	#Parameter
LiteralE with blocking	$\mathrm{\Theta }+\mathrm{(}{\mathit{N}}_{\mathit{d}}\mathit{H}+\mathit{H}\mathrm{)}\mathit{H}$
EAKGAE	$\mathrm{\Theta }+\mathrm{(}{\mathit{N}}_{\mathit{d}}+{\mathit{N}}_{\mathrm{char}}\mathrm{)}\mathit{H}$
MKBE	$\mathrm{\Theta }+\mathrm{(}2\mathrm{(}{\mathit{N}}_{\mathit{d}}+3\mathrm{(}{\mathit{N}}_{\mathrm{char}}+\mathit{H}\mathrm{)}\mathrm{)}+{\mathit{N}}_{\mathit{i}}\mathrm{)}\mathit{H}+{\mathrm{\Theta }}_{\mathrm{CNN}}+{\mathrm{\Theta }}_{\mathrm{ARAE}}+{\mathrm{\Theta }}_{\mathrm{GAN}}$

4.4.1. Models with numeric and text literals

LiteralE with blocking [14] proposes to improve the effectiveness of the data linking task by combining LiteralE with a CER blocking [13] strategy. Unlike LiteralE, given an attributive triple $⟨\mathit{h}\mathrm{,}\mathit{d}\mathrm{,}\mathit{v}⟩$ , in addition to the object literal value v it also takes literals from URI infixes of the head entity h and the data relation d. The CER blocking is based on a two-pass indexing scheme. In the first pass, Levenshtein distance metric is used to process literal objects and URI infixes whereas in the second pass semantic similarity computation with WordNet [48] is applied to process object/data relations. All the extracted literals are tokenized into word lists so as to create inverted indices. The same training procedure as in LiteralE is used to train this model. For every given triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ , the scoring function f from LiteralE is adopted to compute scores for all the triples $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}{\mathit{t}}^{\prime }⟩$ in the knowledge graph. A sigmoid function, $\mathit{p}=\mathit{\sigma }\mathrm{(}\mathit{f}\mathrm{(}·\mathrm{)}\mathrm{)}$ , is used to produce probabilities. Then, the model is trained by minimizing the binary cross-entropy loss of the produced probability function vector with respect to the vector of truth values for the triples.

EAKGAE [67] is an approach designed for entity alignment between KGs by learning a unified embedding space for the KGs. The entity alignment task has three main modules: Predicate alignment, Embedding learning, and Entity alignment. The predicate alignment module merges two KGs together by renaming similar predicates so as to create unified vector space for the relationship embeddings. The embedding learning module jointly learns entity embeddings of two KGs using structure embedding (by adapting TransE) and attribute character embedding. The adapted TransE is customized in a way that more focus can be given to triples with aligned predicates. This is obtained by adding a weight α to control the embedding learning over the triples. Thus, the following objective function ${\mathit{J}}_{\mathrm{SE}}$ is defined for the structure-based embedding: $$\begin{array}{}\text{(44)}& \begin{array}{cc}\hfill {\mathit{J}}_{\mathrm{SE}}& ={\displaystyle \sum }_{{\mathit{t}}_{\mathit{r}}\in {\mathit{T}}_{\mathit{r}}}{\displaystyle \sum }_{{\mathit{t}}_{\mathit{r}}^{\prime }\in {\mathit{T}}_{\mathit{r}}^{\prime }}max\mathrm{(}0\mathrm{,}\hfill \\ \hfill & \mathit{\gamma }+\mathit{\alpha }\mathrm{(}\mathit{f}\mathrm{(}{\mathit{t}}_{\mathit{r}}\mathrm{)}-\mathit{f}\mathrm{(}{\mathit{t}}_{\mathit{r}}^{\prime }\mathrm{\left)}\mathrm{\right)}\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ and $$\begin{array}{}\text{(45)}& \mathit{\alpha }={\displaystyle \frac{count\mathrm{(}\mathit{r}\mathrm{)}}{|\mathit{T}|}}\mathrm{,}\end{array}$$ where ${\mathit{T}}_{\mathit{r}}$ and ${\mathit{T}}_{\mathit{r}}^{\prime }$ are the sets of valid triples and corrupted triples respectively, $count\mathrm{(}\mathit{r}\mathrm{)}$ is the number of occurrences of the relation r, and $|\mathit{T}|$ is the total number of triples in the merged KG.

On the other hand, the attributing character embedding is designed to learn embeddings for entities from the strings occurring in the attributes associated with the entities. The purpose is to enable the entity embeddings from two KGs to fall into the same vector space despite the fact that the attributes come from different KGs. The attribute character embedding is inspired by the concept of translation in TransE. Given a triple $\mathrm{(}\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{a}\mathrm{)}$ , the data property r is interpreted as a translation from the head entity h to the literal value a i.e. $\mathit{h}+\mathit{r}={\mathit{f}}_{\mathit{a}}\mathrm{(}\mathit{a}\mathrm{)}$ where ${\mathit{f}}_{\mathit{a}}\mathrm{(}\mathit{a}\mathrm{)}$ is a compositional function. This function encodes the attribute values into a single vector mapping similar attribute values into similar representation. Three different compositional functions SUM, LSTM, and N-gram-based functions have been proposed. SUM is defined as a summation of all character embeddings of the attribute value. In LSTM, the final hidden state is taken as a vector representation of the attribute value. The N-gram-based function, which shows better performance than the others according to their experiments, uses the summation of n-gram combination of the attribute value.

The following objective function is defined for the attribute character embedding: $$\begin{array}{l}\text{(46)}& \begin{array}{cc}\hfill {\mathit{J}}_{\mathrm{CE}}& ={\displaystyle \sum }_{{\mathit{t}}_{\mathit{a}}\in {\mathit{T}}_{\mathit{a}}}{\displaystyle \sum }_{{\mathit{t}}_{\mathit{a}}^{\prime }\in {\mathit{T}}_{\mathit{a}}^{\prime }}max\mathrm{(}0\mathrm{,}\hfill \\ \hfill & [\mathit{\gamma }+\mathit{\alpha }\mathrm{(}\mathit{f}\mathrm{(}{\mathit{t}}_{\mathit{a}}\mathrm{)}-\mathit{f}\mathrm{\left(}{\mathit{t}}_{\mathit{a}}^{\prime }\mathrm{\right)}\mathrm{\left)}\right]\mathrm{)}\mathrm{,}\hfill \end{array}{\mathit{T}}_{\mathit{a}}=⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{a}⟩\mathit{h}\in \mathit{G};\\ \mathit{f}\mathrm{(}{\mathit{t}}_{\mathit{a}}\mathrm{)}=\Vert \mathit{h}+\mathit{r}-{\mathit{f}}_{\mathit{a}}\mathrm{(}\mathit{a}\mathrm{)}\Vert \mathrm{,}\end{array}$$ and $$\begin{array}{c}{\mathit{T}}_{\mathit{a}}^{\prime }=\{⟨{\mathit{h}}^{\prime }\mathrm{,}\mathit{r}\mathrm{,}\mathit{a}⟩{\mathit{h}}^{\prime }\in \mathit{G}\}\cup \{⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}{\mathit{a}}^{\prime }⟩{\mathit{a}}^{\prime }\in \mathit{A}\}\mathrm{,}\end{array}$$ where, ${\mathit{T}}_{\mathit{a}}$ and ${\mathit{T}}_{\mathit{a}}^{\prime }$ are the sets of valid attribute triples and corrupted attribute triples with A being the set of attributes in a given KG G. The corrupted triples are created by replacing the head entity with a random entity or the attribute with a random attribute value. Here, $\mathit{f}\mathrm{(}{\mathit{t}}_{\mathit{a}}\mathrm{)}$ is the plausibility score computed based on the embedding of the head entity h, the embedding of the relation r, and the vector representation of the attribute value obtained using one of the compositional functions ${\mathit{f}}_{\mathit{a}}\mathrm{(}\mathit{a}\mathrm{)}$ .

The attribute character embedding ${\mathit{h}}_{\mathrm{ce}}$ is used to shift the structure embedding ${\mathit{h}}_{\mathrm{se}}$ into the same vector space by minimizing the following objective function: $$\begin{array}{}\text{(47)}& {\mathit{J}}_{\mathrm{SIM}}={\displaystyle \sum }_{\mathit{h}\in {\mathit{G}}_1\cup {\mathit{G}}_2}[1-\Vert {\mathit{h}}_{\mathrm{se}}{\Vert }_2·\Vert {\mathit{h}}_{\mathrm{ce}}{\Vert }_2]\mathrm{,}\end{array}$$ where, $\Vert \mathit{x}{\Vert }_2$ is the ${\mathit{L}}_2$ -Norm of vector x. This way the similarity of entities from two KGs is captured by the structure embedding based on the entity relationships and by the attribute embedding based on the attribute values.

All the three functions are summed up to an overall objective function J (i.e., $\mathit{J}={\mathit{J}}_{\mathrm{SE}}+{\mathit{J}}_{\mathrm{CE}}+{\mathit{J}}_{\mathrm{SIM}}$ ) for jointly learning both structure and attribute embeddings. Finally, the alignment is done by defining a similarity equation with a specified threshold. Moreover, a transitivity rule has been applied to enrich triples in the KGs to get a better attribute embedding result.

Summary The common drawback with both methods (LiteralE with blocking and EAKGE) is that text and numeric literals are treated in the same way. They also do not consider literal data type semantics or multi-valued literals in their approach. Furthermore, since EAKGAE is using character-based attribute embedding, it fails to capture the semantics behind the co-occurrence of syllables.

4.4.2. Models with numeric, text, and image literals

MKBE [55] is a multi-modal KG embedding, in which the text, numeric and image literals are modelled together. The main objective of this approach is to utilize all the observed subjects, objects, and relations (object properties and data properties) in order to predict whether any fact holds. It extends DistMult, which creates embedding for entities and relations, by adding neural encoders for different data types. Given a triple $⟨\mathit{s}\mathrm{,}\mathit{r}\mathrm{,}\mathit{o}⟩$ , the head entity s and the relation r are encoded as independent embedding vectors using one-hot encoding through a dense layer. Similarly, if the object o is a categorical value, then it will be represented through a dense layer with a relu activation which has the same number of nodes as the embedding space dimension. On the other hand, if the object o is rather a numerical value, then a feed forward layer, after standardizing the input, is used in order to learn embeddings for o by projecting it to a higher-dimensional space. If o is a short text (such as names and titles), it is encoded using character-based stacked, bidirectional GRUs and the final output of the top layer will be taken as the representation of o. On the contrary, if o is a long text such as entity descriptions, CNN over word embeddings will be used to get the embeddings for o. The object o can also be an image, and in such a case, the last hidden layer of VGG pretrained network on ImageNet [59], followed by compact bilinear pooling, is used to obtain the embedding of o. Given the vector representations of the entities, relations and attributes, the same scoring function from DistMult is used to determine the correctness probability of triples.

The binary cross-entropy loss, as defined below, is used to train the model: $$\begin{array}{}\text{(48)}& \begin{array}{cc}\hfill & {\displaystyle \sum }_{\mathrm{(}\mathit{s}\mathrm{,}\mathit{r}\mathrm{)}}{\displaystyle \sum }_{\mathit{o}}{\mathit{t}}_{\mathit{o}}^{\mathit{s}\mathrm{,}\mathit{r}}log\mathrm{\left(}{\mathit{p}}_{\mathit{o}}^{\mathit{s}\mathrm{,}\mathit{r}}\mathrm{\right)}\hfill \\ \hfill & +\mathrm{(}1-{\mathit{t}}_{\mathit{o}}^{\mathit{s}\mathrm{,}\mathit{r}}\mathrm{)}log\mathrm{(}1-{\mathit{p}}_{\mathit{o}}^{\mathit{s}\mathrm{,}\mathit{r}}\mathrm{)}\mathrm{,}\hfill \end{array}\end{array}$$ where for a given subject relation pair $\mathrm{(}\mathit{s}\mathrm{,}\mathit{r}\mathrm{)}$ , binary label vector ${\mathit{t}}^{\mathit{s}\mathrm{,}\mathit{r}}$ over all entities is used to indicate whether $⟨\mathit{s}\mathrm{,}\mathit{r}\mathrm{,}\mathit{o}⟩$ is observed during training. ${\mathit{p}}_{\mathit{o}}^{\mathit{s}\mathrm{,}\mathit{r}}$ denotes the model’s probability of truth for any triple $⟨\mathit{s}\mathrm{,}\mathit{r}\mathrm{,}\mathit{o}⟩$ computed using a sigmoid function.

Moreover, using these learned embeddings and different neural decoders, a novel multimodal imputation model is introduced to generate missing multimodal values, such as numerical data, categorical data, text, and images, from information in the knowledge base. In order to predict the missing numerical and categorical data such as dates, gender, and occupation, a simple feed-forward network on the entity embedding is used. For text, the adversarially regularized autoencoder (ARAE) has been used to train generators that decodes text from continuous codes, having the generator conditioned on the entity embeddings instead of random noise vector. Similarly, the combination of BE-GAN structure with pix2pix-GAN model is used to generate images, conditioning the generator on the entity embeddings.

Summary Despite the attempt made in incorporating text literals, numeric literals, and images into the KG embedding, the model (MKBE) fails to capture the semantics of the data types/units of (numeric) literal values. Besides, similar to IKRL, it takes an image I as an instance of a certain entity e only if, I is initially associated with e in the dataset considered (refer to Section 4.3 for more details).

5. Applications

This section discusses different applications of KG embeddings on which the previously described methods have been trained and/or evaluated.

Link prediction In general terms, link prediction can be defined as a task of identifying missing information in complex networks [41,45]. Specifically in the case of KGs, link prediction models aim at predicting new relations between entities leveraging the existing links for training. Along with predicting relations between the entities link prediction also focuses on the task of predicting either the head or the tail entity with respect to a relation. Then it decides if a new triple, which is not observed in the KG, is valid or not. Formally, let G be a KG with a set of entities $\mathit{E}=\{{\mathit{e}}_1\mathrm{,}\dots \mathrm{,}{\mathit{e}}_{\mathit{n}}\}$ and a set of object relations $\mathit{R}=\{{\mathit{r}}_1\mathrm{,}\dots \mathrm{,}{\mathit{r}}_{\mathit{m}}\}$ , then link prediction can be defined by a mapping function $\mathit{\psi }:\mathit{E}\times \mathit{E}\times \mathit{R}\to \mathit{R}$ which assigns a score to every possible triple $\mathrm{(}{\mathit{e}}_{\mathit{i}}\mathrm{,}{\mathit{e}}_{\mathit{j}}\mathrm{,}{\mathit{r}}_{\mathit{k}}\mathrm{)}\in \mathit{E}\times \mathit{E}\times \mathit{R}$ . A high score indicates that the triple is most likely to be true.

Link prediction is one of the most common tasks used for evaluating the performance of KG embeddings. Head prediction, tail prediction, and relation prediction are different kinds of sub-tasks related to link prediction. Head prediction aims at identifying a missing head entity where the relation and tail entity are given, and analogously for tail prediction and relation prediction. Most of the models discussed in Section 4 have been evaluated on some or all of these prediction tasks. Head and tail prediction are used to evaluate the models LiteralE [37], TransEA [75], KBLRN [22], KDCoE [9], EAKGAE [67], IKRL [77], MKBE [55], MTKGRL [58], Jointly(Desp) [84], Jointly [80], and SSP [76]. On the other hand, DKRL [78] has been evaluated on all kinds of link prediction tasks: head, tail, and relation predictions. In Extended RESCAL [51], two kinds of link prediction experiments have been conducted on the Yago 2 [61], i.e., i) tail prediction by fixing the relation type to rdf:type, and ii) general link prediction experiments for all relation types. Unfortunately, it is not possible to compare the obtained evaluation results of all these models because the experiments have been carried out on different datasets and also different link prediction procedures have been followed. Taking this into consideration, in this survey, experiments have been conducted on head and tail prediction tasks for these models (see Section 6).

Triple classification The goal of the triple classification task is the same as that of link prediction. A potential triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ is classified as 0 (false) or 1 (true), i.e., a binary classification task. The embedding models MTKGNN [78], IKRL [77], MTKGRL [58], Jointly(Desp) [84], and Jointly [80] have been evaluated on this task. However, since they do not use a common evaluation dataset, it is not possible to compare the reported results directly.

Entity classification Given a KG G, with a set of entities E and types T and with an entity $\mathit{e}\in \mathit{E}$ and type $\mathit{t}\in \mathit{T}$ , the task of entity classification is to determine if a potential entity type pair $\mathrm{(}\mathit{e}\mathrm{,}\mathit{t}\mathrm{)}$ which is not observed in G ( $\mathrm{(}\mathit{e}\mathrm{,}\mathit{t}\mathrm{)}\notin \mathit{G}$ ) is a missing fact or not. This task is an entity type prediction using a multi-label classification algorithm considering the entity types in G as given classes. In DKRL [78], Extended RESCAL [51], and SSP [76], Entity classification has been used for model evaluation.

Entity alignment Given two KGs ${\mathit{G}}_1$ and ${\mathit{G}}_2$ , the goal of the entity alignment task is to identify those entity pairs $\mathrm{(}{\mathit{e}}_1\mathrm{,}{\mathit{e}}_2\mathrm{)}$ where ${\mathit{e}}_1$ is an entity in ${\mathit{G}}_1$ and ${\mathit{e}}_2$ is an entity in ${\mathit{G}}_2$ which denote the same real world entities, and hence the integration of ${\mathit{G}}_1$ and ${\mathit{G}}_2$ can be possible through these unified entities, i.e., entity pairs. Different embedding-based models have been proposed recently for the entity alignment task. Among the models that are included in this survey, EAKGAE [67] and KDCoE [9] have been proposed for the entity alignment task. Specifically, KDCoE [9] uses a cross-lingual entity alignment task which determines similar entities in different languages. Despite the fact that both these models use the same task for evaluation, the entity alignment task, their experimental results cannot be compared since they are based on different datasets.

Other applications Attribute-value prediction, nearest-neighbor analysis, data linking, document classification, and relational fact extraction are other application scenarios used for the evaluation of the models under discussion. Attribute-value prediction is the process of predicting the values of (discrete or non-discrete) attributes in a KG. For example, a missing value of a person’s weight can be identified using the attribute value prediction task which is commonly seen as a KG completion task. In MTKGNN [63], attribute-value prediction is applied using an attribute-specific Linear Regression classifier for evaluation. The same task has been employed in MKBE [55] for model evaluation by imputing different multi-modal attribute values.

Nearest Neighbor Analysis is a task of detecting the nearest neighbors of some given entities in the latent space learned by an embedding model. This task has been performed in LiteralE [37] to compare $\mathrm{DistMult}+\text{LiteralE}$ with the base model DistMult. On the other hand, data linking and document classification tasks have been used in LiteralE with blocking [14] and KGlove with literals [10] respectively (refer to [14] and [10] for more details). Relational fact extraction is a task of extracting facts/triples from plain text and has been used as a model evaluation task in Jointly(Desp) [84]. Table 16 summarizes all the applications on which the KG embedding models with literals have been evaluated.

6. Experiments on link prediction

This section provides an empirical evaluation of the methods discussed in the previous section under a unified environmental settings and discusses the results based on the performance of the approaches applied to the task of link prediction. In this work, link prediction is chosen because most of the KG embedding models with literals are trained and evaluated on it. One of the major issues encountered while conducting these experiments is that the source code of some of these models is not openly available and is not easily reproducible. Such methods were excluded from the experimentation. In the subsequent sections, the datasets and the experiments with text, numeric, images and multi-modal literals are presented.

Based on the results of the experiments, a clear comparison is presented between the models with literals on link prediction. In addition, these models are also compared with the standard KG embedding approaches that they extend. Note that these models may inherit the problems that already exist in their corresponding base models – the standard KG embedding models that they extend). For instance, the models that extend DistMult such as $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ inherit the problem of DistMult, which is not being capable to properly capture anti-symmetric relations due to the way its scoring function is defined.

6.1. Datasets

The performance of the aforementioned models was measured using two of the most commonly used datasets for link prediction, i.e., FB15K [5] and FB15K-237 [64] are considered. FB15K is a subset of Freebase [2] which mostly contains triples describing the facts about movies, actors, awards, sports and sport teams. It contains a randomly split training, validation, and test sets. The issue with this dataset is that the test set contains a large number of triples which are obtained by simply inverting triples in the training set. This enables a simple embedding model which is symmetric with respect to the head and tail entity to obtain an excellent performance. In order to avoid this, the dataset FB15K-237 has been created by removing the inverse relations from FB15K. The statistics of these datasets is given in Table 7.

Table 7

The number of entities, object relations, data relations, relational triples, train sets, valid sets, and test sets of the FB15K and the FB15K-237 datasets

	Datasets
	FB15K	FB15K-237
Entities	14951	14541
Object relations	1345	237
Relational triples	592213	310116
Train sets	483142	272115
Valid sets	50000	17535
Test sets	59071	20466

6.2. Experiments with text literals

As discussed in Section 4.1, the embedding models Extended RESCAL, DKRL, KDCoE, and KGloVe with literals utilize text literals. However, all of these models except DKRL are not considered for experimentation due to the following issues:

• The implementation of the model KGloVe with literals is not publicly available and it is not easily reproducible.

• KDCoE is designed specifically for cross-lingual entity alignment task which makes it difficult to apply it for link prediction.

• In case of Extended RESCAL, practically this method is computationally expensive and thus not considered as a feasible embedding model to incorporate literals.

  Moreover, none of the models with literals which are discussed in this paper consider Extended RESCAL in their experiments.

In order to conduct experiments with text literals, 15239 English entity descriptions of the entities common in both datasets FB15K and FB15K-237 shown in Table 7 are taken from LiteralE [37]. The focus lies on the common entity descriptions, i.e., for those entities existing in FB15K but not in FB15K-237 no description is used, because there has already been experiments done using the whole entity descriptions for FB15K dataset in the original paper. This way it would be possible to analyse the effect of the size of the dataset (the entity descriptions) on the performance of the embedding models. The average number of words (tokens) in the descriptions is 143 whereas the maximum and minimum are 804 and 2.

Dataset pre-processing For pre-processing of the text (the entity descriptions), spacy.io1

¹

https://spacy.io/usage

has been used. This includes tokenization, named entity recognition and conversion of numbers to text, i.e., 16 has been converted to ‘sixteen’. After the pre-processing step, all the entities along with the corresponding triples having no or short description of less than 3 words are removed. Also, the triples containing these entities are removed as mentioned by the authors in the paper. Moreover, only one description is chosen randomly for the entities with multiple text descriptions.

Experimental setup The hyperparameters used for DKRL are as follows: learning rate 0.001, embedding size 100, loss margin 1, batch size 100 and epochs 1000. For TransE, learning rate 0.01, embedding size 50, margin 1, and epochs 1000 are used. The experiments with DKRL were performed on Ubuntu 16.04.5 LTS system with 503 GiB RAM and 2.60 GHz speed. On the other hand, the experiments with TransE are performed with TITAN X (Pascal) GPU.

Runtime Note that the codes used in the experiments for both models DKRL and TransE are not implemented in the same environment, i.e., for DKRL, the code that is released by the authors of the paper is used and for TransE the code provided by the authors of TransEA [75] is used. Therefore, it is not fair to compare the runtime results of these two models directly. However, in order to provide some insights into the computational complexity of the models, their runtime results on the FB15K dataset are given as follows. DKRL takes 142 seconds to train 1 epoch using 16 threads whereas the runtime of TransE for a single iteration with batch size 4831 is 3.271 millisecond. This is computed by taking the average of 1000 iterations.

Evaluation procedure and results The performance of the model is evaluated based on the link prediction task. For each triple in the test set, a set of corrupted triples is generated with respect to the head or the tail entity. A triple is said to be corrupted with respect to its head entity if that head entity is replaced with any other entity from the KG, and analogously for a triple corrupted with respect to its tail entity. The set of corrupted triples can also contain true triples that exist in the training, validation or test set. Since it is not a mistake to give these true triples better score than the actual test triple, they are removed from the set of corrupted triples and this is referred to as filtered setting [5]. In order to check if the model assigns a better score to the actual test triple than the corrupted triples which are obtained by corrupting the test triple, it is evaluated using the metrics MR (Mean Rank), MRR (Mean Reciprocal Rank), and Hits@N. First, for every test triple, all of its corrupted triples with respect to head are ranked based on their scores which are computed by the model. Then, the rank of the actual (true) test triples are taken in order to compute the metrics MR, MRR, and Hits@N. MR is the mean of the ranks of all test triples – the lower the better and MRR is their average inverse rank – the higher the better. Hits@N is the percentage of ranks lower than or equal to N – the higher the better. The same procedure is repeated to evaluate the model against the corrupted triples with respect to tail.

Table 8

Experiment results using DKRL model on FB15K and FB15K-237 datasets

		MR	MRR	Hits@1	Hits@3	Hits@10
FB15K
TransE	Head	142	0.219	0.241	0.447	0.622
	Tail	109	0.249	0.339	0.514	0.690
	All	125	0.234	0.290	0.480	0.656
${\mathrm{DKRL}}_{\mathrm{Bern}}$	Head	162	0.289	0.179	0.336	0.502
	Tail	122	0.356	0.24	0.408	0.577
	All	142	0.322	0.209	0.372	0.539
${\mathrm{DKRL}}_{\mathrm{Unif}}$	Head	96	0.289	0.172	0.335	0.52
	Tail	75	0.333	0.211	0.383	0.576
	All	85	0.311	0.191	0.359	0.548
FB15K-237
TransE	Head	468	0.094	0.081	0.163	0.287
	Tail	255	0.190	0.233	0.373	0.517
	All	361	0.142	0.157	0.268	0.402
${\mathrm{DKRL}}_{\mathrm{Bern}}$	Head	145	0.294	0.184	0.337	0.507
	Tail	98	0.359	0.244	0.410	0.585
	All	122	0.327	0.214	0.374	0.546
${\mathrm{DKRL}}_{\mathrm{Unif}}$	Head	104	0.275	0.166	0.312	0.494
	Tail	77	0.322	0.209	0.363	0.552
	All	91	0.298	0.187	0.337	0.523

The results of link prediction on FB15K and FB15K-237 datasets are shown in Table 8 for the models TransE, DKRL with Bernoulli distribution ( ${\mathrm{DKRL}}_{\mathrm{Bern}}$ ), and DKRL with Uniform distribution ( ${\mathrm{DKRL}}_{\mathrm{unif}}$ ). The Bernoulli distribution for sampling as defined in [71] is a probability distribution, $\frac{\mathit{t}\mathit{p}\mathit{h}}{\mathit{t}\mathit{p}\mathit{h}+\mathit{h}\mathit{p}\mathit{t}}$ , where $\mathit{t}\mathit{p}\mathit{h}$ is the average number of tail entities per head entity and $\mathit{h}\mathit{p}\mathit{t}$ is the average number of head entities per tail entity. Given a golden triple $⟨\mathit{h}\mathrm{,}\mathit{r}\mathrm{,}\mathit{t}⟩$ , with the aforementioned probability, the triple is corrupted by replacing the head, and with probability $\frac{\mathit{h}\mathit{p}\mathit{t}}{\mathit{t}\mathit{p}\mathit{h}+\mathit{h}\mathit{p}\mathit{t}}$ , the triple is corrupted by replacing the tail. The results are reported separately for the head entity and tail entity along with the overall results obtained by taking the mean of the head and tail predictions. The best scores are the ones which are highlighted in bold text. The result of the TransE model is presented in order to allow a clear comparison with DKRL because, as shown in Table 2, DKRL extends TransE. This comparison would help to further analyse the advantages of using text literals for KG embedding.

Note that in the original paper, the result of DKRL on FB15K is slightly better than TransE. However, in our experiments, as the results in Table 8 indicate, on the FB15K dataset TransE achieves better result than both versions of DKRL on all metrics except MRR and MR. The reason for this is that, as mentioned above, the set of entity descriptions used in our experiments are common for both datasets FB15K and FB15K-237, i.e., there is less entity descriptions in our experiment than there is in the original paper for FB15K. This indicates that the size of the dataset (the entity description has impact on the performance of the model). On the other hand, on the dataset FB15K-237 TransE is outperformed by ${\mathrm{DKRL}}_{\mathrm{Unif}}$ with respect to MR and by ${\mathrm{DKRL}}_{\mathrm{Bern}}$ with respect to the rest of the metrics.

Furthermore, the result shows that DKRL model with Bernoulli distribution ( ${\mathrm{DKRL}}_{\mathrm{Bern}}$ ) has better performance than the model with Uniform distribution ( ${\mathrm{DKRL}}_{\mathrm{unif}}$ ) for both the datasets. ${\mathrm{DKRL}}_{\mathrm{Bern}}$ works best for the prediction of head, relation, and tail with respect to MRR, Hits@1, and Hits@3 whereas the ${\mathrm{DKRL}}_{\mathrm{Unif}}$ method works better according to MR for both the datasets. ${\mathrm{DKRL}}_{\mathrm{Bern}}$ works slightly better than ${\mathrm{DKRL}}_{\mathrm{Unif}}$ for FB15K-237 dataset. It is to be noted that DKRL has better improvement over TransE on FB15K-237 as compared to FB15K dataset because the former one does not contain symmetric relations, i.e., incorporating textual data to a clean dataset, such as FB15K-237, allows capturing more semantics.

6.3. Experiment with numeric literals

MT-KGNN, KBLRN, LiteralE, and TransEA are the KG embedding models which make use of numeric literals (see Section 4.2). KBLN, the submodel of KBLRN, which excludes the relational information provided by graph feature methods is used in the experiment instead of the main model KBLRN. This is the case because KBLN is directly comparable with the other three models (i.e., MT-KGNN, LiteralE, and TransEA) whereas KBLRN is not. The code2

²

https://github.com/kk0spence/TransEA

for the TransEA model is the original implementation from TransEA [75] where as the source codes3 ³

https://github.com/SmartDataAnalytics/LiteralE

for the models MT-KGNN, KBLN, and LiteralE are taken from the implementation in LiteralE [37]. As described in Section 4.2, the structure-based embedding component of MT-KGNN is based on a neural network and it is referred to as RelNet. However, in the version implemented in LiteralE [37], they have replaced RelNet with DistMult as a baseline in order to have a directly comparable MTKGNN-like method to their proposed approach. Thus, in this survey, the MT-KGNN-like model has been used instead of the original MT-KGNN model.

Moreover, the model LiteralE has different varieties depending on the baseline model and the transformation function used. As discussed in Section 4, in LiteralE there are two transformation functions: g (GRU based function) and $\mathrm{lin}$ (a simple linear function), and there are three baseline models – DistMult, ConvE and ComplEx. Thus, in this experiment, six varieties of the LiteralE model are considered: $\mathrm{DistMult}\text{-}{\mathrm{Literale}}_{\mathit{g}}$ , $\mathrm{ComplEx}\text{-}{\mathrm{Literale}}_{\mathit{g}}$ , $\mathrm{ConvE}\text{-}{\mathrm{Literale}}_{\mathrm{lin}}$ , $\mathrm{DistMult}\text{-}{\mathrm{Literale}}_{\mathrm{lin}}$ , $\mathrm{ComplEx}\text{-}{\mathrm{Literale}}_{\mathrm{lin}}$ , and $\mathrm{ConvE}\text{-}{\mathrm{Literale}}_{\mathrm{lin}}$ . The datasets, the experimental setup, and the evaluation results are discussed in the subsequent sections.

Attributive triples In order to conduct the experiments with numeric literals, both the datasets FB15K and FB15K-237 given in Table 7 are extended with a set of 23521 attributive triples, containing only numeric literals, with 118 data relations. These triples are created based on the attributive triples from TransEA [75]. In TransEA, the authors have provided a set of attributive triples where the object values are numeric. However, it is not possible to directly use this data as the literal values are normalized in the interval [0–1] as required by the model but the other models in this experiment, like LiteralE, use the original unnormalized literal values instead. Therefore, it was necessary to query Freebase to replace the normalized object literal value for each (subject, data relation) pair from the TransEA attributive triples data. Moreover, only those data relations which occur in at least 5 triples are taken into consideration.

Experimental setup For both datasets, the hyperparameters for TransEA are: epoch 3000, dimension 100, batches 100, margin 2, and learning rate 0.3 and for TransE they are described in Section 6.2. For the other models, same as in LiteralE, the hyperparameters used for both datasets are: learning rate 0.001, batch size 128, embedding size 100 (for DistMult, ComplEx and their extensions with literals) and 200 (for KBLN, and MTKGNN, ConvE, and ConvE’s extensions), embedding dropout probability 0.2, label smoothing 0.1, and epochs 1000 for ConvE and 100 for the rest. TITAN X (Pascal) GPU has been used for the models LiteralE, KBLN, and MTKGNN.

Runtime As in the experiments with text literals, not all the models in the experiments with numeric literals are implemented in the same environment, i.e., for TransEA the code that is released by the authors of the paper is used and for the other models the code that is provided by the authors of LiteralE are used. Therefore, direct comparison of the runtime of TransEA and the other models would not be possible. However, the runtime of each of the models is computed on FB15K dataset so as to give insights into the models computational complexity. The running time of TransEA is 3.271 ms per a single iteration with batch size of 4831. For the other models their runtime for a single iteration of batch size 128 is shown in Table 9. Note that the runtime results reported here are the average over runtime values of 1000 iterations.

Table 9

Runtime of models considered in the experiments with numeric literals. The resutls are per single iteration and reported in milliseconds

Models	Time (ms)
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	31.575
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	37.138
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	39.269
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	52.346
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	43.386
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	50.439
KBLN	86.825
DistMult	29.679
ComplEx	33.526
ConvE	40.970

Evaluation procedure and results The same evaluation metrics which are discussed in Section 6.2 has been used to evaluate the performance of the models with numeric literals on the link prediction task. As shown in Table 10, according to the overall result, the model KBLN has considerably better performance than the other models in all metrics except MR. The results from the $\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ model show that it is capable to produce a highly competitive performance having the second best results with respect to the same metrics. This is the case due to the fact that this model is able to handle the inverse relations in FB15K by applying the complex conjugate of an entity embedding when the entity is used as a tail and its normal embedding when it is the head. Moreover, the model KBLN also achieves better result when compared to the standard models without literals presented in Table 11 with respect to all metrics except MR.

Another possible analysis to make is to compare the results of the standard models presented in Table 11 with the results of their extensions shown in the ‘both head and tail Prediction’ part of Table 10. For instance, $\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ achieves better performance than its base model ComplEx according to all metrics which indicates that using numeric literals with ComplEx by applying the approach in LiteralE is beneficial. However, this is not the case with DistMult and ConvE. One reason for this can be the fact that the number of attributive triples used in our experiment is not as big as in the original paper of LiteralE, i.e, increasing the number of numeric literals may improve the result as already seen in the original paper of literalE.

On the other hand, referring to the overall result on FB15K-237 dataset as shown in Table 12, the model $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ outperforms the other models according to all metrics. This fact along with the results in Table 13 indicates that applying LiteralE to DistMult on FB15K-237 provides better performance than applying it to other baseline models. Note that the reason for $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ model to achieve the best result on FB15K-237 dataset, as comapred to FB15K, may be due to the fact that this dataset does not have any symmetric relation, i.e., DistMult already has difficulties in modeling asymmetric relations on FB15k and adding literals might introduce noise but in case of FB15K-237, incorporating literals improves DistMult because symmetric relations are removed. Regarding the transformation functions g and ${\mathit{g}}_{\mathrm{lin}}$ , g leads to better results than ${\mathit{g}}_{\mathrm{lin}}$ on both datasets.

Table 10

Link prediction results on FB15K dataset using filtered setting

Models	MR	MRR	Hits@1	Hits@3	Hits@10
Head Prediction
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	121	0.495	0.383	0.559	0.697
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	71	0.76	0.697	0.801	0.876
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	52	0.612	0.51	0.678	0.795
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	72	0.581	0.479	0.642	0.762
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	63	0.768	0.707	0.809	0.878
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	49	0.72	0.65	0.762	0.849
KBLN	77	0.775	0.705	0.827	0.892
MTKGNN	73	0.702	0.617	0.758	0.855
TransEA	103	0.285	0.367	0.609	0.728
Tail Prediction
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	145	0.447	0.337	0.507	0.645
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	101	0.704	0.64	0.743	0.821
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	74	0.567	0.465	0.63	0.746
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	94	0.528	0.425	0.589	0.712
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	93	0.711	0.65	0.746	0.821
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	79	0.657	0.586	0.698	0.783
KBLN	90	0.727	0.656	0.776	0.848
MTKGNN	91	0.65	0.562	0.708	0.806
TransEA	75	0.314	0.417	0.671	0.805
Both Head and Tail Prediction
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	133	0.471	0.36	0.533	0.671
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	86	0.732	0.668	0.772	0.848
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	63	0.589	0.487	0.654	0.77
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	83	0.554	0.452	0.615	0.737
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	78	0.739	0.678	0.777	0.849
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	64	0.688	0.618	0.73	0.816
KBLN	83	0.751	0.68	0.801	0.87
MTKGNN	82	0.676	0.589	0.733	0.83
TransEA	74	0.299	0.392	0.64	0.766

Table 11

Link prediction results with models without literals on FB15K using filtered setting

FB15K
Models	MR	MRR	Hits@1	Hits@3	Hits@10
DistMult	119	0.67	0.589	0.723	0.817
ComplEx	127	0.692	0.614	0.742	0.833
ConvE	50	0.689	0.593	0.757	0.852
TransE	125	0.234	0.290	0.480	0.656

Table 12

Link prediction results on FB15K-237 dataset using filtered setting

Models	MR	MRR	Hits@1	Hits@3	Hits@10
Head Prediction
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	245	0.377	0.279	0.422	0.568
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	371	0.36	0.271	0.4	0.538
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	208	0.388	0.296	0.427	0.572
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	209	0.413	0.320	0.456	0.591
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	315	0.366	0.277	0.404	0.543
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	236	0.317	0.229	0.345	0.501
KBLN	381	0.386	0.295	0.426	0.564
MTKGNN	437	0.383	0.295	0.423	0.559
TransEA	389	0.111	0.094	0.197	0.342
Tail Prediction
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	426	0.195	0.119	0.214	0.349
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	575	0.17	0.104	0.185	0.306
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	362	0.187	0.112	0.204	0.338
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	359	0.215	0.137	0.234	0.371
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	493	0.175	0.106	0.19	0.312
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	459	0.131	0.07	0.137	0.256
KBLN	501	0.207	0.128	0.23	0.362
MTKGNN	580	0.191	0.12	0.208	0.338
TransEA	203	0.206	0.25	0.409	0.57
Both Head and Tail Prediction
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	335	0.286	0.199	0.318	0.458
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	473	0.265	0.187	0.292	0.422
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{{\mathit{g}}_{\mathrm{lin}}}$	285	0.287	0.204	0.315	0.455
$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	284	0.314	0.228	0.345	0.481
$\mathrm{ComplEx}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	404	0.27	0.191	0.297	0.427
$\mathrm{ConvE}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	347	0.224	0.149	0.241	0.378
KBLN	441	0.296	0.211	0.328	0.463
MTKGNN	508	0.287	0.207	0.315	0.448
TransEA	296	0.158	0.172	0.303	0.456

Table 13

Link prediction results with models without literals on FB15K-237 dataset using filtered setting

Models	MR	MRR	Hits@1	Hits@3	Hits@10
DistMult	630	0.280	0.201	0.309	0.438
ComplEx	623	0.288	0.207	0.318	0.448
ConvE	273	0.310	0.222	0.343	0.484
TransE	361	0.142	0.157	0.268	0.402

6.4. Experiment with images

Note that it is not possible to compare the whole of MKBE [55] with any other model as it is the only embedding model which utilizes the three types of literals together: text, numeric, and images. Therefore, its sub model $\text{S}+\text{I}$ which uses only images has been compared with the embedding model IKRL [77]. Since this comparison has already been done by the authors of MKBE [55], the result shown in Table 14 is directly taken from their paper. They have compared the models $\mathrm{DistMult}+\text{S}+\text{I}$ , $\mathrm{ConvE}+\text{S}+\text{I}$ , and IKRL where S stands for structure and I for Image. Both $\mathrm{DistMult}+\text{S}+\text{I}$ and $\mathrm{ConvE}+\text{S}+\text{I}$ are sub models of MKBE which use only relational triples and Images. The result indicates that $\mathrm{ConvE}+\text{S}+\text{I}$ outperforms the other two models in all metrics on the YAGO-10 dataset (refer to MKBE [55] for more details).

Table 14

MRR results on link prediction task on YAGO-10 taken from MKBE [55]

YAGO-10
Models	MRR	Hits@1	Hits@3	Hits@10
$\mathrm{DistMult}+\text{S}+\text{I}$	0.342	0.235	0.352	0.618
$\mathrm{ConvE}+\text{S}+\text{I}$	0.566	0.471	0.597	0.72
IKRL	0.509	0.423	0.556	0.663

Table 15

Link prediction results on FB15K and FB15K-237 datasets using filtered set

	Models	MR	MRR	Hits@1	Hits@3	Hits@10
FB15K
Head	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	72	0.581	0.479	0.642	0.762
	${\mathrm{DKRL}}_{\mathrm{Bern}}$	162	0.289	0.179	0.336	0.502
	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text	93	0.516	0.405	0.582	0.711
Tail	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	94	0.528	0.425	0.589	0.712
	${\mathrm{DKRL}}_{\mathrm{Bern}}$	122	0.356	0.24	0.408	0.577
	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text	119	0.463	0.351	0.532	0.66
All	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	83	0.554	0.452	0.615	0.737
	${\mathrm{DKRL}}_{\mathrm{Bern}}$	142	0.322	0.209	0.372	0.539
	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text	106	0.489	0.378	0.557	0.685
FB15K-237
Head	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	209	0.413	0.320	0.456	0.591
	${\mathrm{DKRL}}_{\mathrm{Bern}}$	145	0.294	0.184	0.337	0.507
	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text	207	0.416	0.323	0.462	0.594
Tail	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	359	0.215	0.137	0.234	0.371
	${\mathrm{DKRL}}_{\mathrm{Bern}}$	98	0.359	0.244	0.410	0.585
	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text	354	0.223	0.142	0.246	0.385
All	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$	284	0.314	0.228	0.345	0.481
	${\mathrm{DKRL}}_{\mathrm{Bern}}$	122	0.327	0.214	0.374	0.546
	$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text	280	0.319	0.232	0.354	0.489

6.5. Experiment with multi-modal literals

As discussed in Section 4, the existing multi-modal embeddings are categorized into two types: i) models with text literal, numeric literal and image literals and ii) models with text and numeric literals. However, since MKBE is the only model in the first category only its submodel $\mathit{S}+\mathit{I}$ could be compared with IKRL (see Section 6.4). Regarding the models with text and numeric literals, i.e., LiteralE with blocking and EAKGAE, they are not included in the experiment as well. The issue with EAKGAE is the same as that of KDCoE, i.e., it is trained on entity alignment task where as the reason for not having LiteralE with blocking is that its code is not publicly available. On the contrary, LiteralE (a model with numerics) has also been adopted to incorporate text literals in the experiments conducted by the authors. Similarly, in our experiment, the LiteralE approach has been tried out with the combination of text and numeric literals, i.e., the model $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text in Table 15. Then, the result has been compared with LiteralE with just numeric literals ( $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ ) and DKRL (a model using only text literals) so as to investigate the benefits of utilizing information represented by different types of literals.

$\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text is a model which applies the LiteralE approach to DistMult by using both numeric and text literals. Note that DistMult is chosen here as a baseline due to the reason that the best result in the experiments with numerics on the FB15K-237 dataset is achieved using this model as discussed in Section 6.3. The datasets listed in Table 7 are also used for this experiment along with additional text attributive triples which are descriptions of entities. $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text has also been compared with its numeric only equivalent $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ and ${\mathrm{DKRL}}_{\mathrm{Bern}}$ .

The experimental results obtained on the datasets FB15K and FB15K-237 are shown in Table 15. As the result indicates, combining text and numeric literals on FB15K dataset with $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text approach does not produce better results as compared to the other models $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ and ${\mathrm{DKRL}}_{\mathrm{Bern}}$ . As mentioned before, this dataset contains a set of inverse relations which may lead to having a triple whose inverse has a different label. Given the fact that DistMult fails to model such asymmetric relations, incorporating more literals with DistMult may introduce much noise than improving the performance. On the other hand, for FB15K-237 dataset, according to all the measures except MR, $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text model works better for the head entity prediction compared to the other two models. For tail entity prediction, ${\mathrm{DKRL}}_{\mathrm{Bern}}$ works better with respect to all measures for the same dataset.

7. Discussion and conclusion

Given the recent massive attention towards the use of KGs in various applications, different KG embedding techniques have been proposed to enable efficient use of KGs. In some of these techniques, an attempt has been made to utilize the information represented in literals present in KGs for a better quality embedding of the elements of the KGs, i.e., entities and relations. In this paper, a comprehensive survey of those KG embedding models with literals has been presented. The survey provides a detailed analysis and categorization of these models based on the proposed methodology along with their application scenarios and limitations. Moreover, various experiments on link prediction task on these models have been conducted so as to compare the models’ performances. [62] is a very recent work, not included in this survey, which re-evaluates KG completion models.

In this paper, two major research questions are formulated and presented in Section 3. The answers to these questions are given as follows:

RQ1 – How can structured (triples with object relations) and unstructured information (attributive triples) in the KGs be combined into the representation learning?

In order to use both data sources, i.e., triples with object relations and triples with data relations (attributive triples) together for representation learning, in broader terms, the following two techniques are considered in the models discussed in this paper:

• Handling literals separately: defining one task per data source like in TransEA or using a separate encoder for literals as in DKRL. The two tasks are trained simultaneously to make sure that for every entity the information available in both data sources are used to learn its embedding. The embeddings of the entities learned based on each data source can be unified in the vector space or not depending on how the model works. For instance, Jointly(Disp) learns unified representation for entities where as DKRL generates two representations per entity and do not force them to be unified.

• Incorporating literals directly into entity embeddings: as in LiteralE, one way is to extend a certain latent feature method by directly enriching the embeddings with information from literals via a learnable parameter and use the same scoring function from the latent feature method.

RQ2 – How can the heterogeneity of the types of literals present in the KGs be captured and combined into representation learning?

The following are some possible ways to combine different kind of literals, i.e., text, numeric, etc. together for representation learning.

• Encoding each type of literal separately: in order to capture the semantics of literals, different encoders can be used for different types of literals, for example, CNN for textual descriptions. Then, as shown in MKBE, each attributive triple can be treated same as structured triples and use a single scoring function for training.

• Incorporating information present in every kind of literal directly into the entity embedding: as in LiteralE, for a given entity, first the literals associated with it are encoded as vectors – using one vector per type of literal. Then, a mapping function is used to map all these vectors (including the structure-based vector representation of the entity) into a single vector.

As mentioned in Section 4 or seen from the result of the experiments in Section 6, these embedding models have different drawbacks such as:

• The effect that data types/units have on the semantics of literals has not been considered by any of the models.

• Most of the embedding models which make use of numerical literals, such as LiteralE, TransEA, MT-KGNN, and KBLN consider only the year part of date typed literals and ignore the month and day values. This hinders the ability to properly capture the information represented in such kind of literals. For example, given the following three date typed literal values:     "1999-10-29"^^xsd:date,     "1999-04-14"^^xsd:date,     "1999-10-30"^^xsd:date,a model which utilizes only the year part of these values considers all of them to be exactly the same despite the fact that the first date value is more close to the third value than it is to the second value.

• Most of the models also do not have a proper mechanism to handle multi-valued literals.

• The performance of most of the models is dependent on the dataset used for training and testing which shows that these models are not robust. For example, referring to Table 15, the results of the model $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text indicate that combining text and numeric literals yields better performance on FB15K-237 but not on FB15K due to the technique used in the model and the nature of the datasets (see Section 6.5).

• Not all the models are effective in combining different types of literals. For example, the performance of $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ -text (numeric + text literals), which combines text and numeric literals, on the dataset FB15K is lower as compared to $\mathrm{DistMult}\text{-}{\mathrm{LiteralE}}_{\mathit{g}}$ (only numeric literals).

• Only few approaches have been proposed for multi-modal KG embeddings and none of them take into consideration literals with URIs connected to items such as audio, video, or pdf files.

The above described shortcomings of the existing models clearly indicate that thorough investigation is needed on how to address different types of literals that obtain different inherent semantics. For instance, a possible perspective that arises by this detailed analysis is that there is a need to properly handle the data typed literals such as the values of the data relation weight given in kilogram and pound. One possible solution to target this issue could be to normalize these literal values to a standardized measures and to treat different measures like weights and lengths separately in the representation learning process.

One cannot expect that by leaving out available information present in the original KG, its latent representation as being only an approximation of the original KG, will perform equally well on tasks that depend on its semantic information content. Overall, the inclusion of datatyped literals with a proper representation of their semantics into the representation learning process will increase the model’s semantic content and might thereby lead to quality improvement.