Towards a question answering system over the Semantic Web

Abstract

With the development of the Semantic Web, a lot of new structured data has become available on the Web in the form of knowledge bases (KBs). Making this valuable data accessible and usable for end-users is one of the main goals of question answering (QA) over KBs. Most current QA systems query one KB, in one language (namely English). The existing approaches are not designed to be easily adaptable to new KBs and languages.

We first introduce a new approach for translating natural language questions to SPARQL queries. It is able to query several KBs simultaneously, in different languages, and can easily be ported to other KBs and languages. In our evaluation, the impact of our approach is proven using 5 different well-known and large KBs: Wikidata, DBpedia, MusicBrainz, DBLP and Freebase as well as 5 different languages namely English, German, French, Italian and Spanish. Second, we show how we integrated our approach, to make it easily accessible by the research community and by end-users.

To summarize, we provide a conceptional solution for multilingual, KB-agnostic question answering over the Semantic Web. The provided first approximation validates this concept.

1. Introduction

Question answering (QA) is a research field in computer science that started in the sixties [38]. In the Semantic Web, a lot of new structured data has become available in the form of knowledge bases (KBs). Nowadays, there are KBs about media, publications, geography, life sciences and more.1

¹

http://lod-cloud.net

The core purpose of a QA system over KBs is to retrieve the desired information from one or many KBs, using natural language questions. This is generally addressed by translating a natural language question to a SPARQL query. Current research does not address the challenge of multilingual, KB-agnostic QA for both full and keyword questions (Table 1).

Table 1

Selection of QA systems evaluated over the most popular benchmarks. We indicated their capabilities with respect to multilingual questions, different KBs and different typologies of questions (full = “well-formulated natural language questions”, key = “keyword questions”)

QA system	Lang	KBs	Type
gAnswer [66] (QALD-3 Winner)	En	DBpedia	Full
Xser [60] (QALD-4 & 5 Winner)	En	DBpedia	Full
UTQA [59]	En, es, fs	DBpedia	Full
Jain [35] (WebQuestions Winner)	En	Freebase	Full
Lukovnikov [39] (SimpleQuestions Winner)	En	Freebase	Full
Ask Platypus (https://askplatyp.us)	En	Wikidata	Full
WDAqua-core1	En, fr, de, it, es	Wikidata, DBpedia, Freebase, DBLP, MusicBrainz	Full & key

There are multiple reasons for that. Many QA approaches rely on language-specific tools (NLP tools), e.g., SemGraphQA [3], gAnswer [66] and Xser [60]. Therefore, it is difficult or impossible to port them to a language-agnostic system. Additionally, many approaches make particular assumptions on how the knowledge is modelled in a given KB (generally referred to as “structural gap” [12]). This is the case of AskNow [19] and DEANNA [61].

There are also approaches which are difficult to port to new languages or KBs because they need a lot of training data which is difficult and expensive to create. This is for example the case of Bordes et al. [5]. Finally there are approaches where it was not proven that they scale well. This is for example the case of SINA [48].

In this paper, we present an algorithm that addresses all of the above drawbacks and that can compete, in terms of F-measure, with many existing approaches.

This publication is organized as follows. In Section 2, we present related works. In Section 3 and 4, we describe the algorithm providing the foundations of our approach. In Section 5, we provide the results of our evaluation over different benchmarks. In Section 6, we show how we implemented our algorithm as a service so that it is easily accessible to the research community, and how we extended a series of existing services so that our approach can be directly used by end-users. We conclude with Section 7.

2. Related work

In the context of QA, a large number of systems have been developed in the last years. For a complete overview, we refer to [12]. Most of them were evaluated on one of the following three popular benchmarks: WebQuestions [4], SimpleQuestions [5] and QALD.2

²

http://www.sc.cit-ec.uni-bielefeld.de/qald/

WebQuestions contains 5,810 questions that can be answered by one reified statement. SimpleQuestions contains 108,442 questions that can be answered using a single, binary-relation. The QALD challenge versions include more complex questions than the previous ones, and contain between 100 and 450 questions, and are therefore, compared to the other, small datasets.

The high number of questions of WebQuestions and SimpleQuestions led to many supervised-learning approaches for QA. Especially deep learning approaches became very popular in the recent years like Bordes et al. [5] and Zhang et al. [62]. The main drawback of these approaches is the training data itself. Creating a new training dataset for a new language or a new KB might be very expensive. For example, Berant et al. [4], report that they spent several thousands of dollars for the creation of WebQuestions using Amazon Mechanical Turk. The problem of adapting these approaches to new dataset and languages can also be seen by the fact that all these systems work only for English questions over Freebase.

A list of the QA systems that were evaluated with QALD-3, QALD-4, QALD-5, QALD-6, QALD-7, QALD-8 can be found in Table 3. According to [12] less than 10% of the approaches were applied to more than one language and 5% to more than one KB. The reason is the heavy use of NLP tools or NL features like in Xser [60], gAnswer [66] or QuerioDali [37].

The problem of QA in English over MusicBrainz3

³

https://musicbrainz.org

was proposed in QALD-1, in the year 2011. Two QA systems tackled this problem. Since then the MusicBrainz KB4

⁴

https://github.com/LinkedBrainz/MusicBrainz-R2RML

completely changed. We are not aware of any QA system over DBLP.5

⁵

http://dblp.uni-trier.de

In summary, most QA systems work only in English and over one KB. Multilinguality is poorly addressed while portability is not addressed at all. The few systems that address multilinguality rely on syntactic parsing techniques [31,51,59].

The fact that QA systems often reuse existing techniques and need several services to be exposed to the end-user, leads to the idea of developing QA systems in a modular way. At least four frameworks tried to achieve this goal: QALL-ME [21], openQA [40], the Open Knowledge Base and Question-Answering (OKBQA) challenge6

⁶

http://www.okbqa.org/

and Qanary [6 ,13 ,49]. We integrated our system as a Qanary QA component called WDAqua-core1. We choose Qanary for two reasons. First, it offers a series of off-the-shelf services related to QA systems and second, it allows to freely configure a QA system based on existing QA components.

Fig. 1.

Conceptual overview of the approach.

3. Approach for QA over Knowledge Bases

In this section, we present our multilingual, KB-agnostic approach for QA. It is based on the observation that many questions can be understood from the semantics of the words in the question while the syntax of the question has less importance. For example, consider the question “Give me actors born in Berlin”. This question can be reformulated in many ways like “In Berlin were born which actors?” or as a keyword question “Berlin, actors, born in”. In this case by knowing the semantics of the words “Berlin”, “actors”, “born”, we are able to deduce the intention of the user. This holds for many questions, i.e. they can be correctly interpreted without considering the syntax as the semantics of the words is sufficient for them. Taking advantage of this observation is the main idea of our approach. The KB encodes the semantics of the words and it can tell what is the most probable interpretation of the question (w.r.t. the knowledge model described by the KB).

Our approach is decomposed in 4 steps: question expansion, query construction, query ranking and response decision. A conceptual overview is given in Fig. 1. In the following, the processing steps are described. As a running example, we consider the question “Give me philosophers born in Saint-Etienne”. For the sake of simplicity, we use DBpedia as KB to answer the question. However, it is important to recognize that no assumptions either about the language or the KB are made. Hence, even the processing of the running example is language- and KB-agnostic.

Table 2

Expansion step for the question “Give me philosophers born in Saint Étienne”. The first column enumerates the candidates that were found. Here, 117 possible entities, properties and classes were found from the question. The second, third and fourth columns indicate the position of the n-gram in the question and the n-gram itself. The last column is for the associated IRI. Note that many possible meanings are considered: line 9 says that “born” may refer to a crater, line 52 that “saint” may refer to a software and line 114 that the string “Saint Étienne” may refer to a band

n	Start	End	n-gram	Resource
1	2	3	Philosophers	dbrc:Philosophes
2	2	3	Philosophers	dbr:Philosophes
3	2	3	Philosophers	dbo:Philosopher
4	2	3	Philosophers	dbrc:Philosophers
5	2	3	Philosophers	dbr:Philosopher
6	2	3	Philosophers	dbr:Philosophy
7	2	3	Philosophers	dbo:philosophicalSchool
8	3	4	Born	dbr:Born,_Netherlands
9	3	4	Born	dbr:Born_(crater)
10	3	4	Born	dbr:Born_auf_dem_Dar?
11	3	4	Born	dbr:Born,_Saxony-Anhalt
⋮
42	3	4	Born	dbp:bornAs
43	3	4	Born	dbo:birthDate
44	3	4	Born	dbo:birthName
45	3	4	Born	dbp:bornDay
46	3	4	Born	dbp:bornYear
47	3	4	Born	dbp:bornDate
48	3	5	Born in	dbp:bornIn
49	3	5	Born in	dbo:birthPlace
50	3	5	Born in	dbo:hometown

n	Start	End	n-gram	Resource
52	5	6	Saint	dbr:SAINT_(software)
53	5	6	Saint	dbr:Saint
54	5	6	Saint	dbr:Boxers_and_Saints
55	5	6	Saint	dbr:Utah_Saints
56	5	6	Saint	dbr:Saints,_Luton
57	5	6	Saint	dbr:Baba_Brooks
58	5	6	Saint	dbr:Battle_of_the_Saintes
59	5	6	Saint	dbr:New_York_Saints
⋮
106	5	6	Saint	dbp:saintPatron
107	5	6	Saint	dbp:saintsDraft
108	5	6	Saint	dbp:saintsSince
109	5	6	Saint	dbo:patronSaint
110	5	6	Saint	dbp:saintsCollege
111	5	6	Saint	dbp:patronSaintOf
112	5	6	Saint	dbp:patronSaint(s)
113	5	6	Saint	dbp:patronSaint’sDay
114	5	7	Saint etienne	dbr:Saint_Etienne_(band)
115	5	7	Saint etienne	dbr:Saint_Etienne
116	5	7	Saint etienne	dbr:Saint-Étienne
117	6	7	Etienne	dbr:Étienne

3.1. Expansion

Following a recent survey [12], we call a lexicalization, a name of an entity, a property or a class. For example, “first man on the moon” and “Neil Armstrong” are both lexicalizations of dbr:Neil_Armstrong. In this step, we want to identify all Internationalized Resource Identifiers (IRIs) of entities, properties and classes, which the question could refer to. To achieve this, we use the following rules:

All IRIs are searched whose lexicalization (up to stemming) is a word n-gram (up to stemming) in the question.

If an n-gram is a stop word (like “is”, “are”, “of”, “give”, …), then we exclude the IRIs associated to it. This is due to the observation that the semantics are important to understand a question and the fact that stop words do not carry a lot of semantics. Moreover, by removing the stop words the time needed in the next step is decreased.

An example is given in Table 2. The stop words and the lexicalizations used for the different languages and KBs are described in Section 5.1. In this part, we used the well-known Apache Lucene7

⁷

https://lucene.apache.org

technology which allows fast retrieval, while providing a small disk and memory footprint.

3.2. Query construction

In this step, we construct a set of queries that represent possible interpretations of the given question within the given KB. Therefore, we heavily utilize the semantics encoded into the particular KB. We start with a set R of IRIs from the previous step. The goal is to construct all possible queries containing the IRIs in R which give a non-empty result-set. Let V be the set of variables. Based on the complexity of the questions in current benchmarks, we restrict our approach to queries satisfying 4 patterns: $$\begin{array}{ll}\hfill & \mathtt{\text{SELECT / ASK var}}\\ \hfill & \mathtt{\text{WHERE { s1 s2 s3 . }}}\\ \hfill & \mathtt{\text{SELECT / ASK var}}\\ \hfill & \mathtt{\text{WHERE { s1 s2 s3 .}}\\ \hfill & \mathtt{\text{s4 s5 s6 . }}}\end{array}$$ with

and

These correspond to all queries containing one or two triple patterns that can be created starting from the IRIs in R. Moreover, for entity linking, we add the following two patterns: $$\begin{array}{ll}\hfill & \mathtt{\text{SELECT ?x}}\\ \hfill & \mathtt{\text{WHERE { VALUES ?x {iri} . }}}\\ \hfill & \mathtt{\text{SELECT ?x}}\\ \hfill & \mathtt{\text{WHERE { VALUES ?x {iri} . }}\\ \hfill & \mathtt{\text{iri ?p iri1 . }}}\end{array}$$ with iri, iri1 ∈ R. These correspond to all queries returning directly one of the IRIs in R with possibly one additional triple.

Note that these last queries just give back directly an entity and should be generated for a question like: “What is Apple Company?” or “Who is Marie Curie?”. An example of generated queries is given in Fig. 2.

The main challenge is the efficient construction of these SPARQL queries. The main idea is to perform in the KB graph a breadth-first search of depth 2 starting from every IRI in R. While exploring the KB for all IRIs ${\mathit{r}}_{\mathit{j}}\in \mathit{R}$ (where ${\mathit{r}}_{\mathit{j}}\ne {\mathit{r}}_{\mathit{i}}$ ) the distance ${\mathit{d}}_{{\mathit{r}}_{\mathit{i}}\mathrm{,}{\mathit{r}}_{\mathit{j}}}$ between two resources is stored. These numbers are used when constructing the queries shown above. For a detailed algorithm of the query construction phase, please see Section 4. Concluding, in this section, we computed a set of possible SPARQL queries (candidates). They are driven by the lexicalizations computed in Section 3.1 and represent the possible intentions expressed by the question of the user.

Fig. 2.

Some of the 395 queries constructed for the question “Give me philosophers born in Saint Etienne”. Note that all queries could be semantically related to the question. The second one is returning “Saint-Etienne” as a band, the third one the birth date of people born in the city of “Saint-Etienne” and the forth one the birth date of persons related to philosophy.

3.3. Ranking

Now the computed candidates need to be ordered by their probability of answering the question correctly. Hence, we rank them based on the following features:

Number of the words in the question which are covered by the query. For example, the first query in Fig. 2 is covering two words (“Saint” and “born”, where “born” is covered by the property dbo:hometown).

The edit distance of the label of the resource and the word it is associated to. For example, the edit distance between the label of dbp:bornYear (which is “born year”) and the word “born” is 5.

The sum of the relevance of the resources, (e.g. the number of inlinks and the number of outlinks of a resource). This is a knowledge base independent choice, but it is also possible to use a specific score for a KB (like page-rank [16]).

The number of variables in the query.

The number of triples in the query.

If no training data is available, then we rank the queries using a linear combination of the above 5 features, where the weights are determined manually. Otherwise we assume a training dataset of questions together with the corresponding answers set, which can be used to calculate the F-measure for each of the SPARQL query candidates. As a ranking objective, we want to order the SPARQL query candidates in descending order with respect to the F-measure. In our implementation we rank the queries using RankLib8

⁸

https://sourceforge.net/p/lemur/wiki/RankLib/

with Coordinate Ascent [41]. At test time the learned model is used to rank the queries, the top-ranked query is executed against a SPARQL endpoint, and the result is computed. An example is given in Fig. 3. Note that, we do not use syntactic features. However, it is possible to use them to further improve the ranking.

Fig. 3.

The top 4 generated queries for the question “Give me philosophers born in Saint Étienne”. (1) is the query that best matches the question; (2) gives philosophical schools of people born in Saint-Étienne; (3), (4) give people born in Saint-Étienne or that live in Saint-Étienne. The order can be seen as a decreasing approximation to what was asked.

3.4. Answer decision

The computations in the previous section lead to a list of ranked SPARQL queries candidates representing our possible interpretations of the user’s intentions. We could directly give back the result of first ranked query from the previous step. This will (nearly) always generate an answer. However, there are situations where no suitable interpretation is generated for the question, or where the question is not answerable over the given KB. To determine if such a situation occurred, we add an additional step in which we decide if the result-set of the first query should be returned or if the system should not give back any result. This corresponds to a binary classification problem. We train a model based on logistic regression. We use a training set consisting of SPARQL queries and two labels equal to True or False. True indicates if the F-score of the SPARQL query is greater than a threshold ${\mathit{\theta }}_1$ or false otherwise. Once the model is trained, it can compute a confidence score ${\mathit{p}}_{\mathit{Q}}\in [0\mathrm{,}1]$ for a query Q. In our exemplary implementation we assume a correctly ordered list of SPARQL query candidates computed in Section 3.3. Hence, it only needs to be checked whether ${\mathit{p}}_{{\mathit{Q}}_1}⩾{\mathit{\theta }}_2$ is true for the first ranked query ${\mathit{Q}}_1$ of the SPARQL query candidates, or otherwise it is assumed that the whole candidate list does not reflect the user’s intention. Hence, we refuse to answer the question. We answer the question if it is above a threshold ${\mathit{\theta }}_2$ otherwise we do not answer it. Note that ${\mathit{p}}_{\mathit{Q}}$ can be interpreted as the confidence that the QA system has in the generated SPARQL query Q, i.e. in the generated answer.

3.5. Multiple KBs

Note that the approach can also be extended, as it is, to multiple KBs. In the query expansion step, one has just to take in consideration the labels of all KBs. In the query construction step, one can consider multiple KBs as one graph having multiple unconnected components. The query ranking and answer decision step are literally the same.

3.6. Discussion

Overall, we follow a combinatorial approach with efficient pruning, that relies on the semantics encoded in the underlying KB.

In the following, we want to emphasize the advantages of this approach using some examples.

Joint disambiguation of entities and relations: For example, for interpreting the question “How many inhabitants has Paris?” between the hundreds of different meanings of “Paris” and “inhabitants” the top ranked queries contain the resources called “Paris” which are cities, and the property indicating the population, because only these make sense semantically.

Portability to different KBs: One problem in QA over KBs is the semantic gap, i.e. the difference between how we think that the knowledge is encoded in the KB and how it actually is. For example, in our approach, for the question “What is the capital of France?”, we generate the query

which probably most users would have expected, but also the query

which refers to an overview article in Wikipedia about the capitals of France and that most of the users would probably not expect. This important feature allows to port the approach to different KBs while it is independent of how the knowledge is encoded.

Ability to bridge over implicit relations: We are able to bridge over implicit relations. For example, given “Give me German mathematicians” the following query is computed:

Here ?p1 is:

dbo:field

dbo:occupation,

dbo:profession

and ?p2 is:

• dbo:nationality,

• dbo:birthPlace,

• dbo:deathPlace,

• dbo:residence.

Note that all these properties could be intended for the given question, even if dbo:deathPlace could be seen as an over-generalization.

Easy to port to new languages: The only parts where the language is relevant are the stop word removal and stemming. Since these are very easy to adapt to new languages, one can port the approach easily to other languages.

Permanent system refinement: It is possible to improve the system over time. The system generates multiple queries. This fact can be used to easily create new training datasets as is shown in [14]. Using these datasets one can refine the ranker to perform better on the asked questions.

System robust to malformed questions and keyword questions: We are not using part-of-speech tagging or dependency parsers in the approach which makes it very robust to malformed questions. For this reason, keyword questions are also supported.

A disadvantage of our exemplary implementation is that the identification of relations relies on a dictionary. Note that, methods not based on dictionaries follow one of the following strategies. Either they try to learn ways to express the relation from big training corpora (like in [5]), s.t. the problem is shifted to create suitable training sets. Alternatively, text corpora are used to either extract lexicalizations for properties (like in [4]) or learn word embeddings (like in [31]). Hence, possible improvements might be applied to this task in the future.

To conclude, the proposed approach uses the knowledge encoded in the KB to construct candidate queries. This is novel and responsible for the main distinctive features of the approach: easy portability to new languages, easy portability to new KBs and robustness to different types of questions.

4. Fast candidate generation

In this section, we explain how the SPARQL queries described in Section 3.2 can be constructed efficiently.

Let R be a set of resources. We consider the KB as a directed labeled graph G:

Definition 1 (Graph).

A directed labeled graph is an ordered pair $\mathit{G}=\mathrm{(}\mathit{V}\mathrm{,}\mathit{E}\mathrm{,}\mathit{f}\mathrm{)}$ , such that:

V is a non-empty set, called the vertex set;

E is a set, called edge set, such that $\mathit{E}\subset \{\mathrm{(}\mathit{v}\mathrm{,}\mathit{w}\mathrm{)}:\mathit{v}\mathrm{,}\mathit{w}\in \mathit{V}\}$ , i.e. a subset of the pairs of V;

For a set L called labeled set, f is a function $\mathit{f}:\mathit{E}\to \mathit{L}$ , i.e. a function that assigns to each edge a label $\mathit{p}\in \mathit{L}$ . We indicate an edge with label p as $\mathit{e}=\mathrm{(}\mathit{v}\mathrm{,}\mathit{p}\mathrm{,}\mathit{w}\mathrm{)}$ .

To compute the pairwise distance in G between every resource in R, we do a breadth-first search from every resource in R in an undirected way (i.e. we traverse the graph in both directions).

We define a distance function d as follows. Assume we start from a vertex r and find the following two edges ${\mathit{e}}_1=\mathrm{(}\mathit{r}\mathrm{,}{\mathit{p}}_1\mathrm{,}{\mathit{r}}_1\mathrm{)}$ , ${\mathit{e}}_2=\mathrm{(}{\mathit{r}}_1\mathrm{,}{\mathit{p}}_2\mathrm{,}{\mathit{r}}_2\mathrm{)}$ . We say that ${\mathit{d}}_{\mathit{r}\mathrm{,}{\mathit{p}}_1}=1$ , ${\mathit{d}}_{\mathit{r}\mathrm{,}{\mathit{r}}_1}=2$ , ${\mathit{d}}_{\mathit{r}\mathrm{,}{\mathit{p}}_2}=3$ and so on. When an edge is traversed in the opposite direction, we add a minus sign. For example, given the edges ${\mathit{e}}_1=\mathrm{(}\mathit{r}\mathrm{,}{\mathit{p}}_1\mathrm{,}{\mathit{r}}_1\mathrm{)}$ and ${\mathit{e}}_2=\mathrm{(}{\mathit{r}}_2\mathrm{,}{\mathit{p}}_2\mathrm{,}{\mathit{r}}_1\mathrm{)}$ , we say ${\mathit{d}}_{\mathit{r}\mathrm{,}{\mathit{p}}_2}=-3$ . For a vertex or edge r, and a variable x we artificially set ${\mathit{d}}_{\mathit{r}\mathrm{,}\mathit{x}}$ to be any possible integer number. Moreover, we set ${\mathit{d}}_{\mathit{x}\mathrm{,}\mathit{y}}={\mathit{d}}_{\mathit{y}\mathrm{,}\mathit{x}}$ for any $\mathit{x}\mathrm{,}\mathit{y}$ . The algorithm to compute these numbers can be found in Algorithm 1.

Algorithm 1:

Algorithm to compute the pairwise distance between every resource in a set R appearing in a KB

The algorithm of our exemplary implementation simply traverses the graph starting from the nodes in R in a breadth-first search manner and keeps track of the distances as defined above. The breadth-first search is done by using HDT [20] as an indexing structure.9

⁹

https://www.w3.org/Submission/2011/03/

Note that HDT was originally developed as an exchange format for RDF files that is queryable. A rarely mentioned feature of HDT is that it is perfectly suitable for performing breadth-first search operations over RDF data. In HDT, the RDF graph is stored as an adjacency list which is an ideal data structure for breadth-first search operations. This is not the case for traditional triple-stores. The use of HDT at this point is key for two reasons, (1) the performance of the breadth-first search operations, and (2) the low footprint of the index in terms of disk and memory space. Roughly, a 100 GB RDF dump can be compressed to a HDT file of a size of approx. 10 GB [20].

Based on the numbers above, we now want to construct all triple patterns with K triples and one projection variable recursively. Given a triple pattern T, we only want to build connected triple-pattern while adding triples to T. This can be done recursively using the algorithm described in Algorithm 2. Note that thanks to the numbers collected during the breadth-first search operations, this can be performed very fast. Once the triple patterns are constructed, one can choose any of the variables, which are in subject or object position, as a projection variable.

The decision to generate a SELECT or/and ASK query, is made depending on some regular expressions over the beginning of the question.

5. Evaluation

To validate the approach w.r.t. multilinguality, portability and robustness, we evaluated our approach using multiple benchmarks for QA that appeared in the last years. The different benchmarks are not comparable and they focus on different aspects of QA. For example SimpleQuestions focuses on questions that can be solved by one simple triple-pattern, while LC-QuAD focuses on more complex questions. Moreover, the QALD questions address different challenges including multilinguality and the use of keyword questions. Unlike previous works, we do not focus on one benchmark, but we analyze the behaviour of our approach under different scenarios. This is important, because it shows that our approach is not adapted to one particular benchmark, as it is often done by existing QA systems, and proves its portability.

We tested our approach on 5 different datasets namely Wikidata,10

¹⁰

https://www.wikidata.org/

DBpedia,11

¹¹

http://dbpedia.org

MusicBrainz,12

¹²

https://musicbrainz.org

DBLP13

¹³

http://dblp.uni-trier.de

and Freebase.14

¹⁴

https://developers.google.com/freebase/

Moreover, we evaluated our approach on five different languages namely: English, German, French, Italian and Spanish. First, we describe how we selected stop words and collected lexicalizations for the different languages and KBs, then we describe and discuss our results.

5.1. Stop words and lexicalizations

As stop words, we use the lists, for the different languages, provided by Lucene, together with some words which are very frequent in questions like “what”, “which”, “give”.

Algorithm 2:

Recursive algorithm to create all connected triple patterns from a set R of resources with maximal K triple patterns. L contains the triple patterns created recursively and ${\mathit{L}}^{\mathrm{(}\mathit{k}\mathrm{)}}$ indicates the triple patterns with exactly k triples. Moreover, ${\mathit{x}}_{\mathit{k}\mathrm{,}1}\mathrm{,}{\mathit{x}}_{\mathit{k}\mathrm{,}2}\mathrm{,}{\mathit{x}}_{\mathit{k}\mathrm{,}3}$ are new variables that are added in step k. Note that the “if not” conditions correspond to the four possibilities a), b), c), d) of joining two triples which are depicted above. Note that they are very often not fulfilled. This guarantees the speed of the process

Depending on the KB, we followed different strategies to collect lexicalizations. Since Wikidata has a rich number of lexicalizations, we simply took all lexicalizations associated to a resource through rdfs:label,15

¹⁵

rdfs: http://www.w3.org/2000/01/rdf-schema#

skos:prefLabel16

¹⁶

skos: http://www.w3.org/2004/02/skos/core#

and skos: altLabel. For DBpedia, we only used the English DBpedia, where first all lexicalizations associated to a resource through the rdfs:label property were collected. Secondly, we followed the disambiguation and redirect links to get additional ones and took also into account available demonyms dbo:demonym (i.e. to dbr:Europe we associate also the lexicalization “European”). Thirdly, by following the inter-language links, we associated the labels from the other languages to the resources. DBpedia properties are poorly covered with lexicalizations, especially when compared to Wikidata. For example, the property dbo:birthPlace has only one lexicalization namely “birth place”, while the corresponding property over Wikidata P19 has 10 English lexicalizations like “birthplace”, “born in”, “location born”, “birth city”. In our exemplary implementation two strategies were implemented. First, while aiming at a QA system for the Semantic Web we also can take into account interlinkings between properties of distinguished KBs, s.t. lexicalizations are merged from all KBs currently considered. There, the owl:sameAs links from DBpedia relations to Wikidata are used and every lexicalization present in Wikidata is associated to the corresponding DBpedia relation. Secondly, the DBpedia abstracts are used to find more lexicalizations for the relations. To find new lexicalizations of a property $\mathit{\text{p}}$ we follow the strategy proposed by [22]. We extracted from the KB the subject-object pairs (x,y) that are connected by p. Then the abstracts are scanned and all sentences are retrieved which contain both $\mathit{label}\mathrm{(}\mathit{x}\mathrm{)}$ and $\mathit{label}\mathrm{(}\mathit{y}\mathrm{)}$ . At the end, the segments of text between $\mathit{label}\mathrm{(}\mathit{x}\mathrm{)}$ and $\mathit{label}\mathrm{(}\mathit{y}\mathrm{)}$ , or $\mathit{label}\mathrm{(}\mathit{y}\mathrm{)}$ and $\mathit{label}\mathrm{(}\mathit{x}\mathrm{)}$ are extracted. We rank the extracted text segments and we choose the most frequent ones. This was done only for English.

For MusicBrainz we used the lexicalizations attached to purl:title,17

¹⁷

purl: http://purl.org/dc/elements/1.1/

foaf:name,18

¹⁸

foaf: http://xmlns.com/foaf/.

skos:altLabel and rdfs:label. For DBLP only the one attached to rdfs:label. Note, MusicBrainz and DBLP contain only few properties. We aligned them manually with Wikidata and moved the lexicalizations from one KB to the other. The mappings can be found under http://goo.gl/ujbwFW and http://goo.gl/ftzegZ respectively. This took in total 1 hour of manual work.

For Freebase, we considered the lexicalizations attached to rdfs:label. We also followed the few available links to Wikidata. Finally, we took the 20 most prominent properties in the training set of the SimpleQuestions benchmark and looked at the lexicalizations of them in the first 100 questions of SimpleQuestions. We extracted manually the lexicalizations for them. This took 1 hour of manual work. We did not use the other (75,810 training and 10,845 validation) questions, i.e., unlike previous works we only took a small fraction of the available training data.

We want to briefly discuss the strategies we used here. We do not see any option other than manually indicating the lexicalizations for instances. For example, in MusicBrainz the property purl:title must be selected otherwise one cannot find any existing album. On the other hand there could be a property expressing the cover description of an album. We do not see any method to determine automatically why the property purl:title should be used as a lexicalization, while not the one about the cover description. We therefore think that the only available solution is to make the standard more clear on how to express such an information. Regarding the lexicalization of relations, the situation is different. The literature contains a number of approaches that can be used to generate them. For an overview, we refer to Section 7 in [12]. All works suppose one of the three following situations: there is some free text that also contains such knowledge, or they are expressed in some external databases like WordNet, or a training repository to learn them from question and answer pairs is available. On knowledge bases like Musicbrainz any of the 3 alternatives is not available, so also in this case we believe that the manual work cannot be avoided.

Table 3

This table summarizes the results obtained by the QA systems evaluated with QALD-3 (over DBpedia 3.8), QALD-4 (over DBpedia 3.9), QALD-5 (over DBpedia 2014), QALD-6 (over DBpedia 2015-10), QALD-7 (2016-04), QALD-8 (2016-10). We indicated with “∗” the systems that did not participate directly in the challenges, but were evaluated on the same benchmark afterwards. We indicate the average running times of a query for the systems where we found them. Even if the runtime evaluations were executed on different hardware, it still helps to give an idea about the scalability

QA system	Language	Type	Total	Precision	Recall	F-measure	Runtime	Ref
QALD-3
WDAqua-core1	En	Full	100	0.64	0.42	0.51	1.01	-
WDAqua-core1	En	Key	100	0.71	0.37	0.48	0.79	-
WDAqua-core1	De	Key	100	0.79	0.31	0.45	0.22	-
WDAqua-core1	De	Full	100	0.79	0.28	0.42	0.30	-
WDAqua-core1	Fr	Key	100	0.83	0.27	0.41	0.26	-
gAnswer [66]∗	En	full	100	0.40	0.40	0.40	≈1 s	[66]
WDAqua-core1	Fr	Full	100	0.70	0.26	0.38	0.37	-
WDAqua-core1	Es	Full	100	0.77	0.24	0.37	0.27	-
WDAqua-core1	It	Full	100	0.79	0.23	0.36	0.30	-
WDAqua-core1	It	Key	100	0.84	0.23	0.36	0.24	-
WDAqua-core1	Es	Key	100	0.80	0.23	0.36	0.23	-
RTV [28]	En	Full	99	0.32	0.34	0.33	-	[7]
Intui2 [17]	En	Full	99	0.32	0.32	0.32	-	[7]
SINA [48]∗	En	Full	100	0.32	0.32	0.32	≈10-20 s	[48]
DEANNA [61]∗	En	Full	100	0.21	0.21	0.21	≈1-50 s	[66]
SWIP [45]	En	Full	99	0.16	0.17	0.17	-	[7]
Zhu et al. [64]∗	En	Full	99	0.38	0.42	0.38	-	[64]
QALD-4
Xser [60]	En	Full	50	0.72	0.71	0.72	-	[53]
WDAqua-core1	En	Key	50	0.76	0.40	0.52	0.32 s	-
WDAqua-core1	En	Full	50	0.56	0.30	0.39	0.46 s	-
gAnswer [66]	En	Full	50	0.37	0.37	0.37	0.973 s	[53]
CASIA [33]	En	Full	50	0.32	0.40	0.36	-	[53]
WDAqua-core1	De	Key	50	0.92	0.20	0.33	0.04 s	-
WDAqua-core1	Fr	Key	50	0.92	0.20	0.33	0.06 s	-
WDAqua-core1	It	Key	50	0.92	0.20	0.33	0.04 s	-
WDAqua-core1	Es	Key	50	0.92	0.20	0.33	0.05 s	-
WDAqua-core1	De	Full	50	0.90	0.20	0.32	0.06 s	-
WDAqua-core1	It	Full	50	0.92	0.20	0.32	0.16 s	-
WDAqua-core1	Es	Full	50	0.90	0.20	0.32	0.06 s	-
WDAqua-core1	Fr	Full	50	0.86	0.18	0.29	0.09 s	-
Intui3 [18]	En	Full	50	0.23	0.25	0.24	-	[53]
ISOFT [44]	En	Full	50	0.21	0.26	0.23	-	[53]
Hakimov [32]∗	En	Full	50	0.52	0.13	0.21	-	[32]

Table 3

(Continued)

QA system	Language	Type	Total	Precision	Recall	F-measure	Runtime	Ref
QALD-5
Xser [60]	En	Full	50	0.74	0.72	0.73	-	[54]
UTQA [59]	En	Full	50	-	-	0.65	-	[59]
UTQA [59]	Es	Full	50	0.55	0.53	0.54	-	[59]
UTQA [59]	Fs	Full	50	0.53	0.51	0.52	-	[59]
WDAqua-core1	En	Full	50	0.56	0.41	0.47	0.62 s	-
WDAqua-core1	En	Key	50	0.60	0.27	0.37	0.50 s	-
AskNow [19]	En	Full	50	0.32	0.34	0.33		[19]
QAnswer [47]	En	Full	50	0.34	0.26	0.29	-	[54]
WDAqua-core1	De	Full	50	0.92	0.16	0.28	0.20 s	-
WDAqua-core1	De	Key	50	0.90	0.16	0.28	0.19 s	-
WDAqua-core1	Fr	Full	50	0.90	0.16	0.28	0.19 s	-
WDAqua-core1	Fr	Key	50	0.90	0.16	0.28	0.18 s	-
WDAqua-core1	It	Full	50	0.88	0.18	0.30	0.20 s	-
WDAqua-core1	It	Key	50	0.90	0.16	0.28	0.18 s	-
WDAqua-core1	Es	Full	50	0.88	0.14	0.25	0.20 s	-
WDAqua-core1	Es	Key	50	0.90	0.14	0.25	0.20 s	-
SemGraphQA [3]	En	Full	50	0.19	0.20	0.20	-	[54]
YodaQA [2]	En	Full	50	0.18	0.17	0.18	-	[54]
QuerioDali [37]	En	Full	50	?	?	?	?	[37]
QALD-6
WDAqua-core1	En	Full	100	0.55	0.34	0.42	1.28 s	-
WDAqua-core1	De	Full	100	0.73	0.29	0.41	0.41 s	-
WDAqua-core1	De	Key	100	0.85	0.27	0.41	0.30 s	-
WDAqua-core1	En	Key	100	0.51	0.30	0.37	1.00 s	-
SemGraphQA [3]	En	Full	100	0.70	0.25	0.37	-	[55]
WDAqua-core1	Fr	Key	100	0.78	0.23	0.36	0.34 s	-
WDAqua-core1	Fr	Full	100	0.57	0.22	0.32	0.46 s	-
WDAqua-core1	Es	Full	100	0.69	0.19	0.30	0.45 s	-
WDAqua-core1	Es	Key	100	0.83	0.18	0.30	0.35 s	-
WDAqua-core1	It	Key	100	0.75	0.17	0.28	0.34 s	-
AMUSE [31]	En	Full	100	-	-	0.26	-	[31]
WDAqua-core1	It	Full	100	0.62	0.15	0.24	0.43 s	-
AMUSE [31]	Es	Full	100	-	-	0.20	-	[31]
AMUSE [31]	De	Full	100	-	-	0.16	-	[31]
QALD-7
AMAL [46]	Fr	Full	47	0.72	0.72	0.72	-	[57]
gAnswer2 [34]	En	Full	50	0.40	0.46	0.41	3.33 s	[23]
WDAqua-core1	En	Full	50	0.39	0.40	0.37	0.55 s	[23]
WDAqua-core1	De	Full	50	0.23	0.26	0.22	0.53 s	[24]
WDAqua-core1	Fr	Full	50	0.13	0.16	0.14	0.45 s	[25]
WDAqua-core1	It	Full	50	0.09	0.05	0.03	0.44 s	[26]
QALD-8
WDAqua-core1	En	Full	47	0.39	0.40	0.39	1.72 s	[27,56]
gAnswer2 [34]	En	Full	47	0.39	0.39	0.39	1.92 s	[27,56]
QAKIS [8]	En	Full	47	0.06	0.05	0.06	15.41 s	[27,56]

5.2. Experiments

To show the performance of the approach on different scenarios, we benchmarked it using the following benchmarks.

Table 4

The table shows the results of WDAqua-core1 over the QALD-7 task 4 training dataset. We used Wikidata (dated 2016-11-28)

QA System	Language	Type	Number	Precision	Recall	F-measure	Runtime	Ref
QALD-7 task 4, training dataset
WDAqua-core1	En	Full	100	0.37	0.39	0.37	1.68 s	-
WDAqua-core1	En	Key	100	0.35	0.38	0.35	0.80 s	-
WDAqua-core1	Es	Key	100	0.31	0.32	0.31	0.45 s	-
Sorokin et al. [50]	En	Full	100	-	-	0.29	-	[50]
WDAqua-core1	De	Key	100	0.27	0.28	0.27	1.13 s	-
WDAqua-core1	Fr	Key	100	0.27	0.30	0.27	1.14 s	-
WDAqua-core1	Fr	Full	100	0.27	0.31	0.27	1.05 s	-
WDAqua-core1	Es	Full	100	0.24	0.26	0.24	0.65 s	-
WDAqua-core1	De	Full	100	0.18	0.20	0.18	0.82 s	-
WDAqua-core1	It	Full	100	0.19	0.20	0.18	1.00 s	-
WDAqua-core1	It	Key	100	0.17	0.18	0.16	0.44 s	-

5.2.1. Benchmarks

QALD: We evaluated our approach using the QALD benchmarks. These benchmarks allow us to see the performance on multiple languages and over both full-natural language questions and keyword questions. We executed the benchmarks for QALD-3 to QALD-6 locally while respecting the metrics of the original benchmarks. For QALD-7 and QALD-8 we relied on Gerbil for QA [58]. We did not use Gerbil for QA for all the benchmarks since Gerbil for QA uses only the newest version of DBpedia while QALD-3 to QALD-6 were based on different versions of DBpedia. Moreover, Gerbil for QA does not support benchmarking for keyword queries so that the benchmark results are not presented for them on QALD-7 and QALD-8. Finally note that the benchmarking metrics changed from QALD-7.

The results are given in Table 3 together with state-of-the-art systems. To find these, we used Google Scholar to select all publications about QA systems that cited one of the QALD challenge publications. Note that, in the past, QA systems were evaluated only on one or two of the QALD benchmarks. We provide, for the first time, an estimation of the differences between the benchmark series. Over English, we outperformed 90% of the proposed approaches. We achieve similar results as gAnswer2 [34], while we do not beat Xser [60], UTQA [59] and AMAL [46]. Note that Xser and UTQA required additional training data than the one provided in the benchmark, which required a significant cost in terms of manual effort. AMAL uses manual translations of the labels to address the lexical gap and can answer only questions with one triple pattern. Moreover, the robustness of these systems over keyword questions is probably not guaranteed. We cannot prove this claim because for these systems neither the source code nor a web-service is available.

Due to the manual effort required to do an error analysis for all benchmarks and the limited space, we restricted to the QALD-6 benchmark. The error sources over the 100 questions are the following:

40% (26 errors) are due to lexical gap (e.g. for “Who played Gus Fring in Breaking Bad?” the property dbo:portrayer is expected)

28% (18 errors) come from wrong ranking

12% (8 errors) are due to the missing support of superlatives and comparatives in our implementation (e.g. “Which Indian company has the most employees?”)

9% (4 errors) from the need of complex queries with unions or filters (e.g. the question “Give me a list of all critically endangered birds.” requires a filter on dbo:conservationStatus equal “CR”)

6% (4 errors) come from out of scope questions (i.e. question that should not be answered)

2% (1 error) from too ambiguous questions (e.g. “Who developed Slack?” is expected to refer to a “cloud-based team collaboration tool” while we interpret it as “linux distribution”).

One can see that keyword queries always perform worse as compared to full natural language queries. The reason is that the formulation of the keyword queries does not allow us to decide if the query is an ASK query or if a COUNT is needed (e.g. “Did Elvis Presley have children?” is formulated as “Elvis Presley, children”). This means that we automatically get these questions wrong.

To show the performance over Wikidata, we consider the QALD-7 task 4 training dataset. This originally provided only English questions. The QALD-7 task 4 training dataset reuses questions over DBpedia from previous challenges where translations in other languages were available. We moved these translations to the dataset. The results can be seen in Table 4. Except for English, keyword questions are easier than full natural language questions. The reason is the formulation of the questions. For keyword questions the lexical gap is smaller. For example, the keyword question corresponding to the question “Qui écrivit Harry Potter?” is “écrivain, Harry Potter”. Stemming does not suffice to map “écrivit” to “écrivain”, lemmatization would be needed. This problem is much smaller for English, where the effect described over DBpedia dominates. We can see that the best performing language is English, while the worst performing language is Italian. This is mostly related to the poorer number of lexicalizations for Italian. Note that the performance of the QA approach over Wikidata correlates with the number of lexicalizations for resources and properties for the different languages as described in [36]. This indicates that the quality of the data, in different languages, directly affects the performance of the QA system. Hence, we can derive that our results will probably improve while the data quality is increased. Finally we outperform the presented QA system over this benchmark.

Table 5

This table summarizes the QA systems evaluated over SQA2018. This benchmark is designed to measure the scalability of the approach, i.e. how many questions can be answered on increasing load

QA System	Language	Type	Total	Precision	Recall	F-measure	Power	Ref
SQA2018
WDAqua-core1	En	Full	1830	0.37	0.38	0.36	0.47	[43]
GQA [65]	En	Full	1830	0.02	0.02	0.02	0.03	[43]
LAMA [46]	En	Full	1830	0.01	0.02	0.01	0.02	[43]

Table 6

This table summarizes the QA systems evaluated over SimpleQuestions. Every system was evaluated over FB2M except the ones marked with (∗) which were evaluated over FB5M

QA System	Language	Type	Total	Accuracy	Runtime	Ref
SimpleQuestions
Lukovnikov et al.	En	Full	21687	0.712	-	[39]
Golub and He	En	Full	21687	0.709	-	[30]
Yin et al.	En	Full	21687	0.683	-	[63]
Bordes et al.	En	Full	21687	0.627	-	[5]
Dai et al.∗	En	Full	21687	0.626	-	[9]
WDAqua-core1 ∗	En	Full	21687	0.571	2.1 s	-

Table 7

This table summarizes the results of WDAqua-core1 over some newly appeared benchmarks

Benchmark	Lang	Type	Total	Precision	Recall	F-measure	Runtime
LC-QuAD	En	Full	5000	0.59	0.38	0.46	1.5 s
WDAquaCore0Questions	Mixed	Mixed	689	0.79	0.46	0.59	1.3 s

Table 8

Comparison on QALD-6 when querying only DBpedia and multiple KBs at the same time

Dataset	Language	Type	Total	Precision	Recall	F-measure	Runtime
DBpedia	En	Full	100	0.55	0.34	0.42	1.37s
All KBs supported	En	Full	100	0.49	0.39	0.43	11.94 s

SQA2018: SQA2018 is a benchmark to test how a QA system behaves when there is heavy load. The benchmark sends each minute and increasing number of queries. First 1 then 2, 4, 8 and so on. The performance is measured using as a metric the power which takes into consideration the precision, recall and number of queries that could be answered. The exact metric is reported in [43]. The result in Table 5 show that WDAqua-core1 outperforms by a large margin the other competing approaches. This shows that scalability is well addressed.

SimpleQuestions: SimpleQuestions contains 108,442 questions that can be solved using one triple pattern. We trained our system using the first 100 questions in the training set. The results of our system, together with the state-of-the-art systems are presented in Table 6. For this evaluation, we restricted the generated queries with one triple-pattern. The system performance is 14% below the state-of-the-art. Note that we achieve this result by considering only 100 of the 75,810 questions in the training set, and investing 1 hour of manual work for creating lexicalizations for properties manually. Concretely, instead of generating a training dataset with 80,000 questions, which can cost several thousands of euros, we invested 1 hour of manual work with the result of loosing (only) 14% in accuracy!

Note that the SimpleQuestions dataset is highly skewed towards certain properties (it contains 1629 properties, the 20 most frequent properties cover nearly 50% of the questions). Therefore, it is not clear how the other QA systems behave with respect to properties not appearing in the training dataset and with respect to keyword questions. Moreover, it is not clear how to port the existing approaches to new languages and it is not possible to adapt them to more difficult questions. These points are solved using our approach. Hence, we provided here, for the first time, a quantitative analysis of the impact of big training data corpora on the quality of a QA system.

LC-QuAD & WDAquaCore0Questions: Recently, a series of new benchmarks have been published. LC-QuAD [52] is a benchmark containing 5,000 English questions and it concentrates on complex questions. WDAquaCore0Questions [14] is a benchmark containing 689 questions over multiple languages and addressing mainly Wikidata, generated from the logs of a live running QA system. The questions are a mixture of real-world keyword and malformed questions. In Table 7, we present the first baselines for these benchmarks.

Multiple KBs: The only available benchmark that tackles multiple KBs was presented in QALD-4 task 2. The KBs are rather small and perfectly interlinked. This is not the case over the considered KBs. We therefore evaluated the ability to query multiple KBs differently. We run the questions of the QALD-6 benchmark, which was designed for DBpedia, both over DBpedia (only) and over DBpedia, Wikidata, MusicBrainz, DBLP and Freebase. Note that, while the original questions have a solution over DBpedia, a good answer could also be found over the other datasets. We therefore manually checked whether the answers that were found in other KBs are right (independently from which KB was chosen by the QA system to answer it). The results are presented in Table 8. WDAqua-core1 choose 53 times to answer a question over DBpedia, 39 over Wikidata and the other 8 times over a different KB. Note that we get better results when querying multiple KBs. Globally we get better recall and lower precision which is expected. While scalability is an issue, we are able to pick the right KB to find the answer!

Note: We did not tackle the WebQuestions benchmark for the following reasons. While it has been shown that WebQuestions can be addressed using non-reified versions of Freebase, this was not the original goal of the benchmark. More then 60% of the QA systems benchmarked over WebQuestions are tailored towards its reification model. There are two important points here. First, most KBs in the Semantic Web use binary statements. Secondly, in the Semantic Web community, many different reification models have been developed as described in [29].

5.2.2. Setting

All experiments were performed on a virtual machine with 4 cores of Intel Xeon E5-2667 v3 3.2GH, 16 GB of RAM and 500 GB of SSD disk. Note that the whole infrastructure was running on this machine, i.e. all indexes and the triple-stores needed to compute the answers (no external service was used). The original data dumps sum up to 336 GB. Note that across all benchmarks we can answer a question in less then 2 seconds except when all KBs are queried at the same time which shows that the algorithm should be parallelized for further optimization.

6. Provided services for multilingual and multi-KB QA

We have presented an algorithm that can be easily ported to new KBs and that can query multiple KBs at the same time. In the evaluation section, we have shown that our approach is competitive while offering the advantage of being multilingual and robust to keyword questions. Moreover, we have shown that we can achieve acceptable run-times on a modern laptop. In this section, we describe how we integrated the approach to an actual service and how we combine it to existing services so that it can be directly used by end-users.

First, we integrated WDAqua-core1 into Qanary [6,13], a framework to integrate QA components. This way WDAqua-core1 can be accessed via RESTful interfaces for example to benchmark it via Gerbil for QA [58]. It also allows to combine it with services that are already integrated into Qanary like a speech recognition component based on Kaldi19

¹⁹

http://kaldi-asr.org

and a language detection component based on [42]. Moreover, the integration into Qanary allows to reuse Trill [10], a reusable front-end for QA systems. A screenshot of Trill using in the back-end WDAqua-core1 can be found in Fig. 4.

Secondly, we reused and extended Trill to make it easily portable to new KBs. While Trill originally supported only DBpedia and Wikidata, now it also supports also MusicBrainz, DBLP and Freebase. We designed the extension so that it can be easily ported to new KBs. Enabling the support for a new KB is mainly reduced to writing an adapted SPARQL query for the new KB. Additionally, the extension allows to select multiple KBs at the same time.

Thirdly, we adapted some services that are used in Trill to be easily portable to new KBs. These include SPARQLToUser [11], a tool that generates a human readable version of a SPARQL query and SummaServer [16] a service for entity summarization. Finally we extended Trill to present aggregated information in from of collections of images and map aggregation depending on the available information [15]. All these tools now support the 5 mentioned KBs and the 5 mentioned languages.

A public online demo is available under: www.wdaqua.eu/qa

Moreover, there is an open API available that is described at http://wdaqua-frontend.univ-st-etienne.fr/faq

This is for example implemented by Gerbil for QA [58] and the DBpediaChat [1].20

²⁰

http://chat.dbpedia.org/

Fig. 4.

Screenshot of Trill, using in the back-end WDAqua-core1, for the question “Give me museums in Lyon”.

7. Conclusion and future work

In this paper, we introduced a novel concept for QA aimed at multilingual and KB-agnostic QA. Due to the described characteristics of our approach portability is ensured which is a significant advantage in comparison to previous approaches. We have shown the power of our approach in an extensive evaluation over multiple benchmarks. Hence, we clearly have shown our contributions w.r.t. qualitative (language, KBs) and quantitative improvements (outperforming many existing systems and querying multiple KBs) as well as the capability of our approach to scale for very large KBs like DBpedia.

We have applied our algorithm and adapted a set of existing services so that end-users can query, using multiple languages, multiple KBs at the same time, using a unified interface. Hence, we provided here a major step towards QA over the Semantic Web following our larger research agenda of providing QA over the LOD cloud.

In the future, we want to tackle the following points. First, we want to parallelize our approach, s.t. when querying multiple KBs acceptable response times will be achieved. Secondly, we want to query more and more KBs (hints to interesting KBs are welcome). Thirdly, from different lessons learned from querying multiple KBs, we want to give a set of recommendations for RDF datasets, s.t. they are fit for QA. And fourth, we want to extend our approach to also query reified data. Fifth, we would like to extend the approach to be able to answer questions including complex operators like aggregations and functions. We believe that our work can further boost the expansion of the Semantic Web since we presented a solution that easily allows to consume RDF data directly by end-users requiring low hardware investments.

Note.

There is a Patent Pending for the presented approach. It was submitted the 18 January 2018 at the EPO and has the number EP18305035.0.