A System for Answering Simple Questions in Multiple Languages — part 1/4

Cleaned-text code points [0, 15264). Bibliography omitted here; source.md includes it.

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
V olume 3: System Demonstrations, pages 524–537
July 10-12, 2023 ©2023 Association for Computational Linguistics
A System for Answering Simple Questions in Multiple Languages
Anton Razzhigaev1,2 *, Mikhail Salnikov1 *, Valentin Malykh3, Pavel Braslavski4,
and Alexander Panchenko1,2
1 Skolkovo Institute of Science and Technology 2 Artificial Intelligence Research Institute
3 ISP RAS Research Center for Trusted Artificial Intelligence 4 Ural Federal University
Abstract
Our research focuses on the most prevalent type
of queries—simple questions—exemplified by
questions like “What is the capital of France?”.
These questions reference an entity such as
“France”, which is directly connected (one hop)
to the answer entity “Paris” in the underlying
knowledge graph (KG). We propose a multi-
lingual Knowledge Graph Question Answer-
ing (KGQA) technique that orders potential
responses based on the distance between the
question’s text embeddings and the answer’s
graph embeddings. A system incorporating this
novel method is also described in our work.
Through comprehensive experimentation using
various English and multilingual datasets and
two KGs — Freebase and Wikidata — we illus-
trate the comparative advantage of the proposed
method across diverse KG embeddings and lan-
guages. This edge is apparent even against
robust baseline systems, including seq2seq QA
models, search-based solutions and intricate
rule-based pipelines. Interestingly, our research
underscores that even advanced AI systems like
ChatGPT encounter difficulties when tasked
with answering simple questions. This finding
emphasizes the relevance and effectiveness of
our approach, which consistently outperforms
such systems. We are making the source code
and trained models from our study publicly ac-
cessible to promote further advancements in
multilingual KGQA.
1 Introduction
A knowledge graph (KG) is a collection of subject–
predicate–object triples, for example⟨Paris, capi-
tal_of, France⟩. Large KGs are valuable resources
for many tasks, including question answering
(QA) (Ji et al., 2022). Knowledge graph question
answering (KGQA) is an active research area, as
well as a popular application.
Even though all major web search engines imple-
ment KGQA capabilities – KG results can be easily
*The first two authors contributed equally.
recognized in their ‘smart answers’ – there are few
operational KGQA research prototypes available
online. A rare example is QAnswer (Diefenbach
et al., 2020a), a rule-based KGQA system over
Wikidata. There are also only a few free KGQA
codebases available (Huang et al., 2019; Burtsev
et al., 2018; Chen et al., 2021).
In this work, we focus on simple questions such
as “What is the capital of France?”. There exists
an opinion that the task of answering such ques-
tions is nearly solved (Petrochuk and Zettlemoyer,
2018), but openly available systems are scarce and
do not support multiple languages. Besides, their
performance, as will be observed from our work,
is still far from perfect even for models based on
deep neural networks specifically pre-trained on
QA data. In our work, our aim is to address these
limitations of the prior art.
We developed a KGQA method M3M (multilin-
gual triple match) based on text-to-graph embed-
ding search. The key idea illustrated in Figure 1
is to combine a pre-trained multilingual language
model for question representation and pre-trained
graph embeddings that represent KG nodes and
edges as dense vectors. In the training phase, we
learn separate projections of the question text em-
beddings to the subject, predicate, and object of
the KG triple corresponding to the question-answer
pair. In the test phase, we first fetch a set of can-
didate KG triples based on the question’s word
n-grams and extract named entities to make the
process more computationally efficient. Then, we
rank candidate triples according to the sum of three
cosine similarities – between the embeddings of
the triple’s components and respective projections
of the question’s embeddings. Finally, the object
of the top-ranked triple is returned as an answer.
Our approach build upon Huang et al. (2019)
expanding the method beyond a single KG and
a single monolingual dataset. We experimented
with the de facto standard English KGQA dataset
524

Question, q
Candidate
Generator
 text2graph
projection (s)
Answer
KG
Triple
(s, r, o)
Pre-trained
Entity
Embeddings
 text2graph
projection (r)
 text2graph
projection (o)
eq
Text
Encoder
ys
yr
yo
es
er
eo
Index of
Entity
Names
+           +
(s, r, o)
KG
s
cosine:
Figure 1: Workflow of M3M Knowledge Graph Question Answering system for simple questions.
Figure 2: Graphical user interface of the KGQA system for answering one-hop questions.
SimpleQuestions, which is based on the now depre-
cated Freebase, to allow comparison with previous
art. Moreover, we conducted experiments with sev-
eral Wikidata-based datasets: SimpleQuestionsWd
(a Wikidata mapping from the original benchmark),
Russian/English RuBQ 2.0 dataset, as well as a re-
cent Mintaka dataset covering nine languages. Our
experiments demonstrate the applicability of the
proposed method in different KGs and languages.
Our online demo (see Figure 2) implements two
KGQA methods: (1) a T5 model fine-tuned on QA
data and (2) our approach dubbed M3M based on
embedding search. We believe that a combination
of an online demo, publicly available code, as well
as evaluation results on several datasets will con-
tribute to future developments in the field of mul-
tilingual KGQA. To summarize, our contributions
are the following:
• A novel multilingual approach to one-hop
KGQA, which compares favorably to strong
baselines, such as T5 QA system, and previ-
ous embedding-based methods on a battery of
benchmarks.
• Open implementation of an online system for
one-hop QA over Wikidata knowledge graph.
We also release pre-trained models and pro-
vide an API making seamless integration into
NLP applications possible.1
1Source code, link to the demo and video demonstration:
https://github.com/s-nlp/m3m
525

2 Related Work
There are two main directions in KGQA research:
semantic-parsing- and retrieval-based approaches.
Semantic parsing seeks to map a natural lan-
guage question to a meaning representation (e.g.
SPARQL query language) that can be directly ex-
ecuted against a KG. Gu et al. (2022) provide a
comprehensive survey of semantic parsing meth-
ods in KGQA. They also point out that KG schema
richness (diversity of entity and relation types), ir-
regularities and incompleteness of KG, as well as
its large size make the application of semantic pars-
ing methods to KGQA especially challenging.
The latter line of research follows information re-
trieval’s notion of proximity between the query and
sought entity. Early methods relied on the lexical
match of the question and the KG elements (en-
tity labels, descriptions, and relation names) (Ba-
log, 2018). In their pioneering work Bordes et al.
(2014) proposed to embed question words and KG
components in a shared space, with question rep-
resentations close to the answers’ ones. This ap-
proach was further developed in a series of stud-
ies (Dai et al., 2016; Lukovnikov et al., 2017;
Huang et al., 2019; Lukovnikov et al., 2019). No-
tably, more recent approaches leverage existing
knowledge graph embeddings that have been used
for a variety of KG tasks such as link prediction
and entity classification (Wang et al., 2017) and
pre-trained language models such as BERT appli-
cable to a multitude of NLP tasks (Devlin et al.,
2019). The embedding-based approach can be ex-
tended to complex KGQA (Saxena et al., 2020). A
comprehensive overview of KGQA methods can be
found in recent surveys (Chakraborty et al., 2021;
Lan et al., 2021) and a monograph (Roy and Anand,
2021).
Large language models (LLMs) accumulate not
only linguistic, but also factual knowledge, and
thus can be considered as alternatives to KGs.
Petroni et al. (2019) showed that BERT (a rather
small model by today’s standards) possessed some
relational knowledge without fine-tuning. Roberts
et al. (2020) experimented with the T5 model (Raf-
fel et al., 2020) as an open-domain QA system.
In particular, after fine-tuning on question-answer
pairs, at inference time, the model was given a ques-
tion and returned an answer without accessing any
external knowledge source. The results were quite
encouraging; the model with “salient span mask-
ing” (SSM) pre-training proposed by Guu et al.
(2020) performed best. Mallen et al. (2022) found
that LLMs memorise knowledge about common
entities but fall short in the case of less common
ones. They proposed to complement LLM-based
QA with zero-shot and few-shot prompting with
adaptive evidence retrieval. Tan et al. (2023) eval-
uate QA capabilities of FLAN-T5 (Chung et al.,
2022) and several GPT versions (GPT3, GPT3.5,
and chatGPT) in eight KGQA datasets. The results
show that LLMs are yet quite far beyond SotA;
chatGPT outperforms its contenders.
Multilingual KGQA has been relatively rarely
addressed in the literature. Proposed solutions in-
clude question translation (Perevalov et al., 2022a),
language-agnostic translation of syntactic parses
into meaning representations (Reddy et al., 2017;
Hakimov et al., 2017), and unsupervised induction
of bilingual lexicons for word-level translation of
training examples (Zhou et al., 2021). We hope
that the advent of larger multilingual datasets with
Wikidata annotations (Longpre et al., 2021; Sen
et al., 2022) will stimulate interest in this timely
topic.
3 M3M: Multilingual Triple Match QA
In this section, we describe the core of our contribu-
tion: a novel method for one-hop question answer-
ing over a knowledge graph. The proposed method
dubbed M3M for Multilingual Triple Match re-
lies on matching independently KGs subjects, ob-
jects, and predicates with the questions’ represen-
tation. Overall, the approach can be considered as
an improved version of the method by Huang et al.
(2019) with which we perform extensive compar-
isons in the remainder of the paper among other
baselines.
3.1 Question Answering Method
The architecture of the M3M model is illustrated
in Figure 1. M3M uses multilingual BERT (Devlin
et al., 2019) as question encoder. mBERT is a case-
sensitive “base” 12-layer Transformer model that
was trained with a masked language model objec-
tive on a collection of 104 Wikipedias. We encode
the question q with 768-dimensional mBERT-based
embeddings (Menc) averaging the last hidden states
of all tokens eq = Menc(q). To project these rep-
resentations to graph embeddings, we use three
2-layered perceptrons with ELU as an activation
function. These three perceptrons are used to pre-
dict the embeddings of the object, relation, and
526

subject of the answer triple. We train four models
simultaneously with AdamW optimiser with de-
fault parameters and lr = 1 e− 4 for 10 epochs.
We use MSE loss between the predicted and ground
truth graph embeddings.
At the test phase we first generate a set of can-
didate triples. We extract named entities from the
question using an off-the-shelf NER tool, as well
as all word uni-, bi-, and trigrams if there are no
named entities found. We use SpaCy2 as the base
entity detection model and lemmatizer. After ex-
tracting all candidate entity mentions from the ques-
tion, we search the collection of KG labels and
retain all entities containing at least one of the ex-
tracted question parts as a substring in their labels.
On average, the list contains∼300 candidates.
Then, we map the question embedding eq to the
KG embedding space with three models (Ms, Mr,
Mo), thus obtaining three projected embeddings
of the question: es, er, eo. We calculate the corre-
sponding cosine similarities between the projected
embeddings and KG embeddings of the candidate
triples. Then, we sum these three cosine similari-
ties, obtaining the answer score of each candidate
triple. As graph embeddings are normalised and
the model was trained with MSE loss, a sum of
cosine similarities is an appropriate choice for a
search metric. The QA system returns a ranked list
of triples with the corresponding scores. Finally,
the object of the highest-ranked triple is considered
the answer. The overall schema of the proposed
method for one-hop question answering is sum-
marised in Algorithm 1.
3.2 System Implementation
The demo web application illustrated in Figure 2
is implemented using FastAPI framework.3 It con-
tains a Web application backend that can be used
with any KGQA system over Wikidata. The ap-
plication’s frontend communicates with an asyn-
chronous Web API (see Swagger documentation
in Figure 7 in Appendix). The end user communi-
cates with each of the provided pipelines by send-
ing an HTTP POST request to /pipelines/model
with a JSON object containing the question in the
“text” field. Every pipeline responds to the end
user with a JSON object containing an “answers”
field with a sorted array of Wikidata identifiers. In
addition to the “answers”, the pipeline responses
2
3https://fastapi.tiangolo.com
can include additional information, for example,
/pipelines/m3m returns “scores” and “uncertainty”.
The entire KGQA system can be launched on any
platform that supports Docker.4 This demo can be
easily integrated into other applications using the
API endpoints provided.
We employ a conventional user interface for our
Web demo – the KGQA system returns a ranked list
of results along with links to Wikidata. In addition,
the system response provides a visualization of the
intermediate stages and sub-graphs corresponding
to the questions, which helps to better understand
the results. The core M3M method is based on a
deeply reworked implementation of QA method
firstly mentioned in (Chekalina et al., 2022) featur-
ing various optimizations related to (1) effective
candidate selection, and (2) retrieval of candidates
and triples.
Algorithm 1 M3M method for one-hop KGQA
Input: q – question,G – KG, E – KG embeddings,
Menc – text encoder, Ms, Mr, Mo – projection modules,
NER – subject candidates extractor
Output:⟨oa, ra, sa⟩ – triple with subject answer entity sa
eq = Menc(q)
A=[] ▷ Initialize answer-candidates
S=[] ▷ Initialize scores
C=[] ▷ Initialize entity-candidates
candidates = NER(q)
for entity inG do
if entity.name in candidates then C.append(entity)
for entity in C do
for relation in entity.relations do
s = entity.id; r = relation.id; o = entity[r]
A.append(⟨s, r, o⟩)
es = E[s]; er = E[r]; eo = E[o]
ys = Ms(eq); yr = Mr(eq); yo = Mo(eq)
score = cos(eo, yo) + cos(er, yr) + cos(es, ys)
S.append(score)
ind = argmax(S)
⟨sa, ra, oa⟩ = A[ind]
return⟨sa, ra, oa⟩
4 Experiments
In this section, we present the benchmarking of
the presented system on several one-hop KGQA
datasets for Freebase and Wikidata knowledge
graphs. The method is compared to strong base-
lines. Both the datasets and the baselines are de-
scribed below.
4.1 Data
Knowledge graphs. Freebase knowledge
graph (Bollacker et al., 2008), launched in 2007,
was a pioneering project that was developed to
a large extent by community members. In 2010
4https://www.docker.com
527