A System for Answering Simple Questions in Multiple Languages — part 2/4

Cleaned-text code points [15264, 28371). Bibliography omitted here; source.md includes it.

Freebase was acquired by Google, and in 2015
it was discontinued as an independent project.
The ‘frozen’ knowledge base dump was made
publicly available and was also incorporated
into Wikidata (Tanon et al., 2016). Nowadays
Wikidata (Vrandecic and Krötzsch, 2014) is the
largest and most up-to-date open KG. At the
time of writing, Wikidata contains more than
100 million unique entities, 7,500 relation types,
and 12.6 billion statements. 5 Wikidata items are
provided with labels and descriptions in multiple
languages. Many KGQA datasets have been
using Freebase as an underlying knowledge graph.
However, we can observe an increasing uptake of
Wikidata as a basis for KGQA research. Wikidata
underlies recent English (Cao et al., 2022; Dubey
et al., 2019), as well as several multilingual
datasets (Longpre et al., 2021; Rybin et al., 2021;
Sen et al., 2022; Perevalov et al., 2022b). In our
experiments, we use Freebase2M (FB2M) (Bordes
et al., 2015) and Wikidata8M (Korablinov and
Braslavski, 2020)6 KG snapshots containing 2M
and 8M entities, respectively.
KG embeddings. In our experiments, we use
TransE KG embeddings that represent both entities
and relations as vectors in the same space (Bordes
et al., 2013). For KG triples ⟨s, p, o⟩ correspond-
ing embeddings are expected to allow the following
‘translation’: s+p≈ o. We conducted experiments
on two TransE models: (1) TransE embeddings
with dim=250 based on Freebase2M snapshot pro-
vided by Huang et al. (2019); and (2) TransE em-
beddings with dim=200 trained on the full Wikidata
using Pytorch-BigGraph framework (Lerer et al.,
2019).
QA datasets. In contrast to open-domain ques-
tion answering and machine reading comprehen-
sion, there are rather few studies devoted to the
multilingual KGQA (Perevalov et al., 2022a).
This can partly be explained by the lack of
data: until recently, KGQA datasets were al-
most entirely English, and rare exceptions such
as QALD (Perevalov et al., 2022b) contained
only hundreds of questions, which is rather
scarce for training modern data-hungry models.
Larger-scale datasets with KG annotations such
as MKQA (Longpre et al., 2021) and Mintaka (Sen
et al., 2022) have appeared only recently.
5https://www.wikidata.org/wiki/
Property:P10209
6https://zenodo.org/record/3751761
To be able to compare our method with its pre-
decessors, we use SimpleQuestions (Bordes et al.,
2015). The dataset consists of 108,442 questions
formulated by English speakers using Freebase
triples as prompts. The question usually mentions
the triple’s subject and predicate, while the object
is the expected answer. Note that by design the
dataset contains many ambiguous questions having
multiple answers in the KG, while only one answer
is considered correct, see analysis in (Petrochuk
and Zettlemoyer, 2018; Wu et al., 2020). Further,
we make use of SimpleQuestionsWd, a mapping
of SimpleQuestions to Wikidata (Diefenbach et al.,
2017). The dataset contains 16,414/4,751 ques-
tions in train/test sets, respectively. We filtered the
dataset and retained only questions, whose triples
are present in Wikidata8M which resulted in 8,327
train and 2,438 test questions. RuBQ (Rybin et al.,
2021) is a KGQA data set that contains 2,910 Rus-
sian questions of different types along with their
English translations. Due to its limited size, the
official RuBQ split has only development and test
subsets. For our experiments, we kept only 1-hop
questions that constitute the majority of the dataset.
Mintaka (Sen et al., 2022) is a recently published
dataset that contains 20k questions of different
types translated into eight languages. For our ex-
periments, we took only generic questions, whose
entities are one hop away from the answers’ enti-
ties in Wikidata, which resulted in 1,236/181/340
train/dev/test questions in each language, respec-
tively.7 Table 1 summarizes the data we used for
training and evaluation.
Dataset KG Lang. Train Test
SimpleQuestions (SQ) FB en 75,910 21,687
SimpleQuestionsWd WD en 8,327 2,438
RuBQ WD ru, en 300 1,186
Mintaka-Simple WD mult 1,236 340
Table 1: Datasets in our experiments (FB – Freebase,
WD – Wikidata).
4.2 Baselines
We compare our system with three groups of
methodologically diverse state-of-the-art QA sys-
tems: based on rule-based approach, relying on
7These questions are not necessarily simple in terms of
SimpleQuestions dataset. E.g. both entities ( Heath Ledger,
The Dark Knight) in the question Who did Heath Ledger play
as in the movie The Dark Knight? are one hop away from
the answer entity Joker; both triples are essential to provide a
correct answer.
528

embedding search similar to our approach, and gen-
erative neural models, e.g. sequence-to-sequence.
QAnswer is a rule-based multilingual QA sys-
tem proposed by Diefenbach et al. (2020b). It re-
turns a ranked list of Wikidata identifiers as an-
swers and a corresponding SPARQL query. We use
QAnswer API in our experiments.8
KEQA: Knowledge Embedding based Question
Answering. There are no published results on
SimpleQuestions aligned with Wikidata KG which
is why we adopt the official implementation of
KEQA (Huang et al., 2019) – an open-source
embedding-based KGQA solution to SimpleQues-
tionsWd benchmark. It was initially trained and
evaluated on Freebase embeddings. To the best of
our knowledge, there are no open-sourced KGQA
models with better performance than KEQA. We
use this model as the main baseline on Simple-
QuestionsWd and original SimpleQuestions bench-
marks. To make a comparison with KEQA on
the SimpleQuestionsWd test set more fair, we re-
train it on PTBG-Wikidata embeddings. We use
the official implementation with provided hyperpa-
rameters and an internal validation mechanism. 9
As SimpleQuestionsWd is a subset of the original
SimpleQuestions, we evaluate Freebase-pretrained
KEQA on this dataset as well (taking into account
the corresponding entity mapping).
T5-based QA system. Question answering can
also be addressed as a seq2seq task. To pro-
vide a comparison with this type of approaches,
we conducted experiments with T5, an encoder-
decoder transformer-based model pre-trained on
a multi-task mixture of unsupervised and super-
vised tasks (Raffel et al., 2020). T5 works well
on a variety of tasks out-of-the-box by prepend-
ing a prefix to the input corresponding to each of
the tasks. To answer English questions, we used
T5 model fine-tuned on a large NaturalQuestions
dataset (Roberts et al., 2020). For other languages,
we fine-tuned mT5-xl model (Xue et al., 2021) on
Mintaka Simple.
In addition, we carried out experiments employ-
ing the Flan-T5-xl (Chung et al., 2022) model,
a recent development trained on a diverse mix-
ture of tasks. We evaluated this model in
8https://qanswer-frontend.
univ-st-etienne.fr
9https://github.com/xhuang31/KEQA_
WSDM19
two distinct setups using the Mintaka dataset:
firstly in a zero-shot setting, utilizing the prompt
“Question: question Answer:”, and secondly, by
separately fine-tuning the model on the training
data for each individual language.
GPT-3 has gained recognition for its impressive
performance in both few-shot and zero-shot con-
texts (Brown et al., 2020), excelling in a vast array
of benchmarks. A recent study (Chung et al., 2022)
evaluate different GPT versions on complex KG
questions. However, the experiments don’t include
datasets in our study. To address this oversight and
offer a comparative baseline for our system, we
subjected the GPT-3 model (davinci-003) to the
SimpleQuestionsWd and RuBQ 2.0 benchmarks.
Detailed information on the generation parameters
and prompts can be found in the Appendix.
ChatGPT stands as one of the leading systems
in the field of Natural Language Processing (NLP),
demonstrating capabilities for intricate reasoning
and extensive factual knowledge (OpenAI, 2023).
We evaluated this system (GPT-3.5-turbo-0301) us-
ing the RuBQ 2.0 and SimpleQuestionsWd bench-
marks. Specifics about the prompts and generation
parameters are available in the Appendix.
4.3 Experimental Setup
To compare our algorithm with baselines, we use
the Accuracy@1 metric i.e. correctness of the first
retrieved result. The answer of a QA system to
an answerable question is considered correct if its
object matches the answer in terms of Wikidata id
or just by a label string.
It is essential to acknowledge that sequence-to-
sequence (seq2seq) models yield a string instead
of a knowledge graph ID, which may pose a chal-
lenge during evaluation. To mitigate this, we apply
specific transformations to the responses produced
by seq2seq systems. These include converting the
text to lowercase and eliminating any leading and
trailing spaces. This transformation process is also
applied to label-aliases representing the actual an-
swers present in the RuBQ and Mintaka datasets.
Regarding the SimpleQuestionsWd dataset, we
procure aliases for the correct answers via the Wiki-
data API.10 We then determine the accuracy of the
seq2seq model’s prediction by checking for an ex-
act match between the predicted string and one of
the aliases.
10https://pypi.org/project/Wikidata
529

Model SQ SQ-WD RuBQ-ru RuBQ-en
QAnswer (Diefenbach et al., 2020a) – 33.31 30.80 32.30
T5-11b-ssm-nq, fine-tuned (Roberts et al., 2020) – 20.40 – 42.75
ChatGPT – GPT-3.5-turbo – 17.75 26.99 30.12
GPT-3 – davinci-003 – 28.51 18.10 34.20
KEQA (Huang et al., 2019) – TransE FB2M 75.40 40.48 – –
KEQA (Huang et al., 2019) – TransE PTBG – 48.89 – 33.80
M3M (Ours) – TransE FB2M 76.90± 0.30 – – –
M3M (Ours) – TransE PTBG – 53.50± 0.30 48.40± 0.30 49.50± 0.30
Table 2: Comparison of M3M system with KGQA baselines in terms of Accuracy@1 for monolingual one-hop QA
datasets. The best scores are highlighted. M3M scores are averages over models trained with five random seeds.
Model en es de ar fr pt it hi avg
mT5-xl, fine-tuned (Xue et al., 2021) 20.8 19.5 19.3 12.6 19.7 18.3 20.9 9.7 17.6
FlanT5-xl, fine-tuned (Chung et al., 2022) 35.3 22.0 23.3 0.2 25.0 24.0 25.5 0 19.41
FlanT5-xl, zero-shot (Chung et al., 2022) 14.7 6.5 7.6 0 0.7 0.9 0.9 0 3.19
M3M (Ours) – TransE PTBG 26.0 26.1 25.0 24.1 25.0 24.7 25.3 24.1 25.0
Table 3: Results on Mintaka-Simple dataset (one-hop questions) for models trained simultaneously on all languages.
It is noteworthy to mention that in the Mintaka-
Simple test set, about a half of the answers don’t
have labels in Hindi.
4.4 Results
Table 2 contains the results of our M3M model and
several baselines on two versions of the Simple-
Questions dataset and two versions of the RuBQ
dataset. Specifically, for the RuBQ dataset, we de-
tail the outcomes derived from testing both Russian
and English language queries.
Interestingly, ChatGPT, despite being recog-
nized as a more sophisticated system, exhibits
a weaker performance on factoid questions com-
pared to GPT-3. Upon conducting a concise man-
ual error analysis, we observed that ChatGPT fre-
quently dismissed queries with responses such as
“Answer is unknown”, or sought supplemental in-
formation. We suggest that this behavior may be
a consequence of the system’s alignment with hu-
man feedback, implemented to limit the model’s
tendency for generating ungrounded or ‘halluci-
nated’ responses. However, it is plausible that a
more refined prompt design could address this is-
sue and enhance the system’s performance on such
questions. Nonetheless, this exploration extends
beyond the scope of our current research and is
suggested as an avenue for further investigation.
Table 3 features the results obtained on the
Mintaka-Simple dataset, providing an opportunity
to evaluate the mT5, Flan-T5 and M3M mod-
els. This table highlights the multilingual capabili-
ties exhibited by both the generative and the KG-
retrieval approaches. An analysis of these results
reveals that our model’s performance is markedly
stable across languages, indicating a lesser depen-
dence on the language relative to the seq2seq ap-
proach. Our model manifests exceptional perfor-
mance on one-hop simple questions and achieves
a new state-of-the-art on the RuBQ 2.0 (Russian)
benchmark as well as on the English SimpleQues-
tionsWd dataset. These findings illustrate the su-
periority of KG-based models, outperforming both
GPT-3 and ChatGPT by a considerable margin.
5 Conclusion
In this study, we introduced M3M, a multilingual
model, along with an open implementation, de-
vised for one-hop knowledge base question answer-
ing. Our approach leverages the use of a multilin-
gual text encoder and pre-trained KG embeddings,
which are aligned using a triple projection method
of a question to subject/relation/object of KG triple
to facilitate efficient answer search in the embed-
ding space.
For simple questions, our system not only out-
performs previous strong alternatives, including
rule-based approaches, embeddings-based similar-
ity search, and pre-trained sequence-to-sequence
neural models, but also excels when compared to
advanced AI models like ChatGPT. These com-
parative results were drawn from a comprehensive
battery of one-hop QA datasets, including both
monolingual and multilingual data.
530