A System for Answering Simple Questions in Multiple Languages — part 3/4

Cleaned-text code points [28371, 43725). Bibliography omitted here; source.md includes it.

6 Limitations
While a large fraction of users’ information needs
may be fulfilled by answering simple questions,
the main limitation of the proposed system is that
in the current implementation, it can be applied
only to one-hop KG questions. As it may be not
obvious to figure out beforehand which question
is one-hop and which is multi-hop in a KG spe-
cial classifiers or uncertainty estimation techniques
should be ideally combined with the proposed sys-
tem to not let the system answer questions it is
not designed to answer in the first place. At the
same time, our preliminary experiments with train-
ing classifiers of question type based on Mintaka
data show promising results, suggesting that such
classifier effectively could be created and used in
real deployments in conjunction with the proposed
system.
In terms of computational efficiency, communi-
cation with a knowledge graph can be a bottleneck
if based on a public SPARQL endpoint with query
limits but could be substantially sped up using an in-
house SPARQL engine or using indexing of triples
with appropriate data structures. However, in the
latter case, a mechanism for updating such struc-
tures is requires to keep system answers up to date.
7 Ethical Statement
QA systems built on top of large pretrained neu-
ral models, such as those described in this paper,
may transitively reflect biases available in the train-
ing data potentially generating stereotyped answers
to questions. It is therefore recommended in pro-
duction (as compared to research settings) to use
a special version of debiased pre-trained neural
models and/or deploy a special layer of debiasing
systems around the proposed methodology.
Acknowledgements
Pavel Braslavski’s work was supported in part by
the Ministry of Science and Higher Education of
the Russian Federation (project 075-02-2023-935).
The work of Valentin Malykh was supported by a
grant for research centers in the field of artificial
intelligence, provided by the Analytical Center for
the Government of the Russian Federation in accor-
dance with the subsidy agreement (agreement iden-
tifier 000000D730321P5Q0002) and the agreement
with the Ivannikov Institute for System Program-
ming of the Russian Academy of Sciences dated
November 2, 2021 No. 70-2021-00142.
References
Krisztian Balog. 2018. Entity-Oriented Search ,
volume 39 of The Information Retrieval Series .
Springer.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim
Sturge, and Jamie Taylor. 2008. Freebase: A col-
laboratively created graph database for structuring
human knowledge. In Proceedings of the 2008 ACM
SIGMOD International Conference on Management
of Data, SIGMOD ’08, page 1247–1250, New York,
NY , USA. Association for Computing Machinery.
Antoine Bordes, Sumit Chopra, and Jason Weston. 2014.
Question answering with subgraph embeddings. In
Proceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing, EMNLP
2014, October 25-29, 2014, Doha, Qatar, A meeting
of SIGDAT, a Special Interest Group of the ACL ,
pages 615–620. ACL.
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and
Jason Weston. 2015. Large-scale simple ques-
tion answering with memory networks. CoRR,
abs/1506.02075.
Antoine Bordes, Nicolas Usunier, Alberto García-
Durán, Jason Weston, and Oksana Yakhnenko.
2013. Translating embeddings for modeling multi-
relational data. In Advances in Neural Information
Processing Systems 26: 27th Annual Conference on
Neural Information Processing Systems 2013. Pro-
ceedings of a meeting held December 5-8, 2013, Lake
Tahoe, Nevada, United States, pages 2787–2795.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-V oss,
Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens
Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma-
teusz Litwin, Scott Gray, Benjamin Chess, Jack
Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei. 2020.
Language models are few-shot learners. In Ad-
vances in Neural Information Processing Systems ,
volume 33, pages 1877–1901. Curran Associates,
Inc.
Mikhail S. Burtsev, Alexander V . Seliverstov, Rafael
Airapetyan, Mikhail Y . Arkhipov, Dilyara Baymurz-
ina, Nickolay Bushkov, Olga Gureenkova, Taras
Khakhulin, Yuri Kuratov, Denis Kuznetsov, Alexey
Litinsky, Varvara Logacheva, Alexey Lymar, Valentin
Malykh, Maxim Petrov, Vadim Polulyakh, Leonid Pu-
gachev, Alexey Sorokin, Maria Vikhreva, and Marat
Zaynutdinov. 2018. Deeppavlov: Open-source li-
brary for dialogue systems. In Proceedings of ACL
2018, Melbourne, Australia, July 15-20, 2018, Sys-
tem Demonstrations, pages 122–127. Association for
Computational Linguistics.
531

Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie,
Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Han-
wang Zhang. 2022. KQA pro: A dataset with explicit
compositional programs for complex question an-
swering over knowledge base. In Proceedings of the
60th Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), pages
6101–6119, Dublin, Ireland. Association for Compu-
tational Linguistics.
Nilesh Chakraborty, Denis Lukovnikov, Gaurav Ma-
heshwari, Priyansh Trivedi, Jens Lehmann, and Asja
Fischer. 2021. Introduction to neural network-based
question answering over knowledge graphs. WIREs
Data Mining Knowl. Discov., 11(3).
Viktoriia Chekalina, Anton Razzhigaev, Albert Sayapin,
Evgeny Frolov, and Alexander Panchenko. 2022.
MEKER: Memory efficient knowledge embedding
representation for link prediction and question an-
swering. In Proceedings of the 60th Annual Meet-
ing of the Association for Computational Linguistics:
Student Research Workshop, pages 355–365, Dublin,
Ireland. Association for Computational Linguistics.
Shuang Chen, Qian Liu, Zhiwei Yu, Chin-Yew Lin,
Jian-Guang Lou, and Feng Jiang. 2021. ReTraCk:
A flexible and efficient framework for knowledge
base question answering. In Proceedings of the Joint
Conference of the 59th Annual Meeting of the Asso-
ciation for Computational Linguistics and the 11th
International Joint Conference on Natural Language
Processing, ACL 2021 - System Demonstrations, On-
line, August 1-6, 2021, pages 325–336. Association
for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret
Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,
Mostafa Dehghani, Siddhartha Brahma, Albert Web-
son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz-
gun, Xinyun Chen, Aakanksha Chowdhery, Sharan
Narang, Gaurav Mishra, Adams Yu, Vincent Y . Zhao,
Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav
Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam
Roberts, Denny Zhou, Quoc V . Le, and Jason Wei.
2022. Scaling instruction-finetuned language models.
CoRR, abs/2210.11416.
Zihang Dai, Lei Li, and Wei Xu. 2016. CFO: Condi-
tional focused neural question answering with large-
scale knowledge bases. In Proceedings of the 54th
Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 800–810,
Berlin, Germany. Association for Computational Lin-
guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: pre-training of
deep bidirectional transformers for language under-
standing. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Tech-
nologies, NAACL-HLT 2019, Minneapolis, MN, USA,
June 2-7, 2019, Volume 1 (Long and Short Papers),
pages 4171–4186. Association for Computational
Linguistics.
Dennis Diefenbach, Andreas Both, Kamal Singh, and
Pierre Maret. 2020a. Towards a question answer-
ing system over the semantic web. Semantic Web,
11(3):421–439.
Dennis Diefenbach, José M. Giménez-García, Andreas
Both, Kamal Singh, and Pierre Maret. 2020b. Qan-
swer KG: designing a portable question answering
system over RDF data. In The Semantic Web - 17th
International Conference, ESWC 2020, Heraklion,
Crete, Greece, May 31-June 4, 2020, Proceedings ,
volume 12123 of Lecture Notes in Computer Science,
pages 429–445. Springer.
Dennis Diefenbach, Thomas Pellissier Tanon, Ka-
mal Deep Singh, and Pierre Maret. 2017. Question
answering benchmarks for wikidata. In Proceedings
of the ISWC 2017 Posters & Demonstrations and
Industry Tracks co-located with 16th International
Semantic Web Conference (ISWC 2017), Vienna, Aus-
tria, October 23rd - to - 25th, 2017, volume 1963 of
CEUR Workshop Proceedings. CEUR-WS.org.
Mohnish Dubey, Debayan Banerjee, Abdelrahman Ab-
delkawi, and Jens Lehmann. 2019. Lc-quad 2.0: A
large dataset for complex question answering over
wikidata and dbpedia. In The Semantic Web - ISWC
2019 - 18th International Semantic Web Conference,
Auckland, New Zealand, October 26-30, 2019, Pro-
ceedings, Part II, volume 11779 of Lecture Notes in
Computer Science, pages 69–78. Springer.
Yu Gu, Vardaan Pahuja, Gong Cheng, and Yu Su. 2022.
Knowledge base question answering: A semantic
parsing perspective. CoRR, abs/2209.04994.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat,
and Ming-Wei Chang. 2020. Retrieval augmented
language model pre-training. In Proceedings of the
37th International Conference on Machine Learning,
ICML 2020, 13-18 July 2020, Virtual Event, volume
119 of Proceedings of Machine Learning Research,
pages 3929–3938. PMLR.
Sherzod Hakimov, Soufian Jebbara, and Philipp Cimi-
ano. 2017. AMUSE: multilingual semantic parsing
for question answering over linked data. In The Se-
mantic Web - ISWC 2017 - 16th International Se-
mantic Web Conference, Vienna, Austria, October
21-25, 2017, Proceedings, Part I, volume 10587 of
Lecture Notes in Computer Science, pages 329–346.
Springer.
Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping
Li. 2019. Knowledge graph embedding based ques-
tion answering. In Proceedings of the Twelfth ACM
International Conference on Web Search and Data
Mining, WSDM 2019, Melbourne, VIC, Australia,
February 11-15, 2019, pages 105–113. ACM.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Mart-
tinen, and Philip S. Yu. 2022. A survey on knowl-
532

edge graphs: Representation, acquisition, and appli-
cations. IEEE Trans. Neural Networks Learn. Syst.,
33(2):494–514.
Vladislav Korablinov and Pavel Braslavski. 2020.
RuBQ: A Russian dataset for question answering
over Wikidata. In The Semantic Web - ISWC 2020 -
19th International Semantic Web Conference, Athens,
Greece, November 2-6, 2020, Proceedings, Part II,
volume 12507 of Lecture Notes in Computer Science,
pages 97–110. Springer.
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang,
Wayne Xin Zhao, and Ji-Rong Wen. 2021. A sur-
vey on complex knowledge base question answering:
Methods, challenges and solutions. In Proceedings
of the Thirtieth International Joint Conference on
Artificial Intelligence, IJCAI 2021, Virtual Event /
Montreal, Canada, 19-27 August 2021, pages 4483–
4491. ijcai.org.
Adam Lerer, Ledell Wu, Jiajun Shen, Timothée
Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex
Peysakhovich. 2019. Pytorch-BigGraph: A large
scale graph embedding system. In Proceedings of
Machine Learning and Systems 2019, MLSys 2019,
Stanford, CA, USA, March 31 - April 2, 2019 . ml-
sys.org.
Shayne Longpre, Yi Lu, and Joachim Daiber. 2021.
MKQA: A linguistically diverse benchmark for mul-
tilingual open domain question answering. Trans.
Assoc. Comput. Linguistics, 9:1389–1406.
Denis Lukovnikov, Asja Fischer, and Jens Lehmann.
2019. Pretrained transformers for simple question
answering over knowledge graphs. In The Semantic
Web - ISWC 2019 - 18th International Semantic Web
Conference, Auckland, New Zealand, October 26-30,
2019, Proceedings, Part I, pages 470–486.
Denis Lukovnikov, Asja Fischer, Jens Lehmann, and
Sören Auer. 2017. Neural network-based question
answering over knowledge graphs on word and char-
acter level. In Proceedings of the 26th International
Conference on World Wide Web, WWW 2017, Perth,
Australia, April 3-7, 2017, pages 1211–1220. ACM.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das,
Hannaneh Hajishirzi, and Daniel Khashabi. 2022.
When not to trust language models: Investigating
effectiveness and limitations of parametric and non-
parametric memories. CoRR, abs/2212.10511.
OpenAI. 2023. GPT-4 technical report. CoRR,
abs/2303.08774.
Aleksandr Perevalov, Andreas Both, Dennis Diefen-
bach, and Axel-Cyrille Ngonga Ngomo. 2022a. Can
machine translation be a reasonable alternative for
multilingual question answering systems over knowl-
edge graphs? In WWW ’22: The ACM Web Confer-
ence 2022, Virtual Event, Lyon, France, April 25 - 29,
2022, pages 977–986. ACM.
Aleksandr Perevalov, Dennis Diefenbach, Ricardo Us-
beck, and Andreas Both. 2022b. Qald-9-plus: A mul-
tilingual dataset for question answering over dbpedia
and wikidata translated by native speakers. In 16th
IEEE International Conference on Semantic Comput-
ing, ICSC 2022, Laguna Hills, CA, USA, January
26-28, 2022, pages 229–234. IEEE.
Michael Petrochuk and Luke Zettlemoyer. 2018. Sim-
pleQuestions nearly solved: A new upperbound and
baseline approach. In Proceedings of the 2018 Con-
ference on Empirical Methods in Natural Language
Processing, pages 554–558, Brussels, Belgium. As-
sociation for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel,
Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and
Alexander Miller. 2019. Language models as knowl-
edge bases? In Proceedings of the 2019 Confer-
ence on Empirical Methods in Natural Language Pro-
cessing and the 9th International Joint Conference
on Natural Language Processing (EMNLP-IJCNLP),
pages 2463–2473, Hong Kong, China. Association
for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J. Liu. 2020. Exploring the limits
of transfer learning with a unified text-to-text trans-
former. J. Mach. Learn. Res., 21:140:1–140:67.
Siva Reddy, Oscar Täckström, Slav Petrov, Mark Steed-
man, and Mirella Lapata. 2017. Universal semantic
parsing. In Proceedings of the 2017 Conference on
Empirical Methods in Natural Language Processing,
pages 89–101, Copenhagen, Denmark. Association
for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.
How much knowledge can you pack into the param-
eters of a language model? In Proceedings of the
2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 5418–5426,
Online. Association for Computational Linguistics.
Rishiraj Saha Roy and Avishek Anand. 2021. Question
Answering for the Curated Web: Tasks and Meth-
ods in QA over Knowledge Bases and Text Collec-
tions. Synthesis Lectures on Information Concepts,
Retrieval, and Services. Morgan & Claypool Publish-
ers.
Ivan Rybin, Vladislav Korablinov, Pavel Efimov, and
Pavel Braslavski. 2021. RuBQ 2.0: An innovated
russian question answering dataset. In The Semantic
Web - 18th International Conference, ESWC 2021 ,
pages 532–547.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar.
2020. Improving multi-hop question answering over
knowledge graphs using knowledge base embeddings.
In Proceedings of the 58th Annual Meeting of the As-
sociation for Computational Linguistics, pages 4498–
4507, Online. Association for Computational Lin-
guistics.
533