{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T04:30:11Z","timestamp":1778214611785,"version":"3.51.4"},"reference-count":46,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T00:00:00Z","timestamp":1701129600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SW"],"published-print":{"date-parts":[[2023,11,28]]},"abstract":"<jats:p>Knowledge Graph Question Answering (KGQA) has gained attention from both industry and academia over the past decade. Researchers proposed a substantial amount of benchmarking datasets with different properties, pushing the development in this field forward. Many of these benchmarks depend on Freebase, DBpedia, or Wikidata. However, KGQA benchmarks that depend on Freebase and DBpedia are gradually less studied and used, because Freebase is defunct and DBpedia lacks the structural validity of Wikidata. Therefore, research is gravitating toward Wikidata-based benchmarks. That is, new KGQA benchmarks are created on the basis of Wikidata and existing ones are migrated. We present a new, multilingual, complex KGQA benchmarking dataset as the 10th part of the Question Answering over Linked Data (QALD) benchmark series. This corpus formerly depended on DBpedia. Since QALD serves as a base for many machine-generated benchmarks, we increased the size and adjusted the benchmark to Wikidata and its ranking mechanism of properties. These measures foster novel KGQA developments by more demanding benchmarks. Creating a benchmark from scratch or migrating it from DBpedia to Wikidata is non-trivial due to the complexity of the Wikidata knowledge graph, mapping issues between different languages, and the ranking mechanism of properties using qualifiers. We present our creation strategy and the challenges we faced that will assist other researchers in their future work. Our case study, in the form of a conference challenge, is accompanied by an in-depth analysis of the created benchmark.<\/jats:p>","DOI":"10.3233\/sw-233471","type":"journal-article","created":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T11:46:57Z","timestamp":1701172017000},"page":"1-15","source":"Crossref","is-referenced-by-count":18,"title":["QALD-10 \u2013 The 10th challenge on question answering over linked data"],"prefix":"10.1177","author":[{"given":"Ricardo","family":"Usbeck","sequence":"first","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"},{"name":"Hamburger Informatik Technologie-Center HITEC e.V., Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xi","family":"Yan","sequence":"additional","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"},{"name":"Hamburger Informatik Technologie-Center HITEC e.V., Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aleksandr","family":"Perevalov","sequence":"additional","affiliation":[{"name":"Anhalt University of Applied Sciences, Germany"},{"name":"Leipzig University of Applied Sciences, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Longquan","family":"Jiang","sequence":"additional","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"},{"name":"Hamburger Informatik Technologie-Center HITEC e.V., Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julius","family":"Schulz","sequence":"additional","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"},{"name":"Hamburger Informatik Technologie-Center HITEC e.V., Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Angelie","family":"Kraft","sequence":"additional","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cedric","family":"M\u00f6ller","sequence":"additional","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junbo","family":"Huang","sequence":"additional","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"},{"name":"Hamburger Informatik Technologie-Center HITEC e.V., Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jan","family":"Reineke","sequence":"additional","affiliation":[{"name":"Semantic Systems Group, Universit\u00e4t Hamburg, Germany"},{"name":"Hamburger Informatik Technologie-Center HITEC e.V., Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Axel-Cyrille","family":"Ngonga Ngomo","sequence":"additional","affiliation":[{"name":"DICE Group, Universit\u00e4t Paderborn, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muhammad","family":"Saleem","sequence":"additional","affiliation":[{"name":"DICE Group, Universit\u00e4t Paderborn, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Both","sequence":"additional","affiliation":[{"name":"Leipzig University of Applied Sciences, Germany"},{"name":"DATEV eG, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/SW-233471_ref1","doi-asserted-by":"crossref","unstructured":"D.\u00a0Banerjee, P.A.\u00a0Nair, J.N.\u00a0Kaur, R.\u00a0Usbeck and C.\u00a0Biemann, Modern baselines for SPARQL semantic parsing, in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.","DOI":"10.1145\/3477495.3531841"},{"key":"10.3233\/SW-233471_ref2","unstructured":"N.\u00a0Baramiia, A.\u00a0Rogulina, S.\u00a0Petrakov, V.\u00a0Kornilov and A.\u00a0Razzhigaev, Ranking approach to monolingual question answering over knowledge graphs, in: Proceedings of the 7th Natural Language Interfaces for the Web of Data, (NLIWoD) Co-Located with the 19th European Semantic Web Conference (ESWC 2022), 2022."},{"key":"10.3233\/SW-233471_ref3","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376746"},{"key":"10.3233\/SW-233471_ref4","doi-asserted-by":"crossref","unstructured":"A.\u00a0Both, D.\u00a0Diefenbach, K.\u00a0Singh, S.\u00a0Shekarpour, D.\u00a0Cherix and C.\u00a0Lange, Qanary \u2013 A methodology for vocabulary-driven open question answering systems, in: European Semantic Web Conference, Springer, 2016, pp.\u00a0625\u2013641.","DOI":"10.1007\/978-3-319-34129-3_38"},{"key":"10.3233\/SW-233471_ref5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-4021"},{"key":"10.3233\/SW-233471_ref6","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40802-1_30"},{"key":"10.3233\/SW-233471_ref7","unstructured":"Council of Europe, Common European Framework of Reference for Languages: Learning, Teaching, Assessment, Council of Europe, 2001."},{"key":"10.3233\/SW-233471_ref8","doi-asserted-by":"publisher","first-page":"937","DOI":"10.1162\/tacl_a_00499","article-title":"Compositional generalization in multilingual semantic parsing over Wikidata","volume":"10","author":"Cui","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"10.3233\/SW-233471_ref9","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3191541"},{"key":"10.3233\/SW-233471_ref10","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-69146-6_8"},{"key":"10.3233\/SW-233471_ref11","unstructured":"D.\u00a0Diefenbach, T.P.\u00a0Tanon, K.D.\u00a0Singh and P.\u00a0Maret, Question answering benchmarks for Wikidata, in: Proceedings of the ISWC 2017 Posters & Demonstrations and Industry Tracks Co-Located with 16th International Semantic Web Conference (ISWC 2017), Vienna, Austria, October 23rd \u2013 to \u2013 25th, 2017, N.\u00a0Nikitina, D.\u00a0Song, A.\u00a0Fokoue and P.\u00a0Haase, eds, CEUR Workshop Proceedings, Vol.\u00a01963 CEUR-WS.org, 2017. http:\/\/ceur-ws.org\/Vol-1963\/paper555.pdf."},{"key":"10.3233\/SW-233471_ref12","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30796-7_5"},{"key":"10.3233\/SW-233471_ref13","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.websem.2013.01.002","article-title":"Binary RDF representation for publication and exchange (HDT)","volume":"19","author":"Fern\u00e1ndez","year":"2013","journal-title":"J. Web Semant."},{"key":"10.3233\/SW-233471_ref14","doi-asserted-by":"crossref","unstructured":"Y.\u00a0Gu, S.\u00a0Kase, M.\u00a0Vanni, B.\u00a0Sadler, P.\u00a0Liang, X.\u00a0Yan and Y.\u00a0Su, Beyond IID: Three levels of generalization for question answering on knowledge bases, in: Proceedings of the Web Conference 2021, ACM, 2021, pp.\u00a03477\u20133488.","DOI":"10.1145\/3442381.3449992"},{"key":"10.3233\/SW-233471_ref15","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531751"},{"key":"10.3233\/SW-233471_ref16","unstructured":"D.\u00a0Keysers, N.\u00a0Sch\u00e4rli, N.\u00a0Scales, H.\u00a0Buisman, D.\u00a0Furrer, S.\u00a0Kashubin, N.\u00a0Momchev, D.\u00a0Sinopalnikov, L.\u00a0Stafiniak, T.\u00a0Tihon, D.\u00a0Tsarkov, X.\u00a0Wang, M.\u00a0van Zee and O.\u00a0Bousquet, Measuring compositional generalization: A comprehensive method on realistic data, in: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020, OpenReview.net, 2020. https:\/\/openreview.net\/forum?id=SygcCnNKwr."},{"key":"10.3233\/SW-233471_ref17","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-62466-8_7"},{"key":"10.3233\/SW-233471_ref18","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/611"},{"key":"10.3233\/SW-233471_ref19","doi-asserted-by":"crossref","unstructured":"J.\u00a0Lehmann, R.\u00a0Isele, M.\u00a0Jakob, A.\u00a0Jentzsch, D.\u00a0Kontokostas, P.N.\u00a0Mendes, S.\u00a0Hellmann, M.\u00a0Morsey, P.\u00a0van Kleef, S.\u00a0Auer and C.\u00a0Bizer, DBpedia \u2013 A large-scale, multilingual knowledge base extracted from Wikipedia, Semantic Web Journal (2014).","DOI":"10.3233\/SW-140134"},{"key":"10.3233\/SW-233471_ref20","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2013.05.006"},{"key":"10.3233\/SW-233471_ref21","doi-asserted-by":"crossref","unstructured":"C.D.\u00a0Manning, Introduction to Information Retrieval, Syngress Publishing, 2008.","DOI":"10.1017\/CBO9780511809071"},{"key":"10.3233\/SW-233471_ref22","doi-asserted-by":"publisher","DOI":"10.1109\/ICSC52841.2022.00045"},{"key":"10.3233\/SW-233471_ref23","doi-asserted-by":"publisher","DOI":"10.1109\/ICSC52841.2022.00048"},{"key":"10.3233\/SW-233471_ref24","unstructured":"A.\u00a0Perevalov, X.\u00a0Yan, L.\u00a0Kovriguina, L.\u00a0Jiang, A.\u00a0Both and R.\u00a0Usbeck, Knowledge graph question answering leaderboard: A community resource to prevent a replication crisis, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp.\u00a02998\u20133007. https:\/\/aclanthology.org\/2022.lrec-1.321."},{"key":"10.3233\/SW-233471_ref25","first-page":"140:1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"10.3233\/SW-233471_ref26","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-77385-4_32"},{"key":"10.3233\/SW-233471_ref27","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-25010-6_15"},{"key":"10.3233\/SW-233471_ref28","unstructured":"M.\u00a0Saleem, S.N.\u00a0Dastjerdi, R.\u00a0Usbeck and A.-C.N.\u00a0Ngomo, Question answering over linked data: What is difficult to answer? What affects the F scores? in: BLINK\/NLIWoD3@ ISWC, 2017."},{"key":"10.3233\/SW-233471_ref29","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313556"},{"key":"10.3233\/SW-233471_ref30","unstructured":"M.A.B.\u00a0Santana, F.\u00a0Ricca, B.\u00a0Cuteri and V.\u00a0Barbara, SPARQL-QA enters the QALD challenge, in: Proceedings of the 7th Natural Language Interfaces for the Web of Data, (NLIWoD) Co-Located with the 19th European Semantic Web Conference (ESWC 2022), 2022."},{"key":"10.3233\/SW-233471_ref31","unstructured":"K.\u00a0Shivashankar, K.\u00a0Benmaarouf and N.\u00a0Steinmetz, From graph to graph: AMR to SPARQL, in: Proceedings of the 7th Natural Language Interfaces for the Web of Data, (NLIWoD) Co-Located with the 19th European Semantic Web Conference (ESWC 2022), 2022."},{"issue":"2","key":"10.3233\/SW-233471_ref32","doi-asserted-by":"publisher","first-page":"215","DOI":"10.3233\/SW-210440","article-title":"MQALD: Evaluating the impact of modifiers in question answering over knowledge graphs","volume":"13","author":"Siciliani","year":"2022","journal-title":"Semantic Web"},{"key":"10.3233\/SW-233471_ref33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3233\/SW-223015","article-title":"Ngomo, LSQ 2.0: A linked dataset of SPARQL query logs","author":"Stadler","year":"2022","journal-title":"Semantic Web"},{"key":"10.3233\/SW-233471_ref34","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2874809"},{"key":"10.3233\/SW-233471_ref35","unstructured":"C.\u00a0Unger, P.\u00a0Cimiano, V.\u00a0L\u00f3pez, E.\u00a0Motta, P.\u00a0Buitelaar and R.\u00a0Cyganiak\u00a0(eds), Proceedings of the Workshop on Interacting with Linked Data, Heraklion, Greece, May 28, 2012, CEUR Workshop Proceedings, Vol.\u00a0913, CEUR-WS.org, 2012. http:\/\/ceur-ws.org\/Vol-913."},{"key":"10.3233\/SW-233471_ref36","unstructured":"C.\u00a0Unger, C.\u00a0Forascu, V.\u00a0Lopez, A.N.\u00a0Ngomo, E.\u00a0Cabrio, P.\u00a0Cimiano and S.\u00a0Walter, Question answering over linked data (QALD-4), in: CLEF, 2014, pp.\u00a01172\u20131180."},{"key":"10.3233\/SW-233471_ref37","unstructured":"C.\u00a0Unger, C.\u00a0Forascu, V.\u00a0Lopez, A.N.\u00a0Ngomo, E.\u00a0Cabrio, P.\u00a0Cimiano and S.\u00a0Walter, Question answering over linked data (QALD-5), in: CLEF, 2015. http:\/\/ceur-ws.org\/Vol-1391\/173-CR.pdf."},{"key":"10.3233\/SW-233471_ref38","doi-asserted-by":"crossref","unstructured":"C.\u00a0Unger, A.-C.N.\u00a0Ngomo and E.\u00a0Cabrio, 6th open challenge on question answering over linked data (QALD-6), in: Semantic Web Challenges: Third SemWebEval Challenge at ESWC 2016, Heraklion, Crete, Greece, May 29\u2013June 2, 2016, Revised Selected Papers, Springer International Publishing, Cham, 2016, pp.\u00a0171\u2013177. ISBN 978-3-319-46565-4.","DOI":"10.1007\/978-3-319-46565-4_13"},{"key":"10.3233\/SW-233471_ref39","unstructured":"R.\u00a0Usbeck, R.H.\u00a0Gusmita, A.-C.N.\u00a0Ngomo and M.\u00a0Saleem, 9th challenge on question answering over linked data, (QALD-9) (invited paper), in: Semdeep\/NLIWoD@ISWC, 2018."},{"key":"10.3233\/SW-233471_ref40","unstructured":"R.\u00a0Usbeck, A.-C.N.\u00a0Ngomo, F.\u00a0Conrads, M.\u00a0R\u00f6der and G.\u00a0Napolitano, 8th challenge on question answering over linked data (QALD-8) (invited paper), in: Semdeep\/NLIWoD@ISWC, 2018."},{"key":"10.3233\/SW-233471_ref41","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-69146-6_6"},{"key":"10.3233\/SW-233471_ref42","doi-asserted-by":"crossref","unstructured":"R.\u00a0Usbeck, M.\u00a0R\u00f6der, M.\u00a0Hoffmann, F.\u00a0Conrad, J.\u00a0Huthmann, A.-C.\u00a0Ngonga-Ngomo, C.\u00a0Demmler and C.\u00a0Unger, Benchmarking question answering systems, Semantic Web Journal (2018). http:\/\/www.semantic-web-journal.net\/system\/files\/swj1578.pdf.","DOI":"10.3233\/SW-180312"},{"key":"10.3233\/SW-233471_ref43","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7496690"},{"issue":"10","key":"10.3233\/SW-233471_ref44","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1145\/2629489","article-title":"Wikidata: A free collaborative knowledgebase","volume":"57","author":"Vrande\u010di\u0107","year":"2014","journal-title":"Commun. ACM"},{"key":"10.3233\/SW-233471_ref46","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.18"},{"key":"10.3233\/SW-233471_ref47","unstructured":"M.B.X.\u00a0Yan and R.\u00a0Usbeck\u00a0(eds), Proceedings of the 7th Natural Language Interfaces for the Web of Data, (NLIWoD) Co-Located with the 19th European Semantic Web Conference (ESWC 2022), 2022."}],"container-title":["Semantic Web"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/SW-233471","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:27:09Z","timestamp":1777613229000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/SW-233471"}},"subtitle":["Shifting from DBpedia to Wikidata as a KG for KGQA"],"editor":[{"given":"Lucie-Aim\u00e9e","family":"Kaffee","sequence":"additional","affiliation":[{"name":"University of Southampton, United Kingdom"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Simon","family":"Razniewski","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Informatics, Germany"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Pavlos","family":"Vougiouklis","sequence":"additional","affiliation":[{"name":"Huawei Technologies, United Kingdom"}],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2023,11,28]]},"references-count":46,"URL":"https:\/\/doi.org\/10.3233\/sw-233471","relation":{},"ISSN":["2210-4968","1570-0844"],"issn-type":[{"value":"2210-4968","type":"electronic"},{"value":"1570-0844","type":"print"}],"subject":[],"published":{"date-parts":[[2023,11,28]]}}}