{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T15:16:00Z","timestamp":1777130160548,"version":"3.51.4"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,6,13]],"date-time":"2023-06-13T00:00:00Z","timestamp":1686614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2023,6,13]]},"abstract":"<jats:p>Recently, there has been an upsurge in the number of knowledge graphs (KG) that can only be accessed by experts. Non-expert users lack an adequate understanding of the queried knowledge graph's vocabulary and structure, as well as the syntax of the structured query language used to express the user's information needs. To increase the user base of these KGs, a set of Question Answering (QA) systems that use natural language to query these knowledge graphs have been introduced. However, finding a benchmark that accurately evaluates the quality of a QA system is a difficult task due to (1) the high degree of variation in the fine-grained properties among the existing benchmarks, (2) the static nature of the existing benchmarks versus the evolving nature of KGs, and (3) the limited number of KGs targeted by existing benchmarks, which hinders the usability of QA systems in real-world deployment over KGs that are different from those that were used in the evaluation of the QA systems. In this paper, we introduce Maestro, a benchmark generation system for question answering over knowledge graphs. Maestro can generate a new benchmark for any KG given the KG and, optionally, a text corpus that covers this KG. The benchmark generated by Maestro is guaranteed to cover all the properties of the natural language questions and queries that were encountered in the literature as long as the targeted KG includes these properties. Maestro also generates high-quality natural language questions with various utterances that are on par with manually-generated ones to better evaluate QA systems.<\/jats:p>","DOI":"10.1145\/3589322","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T20:26:45Z","timestamp":1687292805000},"page":"1-24","source":"Crossref","is-referenced-by-count":8,"title":["Maestro: Automatic Generation of Comprehensive Benchmarks for Question Answering Over Knowledge Graphs"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-2373-5375","authenticated-orcid":false,"given":"Abdelghny","family":"Orogat","sequence":"first","affiliation":[{"name":"Carleton University, Ottawa, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0737-3400","authenticated-orcid":false,"given":"Ahmed","family":"El-Roby","sequence":"additional","affiliation":[{"name":"Carleton University, Ottawa, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,6,20]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"http:\/\/www.w3.org\/TR\/sparql11-query\/","author":"SPARQL","year":"2013","unstructured":"SPARQL 1.1 query language. http:\/\/www.w3.org\/TR\/sparql11-query\/, 2013."},{"key":"e_1_2_2_2_1","volume-title":"http:\/\/www.w3.org\/TR\/2014\/REC-rdf11-concepts-20140225\/","author":"RDF","year":"2014","unstructured":"RDF 1.1 concepts and abstract syntax. http:\/\/www.w3.org\/TR\/2014\/REC-rdf11-concepts-20140225\/, 2014."},{"key":"e_1_2_2_3_1","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HL)","author":"Abujabal A.","year":"2019","unstructured":"A. Abujabal, R. Saha Roy, M. Yahya, and G. Weikum. ComQA: A community-sourced dataset for complex factoid question answering with paraphrase clusters. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HL), 2019."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052583"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-76298-0_52"},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the International Conference on Computational Linguistics","author":"Azmy M.","year":"2018","unstructured":"M. Azmy, P. Shi, J. Lin, and I. Ilyas. Farewell Freebase: Migrating the simplequestions dataset to dbpedia. In Proceedings of the International Conference on Computational Linguistics, 2018."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D13-1160"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376746"},{"key":"e_1_2_2_9_1","volume-title":"Proceedings of the VLDB Endowment, 11(2)","author":"Bonifati A.","year":"2017","unstructured":"A. Bonifati, W. Martens, and T. Timm. An analytical study of large SPARQL query logs. Proceedings of the VLDB Endowment, 11(2), 2017."},{"key":"e_1_2_2_10_1","volume-title":"Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075","author":"Bordes A.","year":"2015","unstructured":"A. Bordes, N. Usunier, S. Chopra, and J. Weston. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015."},{"key":"e_1_2_2_11_1","volume-title":"CLEF Working Notes Papers","author":"Cabrio E.","year":"2013","unstructured":"E. Cabrio, P. Cimiano, V. Lopez, and S. Walter. QALD-3: Multilingual question answering over linked data. In CLEF Working Notes Papers, 2013."},{"key":"e_1_2_2_12_1","volume-title":"Proceedings of the International Semantic Web Conference (ISWC)","author":"Cabrio E.","year":"2012","unstructured":"E. Cabrio, J. Cojan, A. P. Aprosio, B. Magnini, A. Lavelli, and F. Gandon. QAKiS: an open domain QA system based on relational patterns. In Proceedings of the International Semantic Web Conference (ISWC), 2012."},{"key":"e_1_2_2_13_1","volume-title":"Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Cai Q.","year":"2013","unstructured":"Q. Cai and A. Yates. Large-scale semantic parsing via schema matching and lexicon extension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2013."},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v24i1.7519"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055549"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-69146-6_8"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30796-7_5"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623677"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30796-7_8"},{"key":"e_1_2_2_20_1","volume-title":"Survey on challenges of question answering in the semantic web. Semantic Web, 8(6)","author":"H\u00f6ffner K.","year":"2017","unstructured":"K. H\u00f6ffner, S. Walter, E. Marx, R. Usbeck, J. Lehmann, and A.-C. Ngonga Ngomo. Survey on challenges of question answering in the semantic web. Semantic Web, 8(6), 2017."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3191536"},{"key":"e_1_2_2_22_1","volume-title":"Proceedings of the International Semantic Web Conference (ISWC)","author":"Kaufmann E.","year":"2006","unstructured":"E. Kaufmann, A. Bernstein, and R. Zumstein. Querix: A natural language interface to query ontologies based on clarification dialogs. In Proceedings of the International Semantic Web Conference (ISWC), 2006."},{"key":"e_1_2_2_23_1","volume-title":"PowerAqua: supporting users in querying and exploring the semantic web. Semantic Web, 3(3)","author":"Lopez V.","year":"2012","unstructured":"V. Lopez, M. Fern\u00e1ndez, E. Motta, and N. Stieler. PowerAqua: supporting users in querying and exploring the semantic web. Semantic Web, 3(3), 2012."},{"key":"e_1_2_2_24_1","volume-title":"UMBC Computer Science and Electrical Engineering Department Collection","author":"Matuszek C.","year":"2006","unstructured":"C. Matuszek, M. Witbrock, J. Cabral, and J. DeOliveira. An introduction to the syntax and content of Cyc. UMBC Computer Science and Electrical Engineering Department Collection, 2006."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476311.3476326"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/3554821.3554869"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/3457390.3457398"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1054"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242667"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68204-4_22"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187923"},{"key":"e_1_2_2_32_1","volume-title":"Workshop on Question Answering Over Linked Data (QALD-1)","author":"Unger C.","year":"2011","unstructured":"C. Unger, P. Cimiano, V. Lopez, and E. Motta. Question answering over linked data (QALD-1). In Workshop on Question Answering Over Linked Data (QALD-1), 2011."},{"key":"e_1_2_2_33_1","volume-title":"CLEF Working Notes Papers","author":"Unger C.","year":"2014","unstructured":"C. Unger, C. Forascu, V. Lopez, A. Ngomo, E. Cabrio, P. Cimiano, and S. Walter. Question answering over linked data (QALD-4). In CLEF Working Notes Papers, 2014."},{"key":"e_1_2_2_34_1","volume-title":"CLEF Working Notes Papers","author":"Unger C.","year":"2015","unstructured":"C. Unger, C. Forascu, V. Lopez, A. Ngomo, E. Cabrio, P. Cimiano, and S. Walter. Question answering over linked data (QALD-5). In CLEF Working Notes Papers, 2015."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46565-4_13"},{"key":"e_1_2_2_36_1","volume-title":"Joint Workshop on Natural Language Interfaces for Web of Data (NLIWoD) and Question Answering over Linked Data challenge","author":"Usbeck R.","year":"2018","unstructured":"R. Usbeck, R. H. Gusmita, M. Saleem, and A. Ngomo. 9th challenge on question answering over linked data (QALD-9). Joint Workshop on Natural Language Interfaces for Web of Data (NLIWoD) and Question Answering over Linked Data challenge, 2018."},{"key":"e_1_2_2_37_1","volume-title":"Joint Proceedings of the International Workshop on Benchmarking Linked Data and Natural Language Interfaces for the Web of Data (NLIWoD)","author":"Usbeck R.","year":"2018","unstructured":"R. Usbeck, A. Ngomo, F. Conrads, M. R\u00f6der, and G. Napolitano. 8th challenge on question answering over linked data (QALD-8). Joint Proceedings of the International Workshop on Benchmarking Linked Data and Natural Language Interfaces for the Web of Data (NLIWoD), 2018."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-69146-6_6"},{"key":"e_1_2_2_39_1","volume-title":"Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10)","author":"Vrandecic D.","year":"2014","unstructured":"D. Vrandecic and M. Kr\u00f6tzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10), 2014."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505677"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2009.07.005"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3236187.3236192"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-76298-0_50"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3589322","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3589322","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:14Z","timestamp":1750178774000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3589322"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,13]]},"references-count":43,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,13]]}},"alternative-id":["10.1145\/3589322"],"URL":"https:\/\/doi.org\/10.1145\/3589322","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,13]]}}}