{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T13:07:04Z","timestamp":1765544824715},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"8","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,4]]},"abstract":"<jats:p>Recently, there has been an increase in the number of knowledge graphs that can be only queried by experts. However, describing questions using structured queries is not straightforward for non-expert users who need to have sufficient knowledge about both the vocabulary and the structure of the queried knowledge graph, as well as the syntax of the structured query language used to describe the user's information needs. The most popular approach introduced to overcome the aforementioned challenges is to use natural language to query these knowledge graphs. Although several question answering benchmarks can be used to evaluate question-answering systems over a number of popular knowledge graphs, choosing a benchmark to accurately assess the quality of a question answering system is a challenging task.<\/jats:p>\n          <jats:p>In this paper, we introduce CBench, an extensible, and more informative benchmarking suite for analyzing benchmarks and evaluating question answering systems. CBench can be used to analyze existing benchmarks with respect to several fine-grained linguistic, syntactic, and structural properties of the questions and queries in the benchmark. We show that existing benchmarks vary significantly with respect to these properties deeming choosing a small subset of them unreliable in evaluating QA systems. Until further research improves the quality and comprehensiveness of benchmarks, CBench can be used to facilitate this evaluation using a set of popular benchmarks that can be augmented with other user-provided benchmarks. CBench not only evaluates a question answering system based on popular single-number metrics but also gives a detailed analysis of the linguistic, syntactic, and structural properties of answered and unanswered questions to better help the developers of question answering systems to better understand where their system excels and where it struggles.<\/jats:p>","DOI":"10.14778\/3457390.3457398","type":"journal-article","created":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T22:48:38Z","timestamp":1634856518000},"page":"1325-1337","source":"Crossref","is-referenced-by-count":7,"title":["CBench"],"prefix":"10.14778","volume":"14","author":[{"given":"Abdelghny","family":"Orogat","sequence":"first","affiliation":[{"name":"Carleton University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Isabelle","family":"Liu","sequence":"additional","affiliation":[{"name":"Carleton University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmed","family":"El-Roby","sequence":"additional","affiliation":[{"name":"Carleton University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,10,21]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"http:\/\/www.w3.org\/TR\/sparql11-query\/","author":"SPARQL","year":"2013"},{"key":"e_1_2_1_2_1","volume-title":"http:\/\/www.w3.org\/TR\/2014\/REC-rdf11-concepts-20140225\/","author":"RDF","year":"2014"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HL)","author":"Abujabal A.","year":"2019"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052583"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the International Conference on Computational Linguistics","author":"Akbik A.","year":"2018"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1785162.1785216"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the International Conference on Computational Linguistics","author":"Azmy M.","year":"2018"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Berant J.","year":"2013"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376746"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/3167892.3167895"},{"key":"e_1_2_1_11_1","volume-title":"Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075","author":"Bordes A.","year":"2015"},{"key":"e_1_2_1_12_1","volume-title":"CLEF Working Notes Papers","author":"Cabrio E.","year":"2013"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/2887379.2887382"},{"key":"e_1_2_1_14_1","volume-title":"Qakis@ QALD-2","author":"Cabrio E.","year":"2012"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Cai Q.","year":"2013"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/2898607.2898816"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1038"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055549"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-69146-6_8"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1247480.1247487"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-34129-3_19"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623677"},{"key":"e_1_2_1_23_1","volume-title":"Survey on challenges of question answering in the semantic web. Semantic Web, 8(6)","author":"H\u00f6ffner K.","year":"2017"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1162"},{"issue":"5","key":"e_1_2_1_25_1","volume":"30","author":"Hu S.","year":"2017","journal-title":"IEEE Transactions on Knowledge and Data Engineering (TKDE)"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3191536"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the International Semantic Web Conference (ISWC)","author":"Kaufmann E.","year":"2006"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-020-00383-w"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/2512665.2512668"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1101"},{"key":"e_1_2_1_31_1","volume-title":"UMBC Computer Science and Electrical Engineering Department Collection","author":"Matuszek C.","year":"2006"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2874809"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-93417-4_40"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186023"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1054"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242667"},{"key":"e_1_2_1_37_1","volume-title":"Platypus - A Multilingual Question Answering Platform for Wikidata. Technical report","author":"Tanon T. P.","year":"2018"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68204-4_22"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187923"},{"key":"e_1_2_1_40_1","volume-title":"Workshop on Question Answering Over Linked Data (QALD-1)","author":"Unger C.","year":"2011"},{"key":"e_1_2_1_41_1","volume-title":"CLEF Working Notes Papers","author":"Unger C.","year":"2014"},{"key":"e_1_2_1_42_1","volume-title":"CLEF Working Notes Papers","author":"Unger C.","year":"2015"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46565-4_13"},{"key":"e_1_2_1_44_1","volume-title":"Joint Workshop on Natural Language Interfaces for Web of Data (NLIWoD) and Question Answering over Linked Data challenge","author":"Usbeck R.","year":"2018"},{"key":"e_1_2_1_45_1","volume-title":"Joint Proceedings of the International Workshop on Benchmarking Linked Data and Natural Language Interfaces for the Web of Data (NLIWoD)","author":"Usbeck R.","year":"2018"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-69146-6_6"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2629489"},{"key":"e_1_2_1_48_1","volume-title":"Linguistic Data Consortium","author":"Weischedel R.","year":"2013"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505677"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2009.07.005"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/3236187.3236192"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/1785162.1785213"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610525"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3457390.3457398","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:45:53Z","timestamp":1672224353000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3457390.3457398"}},"subtitle":["towards better evaluation of question answering over knowledge graphs"],"short-title":[],"issued":{"date-parts":[[2021,4]]},"references-count":53,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2021,4]]}},"alternative-id":["10.14778\/3457390.3457398"],"URL":"https:\/\/doi.org\/10.14778\/3457390.3457398","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,4]]}}}