{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:14:54Z","timestamp":1750220094831,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":65,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,8,23]],"date-time":"2022-08-23T00:00:00Z","timestamp":1661212800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,8,23]]},"DOI":"10.1145\/3539813.3545146","type":"proceedings-article","created":{"date-parts":[[2022,8,25]],"date-time":"2022-08-25T22:18:32Z","timestamp":1661465912000},"page":"3-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Towards Formally Grounded Evaluation Measures for Semantic Parsing-based Knowledge Graph Question Answering"],"prefix":"10.1145","author":[{"given":"Trond","family":"Linjordet","sequence":"first","affiliation":[{"name":"University of Stavanger, Stavanger, Norway"}]},{"given":"Krisztian","family":"Balog","sequence":"additional","affiliation":[{"name":"University of Stavanger, Stavanger, Norway"}]},{"given":"Vinay","family":"Setty","sequence":"additional","affiliation":[{"name":"University of Stavanger, Stavanger, Norway"}]}],"member":"320","published-online":{"date-parts":[[2022,8,25]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_3_2_1_1_1","DOI":"10.1145\/3291992.3291996"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_2_1","DOI":"10.1145\/3077136.3084369"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_3_1","DOI":"10.1007\/s10791-008-9066-8"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_4_1","DOI":"10.1007\/s10791-019-09355-y"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_5_1","DOI":"10.1007\/s10791-020-09374-0"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_6_1","DOI":"10.1145\/3209978.3210024"},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (ACL '13)","author":"Andreas Jacob","year":"2013","unstructured":"Jacob Andreas , Andreas Vlachos , and Stephen Clark . 2013 . Semantic Parsing as Machine Translation . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (ACL '13) . 47--52. Jacob Andreas, Andreas Vlachos, and Stephen Clark. 2013. Semantic Parsing as Machine Translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (ACL '13). 47--52."},{"unstructured":"Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 [cs.CL]  Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 [cs.CL]","key":"e_1_3_2_1_8_1"},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (COLING '16)","author":"Bao Junwei","year":"2016","unstructured":"Junwei Bao , Nan Duan , Zhao Yan , Ming Zhou , and Tiejun Zhao . 2016 . Constraintbased Question Answering with Knowledge Graph . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (COLING '16) . 2503--2514. Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraintbased Question Answering with Knowledge Graph. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (COLING '16). 2503--2514."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_10_1","DOI":"10.1145\/2806416.2806472"},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP '13)","author":"Berant Jonathan","year":"2013","unstructured":"Jonathan Berant , Andrew Chou , Roy Frostig , and Percy Liang . 2013 . Semantic Parsing on Freebase from Question-Answer Pairs . In Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP '13) . 1533--1544. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP '13). 1533--1544."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_12_1","DOI":"10.1016\/j.websem.2009.07.002"},{"unstructured":"Antoine Bordes Nicolas Usunier Sumit Chopra and Jason Weston. 2015. Large-scale Simple Question Answering with Memory Networks. arXiv:1506.02075 [cs.CL]  Antoine Bordes Nicolas Usunier Sumit Chopra and Jason Weston. 2015. Large-scale Simple Question Answering with Memory Networks. arXiv:1506.02075 [cs.CL]","key":"e_1_3_2_1_13_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_14_1","DOI":"10.5555\/146680.146685"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_15_1","DOI":"10.1145\/2499178.2499182"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_16_1","DOI":"10.18653\/v1\/2021.findings-acl.19"},{"key":"e_1_3_2_1_17_1","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL '13)","author":"Cai Qingqing","year":"2013","unstructured":"Qingqing Cai and Alexander Yates . 2013 . Large-Scale Semantic Parsing via Schema Matching and Lexicon Extension . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL '13) . 423--433. Qingqing Cai and Alexander Yates. 2013. Large-Scale Semantic Parsing via Schema Matching and Lexicon Extension. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL '13). 423--433."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_18_1","DOI":"10.1002\/widm.1389"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_19_1","DOI":"10.1007\/978-3-030-49461-2_25"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_20_1","DOI":"10.1007\/s10115-017-1100-y"},{"key":"e_1_3_2_1_21_1","volume-title":"Proceedings of the 16th International Semantic Web conference (ISWC '17)","author":"Diefenbach Dennis","year":"2017","unstructured":"Dennis Diefenbach , Thomas Tanon , Kamal Singh , and Pierre Maret . 2017 . Question Answering Benchmarks for Wikidata . In Proceedings of the 16th International Semantic Web conference (ISWC '17) . Dennis Diefenbach, Thomas Tanon, Kamal Singh, and Pierre Maret. 2017. Question Answering Benchmarks for Wikidata. In Proceedings of the 16th International Semantic Web conference (ISWC '17)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_22_1","DOI":"10.18653\/v1\/D19-1263"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_23_1","DOI":"10.18653\/v1\/P16-1004"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_24_1","DOI":"10.1007\/978-3-030-30796-7_5"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_25_1","DOI":"10.1007\/s10791-019-09362-z"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_26_1","DOI":"10.1145\/2808194.2809452"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_27_1","DOI":"10.1145\/3121050.3121058"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_28_1","DOI":"10.1109\/TKDE.2018.2840708"},{"key":"e_1_3_2_1_29_1","volume-title":"Proceedings of the 34th International Conference on Machine Learning -","volume":"70","author":"Gehring Jonas","unstructured":"Jonas Gehring , Michael Auli , David Grangier , Denis Yarats , and Yann N. Dauphin . 2017. Convolutional Sequence to Sequence Learning . In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML'17). 1243--1252. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML'17). 1243--1252."},{"key":"e_1_3_2_1_30_1","volume-title":"Workshop on Linked Data Management, co-located with the W3C WEBBR (W3C WEBBR '18)","author":"Hartmann Ann-Kathrin","year":"2018","unstructured":"Ann-Kathrin Hartmann , Edgard Marx , and Tommaso Soru . 2018 . Generating a Large Dataset for Neural Question Answering over the DBpedia Knowledge Base . In Workshop on Linked Data Management, co-located with the W3C WEBBR (W3C WEBBR '18) . Ann-Kathrin Hartmann, Edgard Marx, and Tommaso Soru. 2018. Generating a Large Dataset for Neural Question Answering over the DBpedia Knowledge Base. In Workshop on Linked Data Management, co-located with the W3C WEBBR (W3C WEBBR '18)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_31_1","DOI":"10.1007\/s10115-020-01534-4"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_32_1","DOI":"10.1017\/CBO9780511790423"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_33_1","DOI":"10.24963\/ijcai.2021\/611"},{"key":"e_1_3_2_1_34_1","volume-title":"ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. 74--81.","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin . 2004 . ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. 74--81. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. 74--81."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_35_1","DOI":"10.1145\/3409256.3409836"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_36_1","DOI":"10.18653\/v1\/D18-1242"},{"doi-asserted-by":"crossref","unstructured":"Minh-Thang Luong Hieu Pham and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. arXiv:1508.04025 [cs.CL]  Minh-Thang Luong Hieu Pham and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. arXiv:1508.04025 [cs.CL]","key":"e_1_3_2_1_37_1","DOI":"10.18653\/v1\/D15-1166"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_38_1","DOI":"10.1007\/978-3-030-30793-6_28"},{"volume-title":"Introduction to Information Retrieval","author":"Manning Christopher D.","unstructured":"Christopher D. Manning , Prabhakar Raghavan , and Hinrich Sch\u00fctze . 2008. Introduction to Information Retrieval . Cambridge University Press . Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch\u00fctze. 2008. Introduction to Information Retrieval. Cambridge University Press.","key":"e_1_3_2_1_39_1"},{"key":"e_1_3_2_1_40_1","volume-title":"Learning for Semantic Parsing. In International Conference on Intelligent Text Processing and Computational Linguistics (CICLing' 07)","author":"Mooney Raymond J","year":"2007","unstructured":"Raymond J Mooney . 2007 . Learning for Semantic Parsing. In International Conference on Intelligent Text Processing and Computational Linguistics (CICLing' 07) . 311--324. Raymond J Mooney. 2007. Learning for Semantic Parsing. In International Conference on Intelligent Text Processing and Computational Linguistics (CICLing' 07). 311--324."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_41_1","DOI":"10.1007\/978-3-030-88361-4_14"},{"key":"e_1_3_2_1_42_1","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL '02)","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . Bleu: A Method for Automatic Evaluation of Machine Translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL '02) . 311--318. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL '02). 311--318."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_43_1","DOI":"10.1145\/3460231.3474234"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_44_1","DOI":"10.18653\/v1\/D18-1051"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_45_1","DOI":"10.1007\/978-3-030-77385-4_21"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_46_1","DOI":"10.1561\/1500000001"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_47_1","DOI":"10.18653\/v1\/D16-1264"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_48_1","DOI":"10.2200\/S0113ED1V01Y202109ICR076"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_49_1","DOI":"10.1145\/3331184.3331215"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_50_1","DOI":"10.1145\/2808194.2809449"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_51_1","DOI":"10.1007\/s10791-019-09363-y"},{"doi-asserted-by":"crossref","unstructured":"Rico Sennrich Barry Haddow and Alexandra Birch. 2015. Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909 [cs.CL]  Rico Sennrich Barry Haddow and Alexandra Birch. 2015. Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909 [cs.CL]","key":"e_1_3_2_1_52_1","DOI":"10.18653\/v1\/P16-1162"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_53_1","DOI":"10.18653\/v1\/D19-1248"},{"key":"e_1_3_2_1_54_1","volume-title":"Saeedeh Shekarpour, Maria-Esther Vidal, and Jens Lehmann.","author":"Singh Kuldeep","year":"2020","unstructured":"Kuldeep Singh , Ioanna Lytra , Arun Sethupat Radhakrishna , Saeedeh Shekarpour, Maria-Esther Vidal, and Jens Lehmann. 2020 . No One Is Perfect: Analysing the Performance of Question Answering Components over the DBpedia Knowledge Graph. Journal of Web Semantics 65 (2020). Kuldeep Singh, Ioanna Lytra, Arun Sethupat Radhakrishna, Saeedeh Shekarpour, Maria-Esther Vidal, and Jens Lehmann. 2020. No One Is Perfect: Analysing the Performance of Question Answering Components over the DBpedia Knowledge Graph. Journal of Web Semantics 65 (2020)."},{"unstructured":"Tommaso Soru Edgard Marx Diego Moussallem Gustavo Publio Andr\u00e9 Valdestilhas Diego Esteves and Ciro Baron Neto. 2017. SPARQL as a Foreign Language. arXiv:1708.07624 [cs.CL]  Tommaso Soru Edgard Marx Diego Moussallem Gustavo Publio Andr\u00e9 Valdestilhas Diego Esteves and Ciro Baron Neto. 2017. SPARQL as a Foreign Language. arXiv:1708.07624 [cs.CL]","key":"e_1_3_2_1_55_1"},{"key":"e_1_3_2_1_56_1","volume-title":"Neural Machine Translation for Query Construction and Composition. In ICML Workshop on Neural Abstract Machines & Program Induction (NAMPI v2) (ICML '18)","author":"Soru Tommaso","year":"2018","unstructured":"Tommaso Soru , Edgard Marx , Andr\u00e9 Valdestilhas , Diego Esteves , Diego Moussallem , and Gustavo Publio . 2018 . Neural Machine Translation for Query Construction and Composition. In ICML Workshop on Neural Abstract Machines & Program Induction (NAMPI v2) (ICML '18) . Tommaso Soru, Edgard Marx, Andr\u00e9 Valdestilhas, Diego Esteves, Diego Moussallem, and Gustavo Publio. 2018. Neural Machine Translation for Query Construction and Composition. In ICML Workshop on Neural Abstract Machines & Program Induction (NAMPI v2) (ICML '18)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_57_1","DOI":"10.18653\/v1\/D16-1054"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_58_1","DOI":"10.18653\/v1\/N18-1059"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_59_1","DOI":"10.1007\/978-3-319-68204-4_22"},{"doi-asserted-by":"crossref","unstructured":"Christina Unger Andr\u00e9 Freitas and Philipp Cimiano. 2014. An Introduction to Question Answering over Linked Data. 100--140.  Christina Unger Andr\u00e9 Freitas and Philipp Cimiano. 2014. An Introduction to Question Answering over Linked Data. 100--140.","key":"e_1_3_2_1_60_1","DOI":"10.1007\/978-3-319-10587-1_2"},{"key":"e_1_3_2_1_61_1","volume-title":"Bastian Haarmann, Anastasia Krithara, Michael R\u00f6der, and Giulio Napolitano.","author":"Usbeck Ricardo","year":"2017","unstructured":"Ricardo Usbeck , Axel-Cyrille Ngonga Ngomo , Bastian Haarmann, Anastasia Krithara, Michael R\u00f6der, and Giulio Napolitano. 2017 . 7th Open Challenge on Question Answering over Linked Data (QALD-7). In Semantic Web Evaluation Challenge (SWEC '17). 59--69. Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael R\u00f6der, and Giulio Napolitano. 2017. 7th Open Challenge on Question Answering over Linked Data (QALD-7). In Semantic Web Evaluation Challenge (SWEC '17). 59--69."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_62_1","DOI":"10.3233\/SW-180312"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_63_1","DOI":"10.5555\/3295222.3295349"},{"unstructured":"Wen-tau Yih Ming-Wei Chang Xiaodong He and Jianfeng Gao. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (ACL-IJCNLP '15). 1321--1331.  Wen-tau Yih Ming-Wei Chang Xiaodong He and Jianfeng Gao. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (ACL-IJCNLP '15). 1321--1331.","key":"e_1_3_2_1_64_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_65_1","DOI":"10.1016\/j.future.2020.12.013"}],"event":{"sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"],"acronym":"ICTIR '22","name":"ICTIR '22: The 2022 ACM SIGIR International Conference on the Theory of Information Retrieval","location":"Madrid Spain"},"container-title":["Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3539813.3545146","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3539813.3545146","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:10:03Z","timestamp":1750183803000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3539813.3545146"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,23]]},"references-count":65,"alternative-id":["10.1145\/3539813.3545146","10.1145\/3539813"],"URL":"https:\/\/doi.org\/10.1145\/3539813.3545146","relation":{},"subject":[],"published":{"date-parts":[[2022,8,23]]},"assertion":[{"value":"2022-08-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}