{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T21:49:53Z","timestamp":1774734593211,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T00:00:00Z","timestamp":1623196800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,6,9]]},"DOI":"10.1145\/3448016.3452836","type":"proceedings-article","created":{"date-parts":[[2021,6,18]],"date-time":"2021-06-18T17:22:39Z","timestamp":1624036959000},"page":"632-644","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["An In-Depth Benchmarking of Text-to-SQL Systems"],"prefix":"10.1145","author":[{"given":"Orest","family":"Gkini","sequence":"first","affiliation":[{"name":"Athena Research Center, Athens, Greece"}]},{"given":"Theofilos","family":"Belmpas","sequence":"additional","affiliation":[{"name":"Athena Research Center, Athens, Greece"}]},{"given":"Georgia","family":"Koutrika","sequence":"additional","affiliation":[{"name":"Athena Research Center, Athens, Greece"}]},{"given":"Yannis","family":"Ioannidis","sequence":"additional","affiliation":[{"name":"University of Athens & Athena Research Center, Athens, Greece"}]}],"member":"320","published-online":{"date-parts":[[2021,6,18]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-019-00567-8"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"crossref","unstructured":"Sanjay Agrawal Surajit Chaudhuri and Gautam Das. 2002. DBXplorer: A system for keyword-based search over relational databases. In ICDE. 5--16.  Sanjay Agrawal Surajit Chaudhuri and Gautam Das. 2002. DBXplorer: A system for keyword-based search over relational databases. In ICDE. 5--16.","DOI":"10.1145\/564691.564782"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1017\/S135132490000005X"},{"key":"e_1_3_2_2_4_1","unstructured":"Gaurav Bhalotia Arvind Hulgeri Charuta Nakhe Soumen Chakrabarti and Shashank Sudarshan. 2002. Keyword searching and browsing in databases using BANKS. In ICDE. 431--440.  Gaurav Bhalotia Arvind Hulgeri Charuta Nakhe Soumen Chakrabarti and Shashank Sudarshan. 2002. Keyword searching and browsing in databases using BANKS. In ICDE. 431--440."},{"key":"e_1_3_2_2_5_1","first-page":"932","article-title":"SODA","volume":"5","author":"Blunschi Lukas","year":"2012","unstructured":"Lukas Blunschi , Claudio Jossen , Donald Kossmann , Magdalini Mori , and Kurt Stockinger . 2012 . SODA : Generating SQL for Business Users. PVLDB , Vol. 5 , 10 (2012), 932 -- 943 . Lukas Blunschi, Claudio Jossen, Donald Kossmann, Magdalini Mori, and Kurt Stockinger. 2012. SODA: Generating SQL for Business Users. PVLDB , Vol. 5, 10 (2012), 932--943.","journal-title":"Generating SQL for Business Users. PVLDB"},{"key":"e_1_3_2_2_6_1","volume-title":"Natural language to SQL: Where are we today? [Experiments and Analyses]. PVLDB","author":"ByeongHoon Hyeonji Kimy","year":"2020","unstructured":"Hyeonji Kimy ByeongHoon , Soy WookShin , and Hany Hongrae Lee . 2020. Natural language to SQL: Where are we today? [Experiments and Analyses]. PVLDB ( 2020 ). Hyeonji Kimy ByeongHoon, Soy WookShin, and Hany Hongrae Lee. 2020. Natural language to SQL: Where are we today? [Experiments and Analyses]. PVLDB (2020)."},{"key":"e_1_3_2_2_7_1","volume-title":"Manning","author":"de Marneffe Marie-Catherine","year":"2006","unstructured":"Marie-Catherine de Marneffe , Bill MacCartney , and Christopher D . Manning . 2006 . Generating typed dependency parses from phrase structure parses. LREC ( 2006), 449--454. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. LREC (2006), 449--454."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025227"},{"key":"e_1_3_2_2_9_1","volume-title":"Shan Wang, Lu Qin, Xiao Zhang, and Xuemin Lin.","author":"Ding Bolin","year":"2007","unstructured":"Bolin Ding , Jeffrey Xu Yu , Shan Wang, Lu Qin, Xiao Zhang, and Xuemin Lin. 2007 . Finding top-k min-cost connected trees in databases. In ICDE. 836--845. Bolin Ding, Jeffrey Xu Yu, Shan Wang, Lu Qin, Xiao Zhang, and Xuemin Lin. 2007. Finding top-k min-cost connected trees in databases. In ICDE. 836--845."},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1033"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1124"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"crossref","unstructured":"Hao He Haixun Wang Jun Yang and Philip S Yu. 2007. BLINKS: ranked keyword searches on graphs. In ACM SIGMOD . ACM 305--316.  Hao He Haixun Wang Jun Yang and Philip S Yu. 2007. BLINKS: ranked keyword searches on graphs. In ACM SIGMOD . ACM 305--316.","DOI":"10.1145\/1247480.1247516"},{"key":"e_1_3_2_2_13_1","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018","author":"Hosu Ionel-Alexandru","year":"2018","unstructured":"Ionel-Alexandru Hosu , Radu Cristian Alexandru Iacob , Florin Brad , Stefan Ruseti , and Traian Rebedea . 2018 . Natural Language Interface for Databases Using a Dual-Encoder Model . In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018 , Santa Fe, New Mexico, USA, August 20--26 , 2018 , , Emily M. Bender, Leon Derczynski, and Pierre Isabelle (Eds.). Association for Computational Linguistics, 514--524. Ionel-Alexandru Hosu, Radu Cristian Alexandru Iacob, Florin Brad, Stefan Ruseti, and Traian Rebedea. 2018. Natural Language Interface for Databases Using a Dual-Encoder Model. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20--26, 2018 , , Emily M. Bender, Leon Derczynski, and Pierre Isabelle (Eds.). Association for Computational Linguistics, 514--524."},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"crossref","unstructured":"Vagelis Hristidis Luis Gravano and Yannis Papakonstantinou. 2003. Efficient IR-style Keyword Search over Relational Databases. In VLDB. 850--861.  Vagelis Hristidis Luis Gravano and Yannis Papakonstantinou. 2003. Efficient IR-style Keyword Search over Relational Databases. In VLDB. 850--861.","DOI":"10.1016\/B978-012722442-8\/50080-X"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/1287369.1287427"},{"key":"e_1_3_2_2_16_1","unstructured":"Srinivasan Iyer Ioannis Konstas Alvin Cheung Jayant Krishnamurthy and Luke Zettlemoyer. 2017. Learning a Neural Semantic Parser from User Feedback. In ACL. 963--973.  Srinivasan Iyer Ioannis Konstas Alvin Cheung Jayant Krishnamurthy and Luke Zettlemoyer. 2017. Learning a Neural Semantic Parser from User Feedback. In ACL. 963--973."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"crossref","unstructured":"Mehdi Kargar Aijun An Nick Cercone Parke Godfrey Jaroslaw Szlichta and Xiaohui Yu. 2014. MeanKS: Meaningful Keyword Search in Relational Databases with Complex Schema .ACM.  Mehdi Kargar Aijun An Nick Cercone Parke Godfrey Jaroslaw Szlichta and Xiaohui Yu. 2014. MeanKS: Meaningful Keyword Search in Relational Databases with Complex Schema .ACM.","DOI":"10.1109\/ICDE.2015.7113302"},{"key":"e_1_3_2_2_18_1","volume-title":"Mooney","author":"Kate Rohit J.","year":"2006","unstructured":"Rohit J. Kate and Raymond J . Mooney . 2006 . Using String-Kernels for Learning Semantic Parsers. In ACL. Rohit J. Kate and Raymond J. Mooney. 2006. Using String-Kernels for Learning Semantic Parsers. In ACL."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735461.2735468"},{"key":"e_1_3_2_2_20_1","unstructured":"Yunyao Li and Davood Rafiei. 2017. Natural Language Data Management and Interfaces: Recent Development and Open Challenges. In ACM SIGMOD. 1765--1770.  Yunyao Li and Davood Rafiei. 2017. Natural Language Data Management and Interfaces: Recent Development and Open Challenges. In ACM SIGMOD. 1765--1770."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2866568"},{"key":"e_1_3_2_2_22_1","volume-title":"Spark: Top-k Keyword Query in Relational Databases. In ACM SIGMOD . 115--126.","author":"Luo Yi","year":"2007","unstructured":"Yi Luo , Xuemin Lin , Wei Wang , and Xiaofang Zhou . 2007 . Spark: Top-k Keyword Query in Relational Databases. In ACM SIGMOD . 115--126. Yi Luo, Xuemin Lin, Wei Wang, and Xiaofang Zhou. 2007. Spark: Top-k Keyword Query in Relational Databases. In ACM SIGMOD . 115--126."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.60"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"crossref","unstructured":"Ana-Maria Popescu Alex Armanasu Oren Etzioni David Ko and Alexander Yates. 2004. Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability. In COLING.  Ana-Maria Popescu Alex Armanasu Oren Etzioni David Ko and Alexander Yates. 2004. Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability. In COLING.","DOI":"10.3115\/1220355.1220376"},{"key":"e_1_3_2_2_26_1","unstructured":"EU Open Data Portal. [Online; accessed Jan-2021]. CORDIS - EU research projects under Horizon. https:\/\/data.europa.eu\/euodp\/en\/data\/dataset\/cordisH2020projects .  EU Open Data Portal. [Online; accessed Jan-2021]. CORDIS - EU research projects under Horizon. https:\/\/data.europa.eu\/euodp\/en\/data\/dataset\/cordisH2020projects ."},{"key":"e_1_3_2_2_27_1","volume-title":"Ashish R. Mittal, and Fatma \u00d6zcan.","author":"Saha Diptikalyan","year":"2016","unstructured":"Diptikalyan Saha , Avrilia Floratou , Karthik Sankaranarayanan , Umar Farooq Minhas , Ashish R. Mittal, and Fatma \u00d6zcan. 2016 . ATHENA : An Ontology-Driven System for Natural Language Querying over Relational Data Stores .VLDB. Diptikalyan Saha, Avrilia Floratou, Karthik Sankaranarayanan, Umar Farooq Minhas, Ashish R. Mittal, and Fatma \u00d6zcan. 2016. ATHENA: An Ontology-Driven System for Natural Language Querying over Relational Data Stores .VLDB."},{"key":"e_1_3_2_2_28_1","volume-title":"Stanford Encyclopedia of Philosophy. https:\/\/stanford.io\/2YXcECi","author":"Sennet Adam","unstructured":"Adam Sennet . 2016. Stanford Encyclopedia of Philosophy. https:\/\/stanford.io\/2YXcECi . Adam Sennet. 2016. Stanford Encyclopedia of Philosophy. https:\/\/stanford.io\/2YXcECi ."},{"key":"e_1_3_2_2_29_1","volume-title":"IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles. CoRR","author":"Shi Tianze","year":"2018","unstructured":"Tianze Shi , Kedar Tatwawadi , Kaushik Chakrabarti , Yi Mao , Oleksandr Polozov , and Weizhu Chen . 2018. IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles. CoRR , Vol. abs\/ 1809 .05054 ( 2018 ). Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. 2018. IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles. CoRR , Vol. abs\/1809.05054 (2018)."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-007-0075-9"},{"key":"e_1_3_2_2_31_1","unstructured":"Sloan Digital Sky Survey. [Online; accessed Jan-2021]. SkyServer. http:\/\/skyserver.sdss.org\/dr16\/en\/home.aspx .  Sloan Digital Sky Survey. [Online; accessed Jan-2021]. SkyServer. http:\/\/skyserver.sdss.org\/dr16\/en\/home.aspx ."},{"key":"e_1_3_2_2_32_1","volume-title":"Lohman","author":"Tata Sandeep","year":"2008","unstructured":"Sandeep Tata and Guy M . Lohman . 2008 . SQAK : Doing More with Keywords. In ACM SIGMOD. 889--902. Sandeep Tata and Guy M. Lohman. 2008. SQAK: Doing More with Keywords. In ACM SIGMOD. 889--902."},{"key":"e_1_3_2_2_33_1","volume-title":"38th ACM SIGPLAN. 452--466.","author":"Wang Chenglong","unstructured":"Chenglong Wang , Alvin Cheung , and Rastislav Bod'i k. 2017. Synthesizing highly expressive SQL queries from input-output examples . In 38th ACM SIGPLAN. 452--466. Chenglong Wang, Alvin Cheung, and Rastislav Bod'i k. 2017. Synthesizing highly expressive SQL queries from input-output examples. In 38th ACM SIGPLAN. 452--466."},{"key":"e_1_3_2_2_34_1","unstructured":"Yushi Wang Jonathan Berant and Percy Liang. 2015. Building a Semantic Parser Overnight. In ACL. 1332--1342.  Yushi Wang Jonathan Berant and Percy Liang. 2015. Building a Semantic Parser Overnight. In ACL. 1332--1342."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.3115\/981732.981751"},{"key":"e_1_3_2_2_36_1","volume-title":"SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning. CoRR","author":"Xu Xiaojun","year":"2017","unstructured":"Xiaojun Xu , Chang Liu , and Dawn Song . 2017. SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning. CoRR , Vol. abs\/ 1711 .04436 ( 2017 ). Xiaojun Xu, Chang Liu, and Dawn Song. 2017. SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning. CoRR , Vol. abs\/1711.04436 (2017)."},{"key":"e_1_3_2_2_37_1","unstructured":"Navid Yaghmazadeh. 2016. SQLizer Datasets and Queries. http:\/\/bit.ly\/2ueiGBv .  Navid Yaghmazadeh. 2016. SQLizer Datasets and Queries. http:\/\/bit.ly\/2ueiGBv ."},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133887"},{"key":"e_1_3_2_2_39_1","volume-title":"Neural Enquirer: Learning to Query Tables in Natural Language. In IJCAI. 2308--2314.","author":"Yin Pengcheng","year":"2016","unstructured":"Pengcheng Yin , Zhengdong Lu , Hang Li , and Ben Kao . 2016 . Neural Enquirer: Learning to Query Tables in Natural Language. In IJCAI. 2308--2314. Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. 2016. Neural Enquirer: Learning to Query Tables in Natural Language. In IJCAI. 2308--2314."},{"key":"e_1_3_2_2_40_1","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT","volume":"2","author":"Yu Tao","year":"2018","unstructured":"Tao Yu , Zifan Li , Zilin Zhang , Rui Zhang , and Dragomir R. Radev . 2018a. TypeSQL: Knowledge-Based Type-Aware Neural Text-to-SQL Generation . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT , New Orleans, Louisiana, USA, June 1--6 , 2018 , Volume 2 (Short Papers) , , Marilyn A. Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, 588--594. Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir R. Radev. 2018a. TypeSQL: Knowledge-Based Type-Aware Neural Text-to-SQL Generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1--6, 2018, Volume 2 (Short Papers) , , Marilyn A. Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, 588--594."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1425"},{"key":"e_1_3_2_2_42_1","volume-title":"Mong-Li Lee, and Tok Wang Ling.","author":"Zeng Zhong","year":"2014","unstructured":"Zhong Zeng , Zhifeng Bao , Thuy Ngoc Le , Mong-Li Lee, and Tok Wang Ling. 2014 . ExpressQ: Identifying Keyword Context and Search Target in Relational Keyword Queries. In CIKM . Zhong Zeng, Zhifeng Bao, Thuy Ngoc Le, Mong-Li Lee, and Tok Wang Ling. 2014. ExpressQ: Identifying Keyword Context and Search Target in Relational Keyword Queries. In CIKM ."},{"key":"e_1_3_2_2_43_1","volume-title":"Mong Li Lee, and Tok Wang Ling","author":"Zeng Zhong","year":"2016","unstructured":"Zhong Zeng , Mong Li Lee, and Tok Wang Ling . 2016 . Answering Keyword Queries involving Aggregates and GROUPBY on Relational Databases. EDBT ( 2016), 161--172. Zhong Zeng, Mong Li Lee, and Tok Wang Ling. 2016. Answering Keyword Queries involving Aggregates and GROUPBY on Relational Databases. EDBT (2016), 161--172."},{"key":"e_1_3_2_2_44_1","volume-title":"Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR","author":"Zhong Victor","year":"2017","unstructured":"Victor Zhong , Caiming Xiong , and Richard Socher . 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR , Vol. abs\/ 1709 .00103 ( 2017 ). arxiv: 1709.00103 http:\/\/arxiv.org\/abs\/1709.00103 Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR , Vol. abs\/1709.00103 (2017). arxiv: 1709.00103 http:\/\/arxiv.org\/abs\/1709.00103"}],"event":{"name":"SIGMOD\/PODS '21: International Conference on Management of Data","location":"Virtual Event China","acronym":"SIGMOD\/PODS '21","sponsor":["SIGMOD ACM Special Interest Group on Management of Data"]},"container-title":["Proceedings of the 2021 International Conference on Management of Data"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448016.3452836","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3448016.3452836","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:05Z","timestamp":1750195685000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448016.3452836"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,9]]},"references-count":44,"alternative-id":["10.1145\/3448016.3452836","10.1145\/3448016"],"URL":"https:\/\/doi.org\/10.1145\/3448016.3452836","relation":{},"subject":[],"published":{"date-parts":[[2021,6,9]]},"assertion":[{"value":"2021-06-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}