{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T18:29:10Z","timestamp":1784399350087,"version":"3.55.0"},"reference-count":88,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,5,29]],"date-time":"2024-05-29T00:00:00Z","timestamp":1716940800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"PCC@RUC"},{"name":"National Key Research & Develop Plan","award":["2023YFF0725100"],"award-info":[{"award-number":["2023YFF0725100"]}]},{"DOI":"10.13039\/501100006374","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62322214, U23A20299, 62076245, 62072460, 62172424, 62276270"],"award-info":[{"award-number":["62322214, U23A20299, 62076245, 62072460, 62172424, 62276270"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"NSF of China","doi-asserted-by":"crossref","award":["62122090, 62072461"],"award-info":[{"award-number":["62122090, 62072461"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Beijing Natural Science Foundation","award":["L222006"],"award-info":[{"award-number":["L222006"]}]},{"name":"Research Funds of Renmin University of China"},{"name":"Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China","award":["23XNH149"],"award-info":[{"award-number":["23XNH149"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2024,5,29]]},"abstract":"<jats:p>Language models have shown promising performance on the task of translating natural language questions into SQL queries (Text-to-SQL). However, most of the state-of-the-art (SOTA) approaches rely on powerful yet closed-source large language models (LLMs), such as ChatGPT and GPT-4, which may have the limitations of unclear model architectures, data privacy risks, and expensive inference overheads. To address the limitations, we introduce CodeS, a series of pre-trained language models with parameters ranging from 1B to 15B, specifically designed for the text-to-SQL task. CodeS is a fully open-source language model, which achieves superior accuracy with much smaller parameter sizes. This paper studies the research challenges in building CodeS. To enhance the SQL generation abilities of CodeS, we adopt an incremental pre-training approach using a specifically curated SQL-centric corpus. Based on this, we address the challenges of schema linking and rapid domain adaptation through strategic prompt construction and a bi-directional data augmentation technique. We conduct comprehensive evaluations on multiple datasets, including the widely used Spider benchmark, the newly released BIRD benchmark, robustness-diagnostic benchmarks such as Spider-DK, Spider-Syn, Spider-Realistic, and Dr.Spider, as well as two real-world datasets created for financial and academic applications. The experimental results show that our CodeS achieves new SOTA accuracy and robustness on nearly all challenging text-to-SQL benchmarks.<\/jats:p>","DOI":"10.1145\/3654930","type":"journal-article","created":{"date-parts":[[2024,5,30]],"date-time":"2024-05-30T09:44:53Z","timestamp":1717062293000},"page":"1-28","source":"Crossref","is-referenced-by-count":106,"title":["CodeS: Towards Building Open-source Language Models for Text-to-SQL"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6610-3841","authenticated-orcid":false,"given":"Haoyang","family":"Li","sequence":"first","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2019-225X","authenticated-orcid":false,"given":"Jing","family":"Zhang","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4582-3340","authenticated-orcid":false,"given":"Hanbing","family":"Liu","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4729-9903","authenticated-orcid":false,"given":"Ju","family":"Fan","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3855-3093","authenticated-orcid":false,"given":"Xiaokang","family":"Zhang","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-1127-3057","authenticated-orcid":false,"given":"Jun","family":"Zhu","sequence":"additional","affiliation":[{"name":"BEIJING AI-FINANCE TECHNOLOGIES CO. LTD, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5786-1513","authenticated-orcid":false,"given":"Renjie","family":"Wei","sequence":"additional","affiliation":[{"name":"BEIJING AI-FINANCE TECHNOLOGIES CO. LTD, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8367-1518","authenticated-orcid":false,"given":"Hongyan","family":"Pan","sequence":"additional","affiliation":[{"name":"BEIJING AI-FINANCE TECHNOLOGIES CO. LTD, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0089-1045","authenticated-orcid":false,"given":"Cuiping","family":"Li","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8132-9382","authenticated-orcid":false,"given":"Hong","family":"Chen","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,5,30]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Rohan Anil Andrew M. Dai Orhan Firat Melvin Johnson Dmitry Lepikhin Alexandre Passos Siamak Shakeri Emanuel Taropa Paige Bailey Zhifeng Chen and et al. 2023. PaLM 2 Technical Report. CoRR abs\/2305.10403 (2023). arXiv:2305.10403"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1378"},{"key":"e_1_2_2_3_1","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","author":"Brown Tom B.","year":"2020","unstructured":"Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE51399.2021.00220"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.198"},{"key":"e_1_2_2_6_1","volume-title":"How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Singledomain, and Cross-domain Settings. CoRR abs\/2305.11853","author":"Chang Shuaichen","year":"2023","unstructured":"Shuaichen Chang and Eric Fosler-Lussier. 2023. How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Singledomain, and Cross-domain Settings. CoRR abs\/2305.11853 (2023). arXiv:2305.11853"},{"key":"e_1_2_2_7_1","volume-title":"The Eleventh International Conference on Learning Representations, ICLR 2023","author":"Chang Shuaichen","year":"2023","unstructured":"Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, and et al. 2023. Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1--5, 2023."},{"key":"e_1_2_2_8_1","volume-title":"Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and et al.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond\u00e9 de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and et al. 2021. Evaluating Large Language Models Trained on Code. CoRR abs\/2107.03374 (2021). arXiv:2107.03374"},{"key":"e_1_2_2_9_1","volume-title":"Teaching Large Language Models to Self-Debug. CoRR abs\/2304.05128","author":"Chen Xinyun","year":"2023","unstructured":"Xinyun Chen, Maxwell Lin, Nathanael Sch\u00e4rli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. CoRR abs\/2304.05128 (2023). arXiv:2304.05128"},{"key":"e_1_2_2_10_1","volume-title":"Charles Sutton, Sebastian Gehrmann, and et al.","author":"Chowdhery Aakanksha","year":"2022","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and et al. 2022. PaLM: Scaling Language Modeling with Pathways. CoRR abs\/2204.02311 (2022). arXiv:2204.02311"},{"key":"e_1_2_2_11_1","volume-title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. CoRR abs\/2307.08691","author":"Dao Tri","year":"2023","unstructured":"Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. CoRR abs\/2307.08691 (2023). arXiv:2307.08691"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.105"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.105"},{"key":"e_1_2_2_14_1","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","volume":"1","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2--7, 2019, Volume 1 (Long and Short Papers). 4171--4186."},{"key":"e_1_2_2_15_1","volume-title":"Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. CoRR abs\/2305.14233","author":"Ding Ning","year":"2023","unstructured":"Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. CoRR abs\/2305.14233 (2023). arXiv:2305.14233"},{"key":"e_1_2_2_16_1","volume-title":"C3: Zero-shot Text-to-SQL with ChatGPT. CoRR abs\/2307.07306","author":"Dong Xuemei","year":"2023","unstructured":"Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. 2023. C3: Zero-shot Text-to-SQL with ChatGPT. CoRR abs\/2307.07306 (2023). arXiv:2307.07306"},{"key":"e_1_2_2_17_1","volume-title":"UniSAr: A Unified Structure-Aware Autoregressive Language Model for Text-to-SQL. CoRR abs\/2203.07781","author":"Dou Longxu","year":"2022","unstructured":"Longxu Dou, Yan Gao, Mingyang Pan, Dingzirui Wang, Jian-Guang Lou, Wanxiang Che, and Dechen Zhan. 2022. UniSAr: A Unified Structure-Aware Autoregressive Language Model for Text-to-SQL. CoRR abs\/2203.07781 (2022). arXiv:2203.07781"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/3583140.3583165"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.195"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.702"},{"key":"e_1_2_2_21_1","volume-title":"Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. CoRR abs\/2308.15363","author":"Gao Dawei","year":"2023","unstructured":"Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. CoRR abs\/2308.15363 (2023). arXiv:2308.15363"},{"key":"e_1_2_2_22_1","volume-title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling. CoRR abs\/2101.00027","author":"Gao Leo","year":"2021","unstructured":"Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and et al. 2021. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. CoRR abs\/2101.00027 (2021). arXiv:2101.00027"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.552"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589292"},{"key":"e_1_2_2_25_1","volume-title":"Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation. CoRR abs\/2306.08891","author":"Gu Zihui","year":"2023","unstructured":"Zihui Gu, Ju Fan, Nan Tang, Songyue Zhang, Yuxin Zhang, Zui Chen, Lei Cao, Guoliang Li, Sam Madden, and Xiaoyong Du. 2023. Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation. CoRR abs\/2306.08891 (2023). arXiv:2306.08891"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1444"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.398"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.806"},{"key":"e_1_2_2_29_1","volume-title":"Importance of Synthesizing High-quality Data for Text-to-SQL Parsing. In Findings of the Association for Computational Linguistics: ACL 2023","author":"Hu Yiqun","year":"2023","unstructured":"Yiqun Hu, Yiyun Zhao, Jiarong Jiang, Wuwei Lan, Henghui Zhu, Anuj Chauhan, Alexander Hanbo Li, Lin Pan, Jun Wang, Chung-Wei Hang, and et al. 2023. Importance of Synthesizing High-quality Data for Text-to-SQL Parsing. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9--14, 2023. 1327--1343."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1089"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.574"},{"key":"e_1_2_2_32_1","volume-title":"Jia Li, Chenghao Mou, Carlos Mu\u00f1oz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, and et al.","author":"Kocetkov Denis","year":"2022","unstructured":"Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Mu\u00f1oz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, and et al. 2022. The Stack: 3 TB of permissively licensed source code. CoRR abs\/2211.15533 (2022). arXiv:2211.15533"},{"key":"e_1_2_2_33_1","volume-title":"Camille Elepa\u00f1o, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and et al.","author":"Kung Tiffany H.","year":"2023","unstructured":"Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepa\u00f1o, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health 2, 2 (02 2023), 1--12."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.564"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735461.2735468"},{"key":"e_1_2_2_37_1","volume-title":"RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI","author":"Li Haoyang","year":"2023","unstructured":"Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7--14, 2023. 13067--13075."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i11.26536"},{"key":"e_1_2_2_39_1","unstructured":"Jinyang Li Binyuan Hui Ge Qu Binhua Li Jiaxi Yang Bowen Li BailinWang Bowen Qin Rongyu Cao Ruiying Geng and et al. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. CoRR abs\/2305.03111 (2023). arXiv:2305.03111"},{"key":"e_1_2_2_40_1","volume-title":"Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, and et al.","author":"Li Raymond","year":"2023","unstructured":"Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, and et al. 2023. StarCoder: may the source be with you! CoRR abs\/2305.06161 (2023). arXiv:2305.06161"},{"key":"e_1_2_2_41_1","volume-title":"Yu","author":"Liu Aiwei","year":"2023","unstructured":"Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability. CoRR abs\/2303.13547 (2023). arXiv:2303.13547"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.deelio-1.10"},{"key":"e_1_2_2_43_1","volume-title":"A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput. Surv. 55, 9","author":"Liu Pengfei","year":"2023","unstructured":"Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput. Surv. 55, 9 (2023), 195:1--195:35."},{"key":"e_1_2_2_44_1","volume-title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs\/1907.11692","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs\/1907.11692 (2019). arXiv:1907.11692"},{"key":"e_1_2_2_45_1","volume-title":"Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019","author":"Loshchilov Ilya","year":"2019","unstructured":"Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6--9, 2019."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2402.05121"},{"key":"e_1_2_2_47_1","volume-title":"CodeGen2: Lessons for Training LLMs on Programming and Natural Languages. CoRR abs\/2305.02309","author":"Nijkamp Erik","year":"2023","unstructured":"Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023. CodeGen2: Lessons for Training LLMs on Programming and Natural Languages. CoRR abs\/2305.02309 (2023). arXiv:2305.02309"},{"key":"e_1_2_2_48_1","volume-title":"CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023","author":"Nijkamp Erik","year":"2023","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1--5, 2023."},{"key":"e_1_2_2_49_1","volume-title":"Passage Re-ranking with BERT. CoRR abs\/1901.04085","author":"Nogueira Rodrigo Frassetto","year":"2019","unstructured":"Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. CoRR abs\/1901.04085 (2019). arXiv:1901.04085 http:\/\/arxiv.org\/abs\/1901.04085"},{"key":"e_1_2_2_50_1","volume-title":"Dean Carignan, and Eric Horvitz.","author":"Nori Harsha","year":"2023","unstructured":"Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of GPT-4 on Medical Challenge Problems. CoRR abs\/2303.13375 (2023). arXiv:2303.13375"},{"key":"e_1_2_2_51_1","unstructured":"OpenAI. 2023. GPT-4 Technical Report. CoRR abs\/2303.08774 (2023). arXiv:2303.08774"},{"key":"e_1_2_2_52_1","unstructured":"Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll L. Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray and et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS."},{"key":"e_1_2_2_53_1","volume-title":"Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI 2003","author":"Popescu Ana-Maria","year":"2003","unstructured":"Ana-Maria Popescu, Oren Etzioni, and Henry A. Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI 2003, Miami, FL, USA, January 12--15, 2003. 149--157."},{"key":"e_1_2_2_54_1","volume-title":"DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. CoRR abs\/2304.11015","author":"Pourreza Mohammadreza","year":"2023","unstructured":"Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. CoRR abs\/2304.11015 (2023). arXiv:2304.11015"},{"key":"e_1_2_2_55_1","unstructured":"Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever et al. 2018. Improving language understanding by generative pre-training. (2018)."},{"key":"e_1_2_2_56_1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020), 140:1--140:67.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00024"},{"key":"e_1_2_2_58_1","volume-title":"Evaluating the Text-to-SQL Capabilities of Large Language Models. CoRR abs\/2204.00498","author":"Rajkumar Nitarshan","year":"2022","unstructured":"Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the Text-to-SQL Capabilities of Large Language Models. CoRR abs\/2204.00498 (2022). arXiv:2204.00498"},{"key":"e_1_2_2_59_1","volume-title":"SmBoP: Semi-autoregressive Bottom-up Semantic Parsing. CoRR abs\/2010.12412","author":"Rubin Ohad","year":"2020","unstructured":"Ohad Rubin and Jonathan Berant. 2020. SmBoP: Semi-autoregressive Bottom-up Semantic Parsing. CoRR abs\/2010.12412 (2020). arXiv:2010.12412"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.779"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.75"},{"key":"e_1_2_2_62_1","volume-title":"Jun Wang, C\u00edcero Nogueira dos Santos, and Bing Xiang.","author":"Shi Peng","year":"2021","unstructured":"Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, C\u00edcero Nogueira dos Santos, and Bing Xiang. 2021. Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2--9, 2021. 13806--13814."},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742839"},{"key":"e_1_2_2_64_1","volume-title":"SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL. CoRR abs\/2306.00739","author":"Sun Ruoxi","year":"2023","unstructured":"Ruoxi Sun, Sercan \u00d6. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. 2023. SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL. CoRR abs\/2306.00739 (2023). arXiv:2306.00739"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1402008"},{"key":"e_1_2_2_66_1","volume-title":"Mooney","author":"Tang Lappoon R.","year":"2001","unstructured":"Lappoon R. Tang and Raymond J. Mooney. 2001. Using Multiple Clause Constructors in Inductive Logic Programming for Semantic Parsing. In Machine Learning: EMCL 2001, 12th European Conference on Machine Learning, Freiburg, Germany, September 5--7, 2001, Proceedings (Lecture Notes in Computer Science, Vol. 2167). 466--477."},{"key":"e_1_2_2_67_1","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar and et al. 2023. LLaMA: Open and Efficient Foundation Language Models. CoRR abs\/2302.13971 (2023). arXiv:2302.13971"},{"key":"e_1_2_2_68_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale and et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR abs\/2307.09288 (2023). arXiv:2307.09288"},{"key":"e_1_2_2_69_1","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4--9, 2017, Long Beach, CA, USA. 5998--6008."},{"key":"e_1_2_2_70_1","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5--10","author":"Shin Richard","year":"2020","unstructured":"BailinWang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5--10, 2020. 7567--7578."},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.220"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.754"},{"key":"e_1_2_2_73_1","volume-title":"Quoc V. Le, and Denny Zhou.","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS."},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.707"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133887"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186081"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196408"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1041"},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.745"},{"key":"e_1_2_2_80_1","volume-title":"GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing. In 9th International Conference on Learning Representations, ICLR 2021","author":"Yu Tao","year":"2021","unstructured":"Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir R. Radev, Richard Socher, and Caiming Xiong. 2021. GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3--7, 2021."},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1193"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1425"},{"key":"e_1_2_2_83_1","volume-title":"Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence Conference, AAAI 96","volume":"2","author":"John","year":"1996","unstructured":"John M. Zelle and Raymond J. Mooney. 1996. Learning to Parse Database Queries Using Inductive Logic Programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence Conference, AAAI 96, IAAI 96, Portland, Oregon, USA, August 4--8, 1996, Volume 2. 1050--1055."},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2022.3222168"},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330785"},{"key":"e_1_2_2_86_1","volume-title":"ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems. CoRR abs\/2306.04743","author":"Zhang Yi","year":"2023","unstructured":"Yi Zhang, Jan Deriu, George Katsogiannis-Meimarakis, Catherine Kosten, Georgia Koutrika, and Kurt Stockinger. 2023. ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems. CoRR abs\/2306.04743 (2023). arXiv:2306.04743"},{"key":"e_1_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.622"},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.29"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3654930","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3654930","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T14:38:11Z","timestamp":1755787091000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3654930"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,29]]},"references-count":88,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,5,29]]}},"alternative-id":["10.1145\/3654930"],"URL":"https:\/\/doi.org\/10.1145\/3654930","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,29]]}}}