{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,20]],"date-time":"2026-06-20T01:07:06Z","timestamp":1781917626691,"version":"3.54.5"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"5","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:p>Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborate their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar.<\/jats:p>\n          <jats:p>To explore the potential of open-source LLM, we investigate them in various scenarios, and further enhance their performance with supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the supervised fine-tuning. Additionally, towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspires further investigations and broad applications.<\/jats:p>","DOI":"10.14778\/3641204.3641221","type":"journal-article","created":{"date-parts":[[2024,5,2]],"date-time":"2024-05-02T22:05:43Z","timestamp":1714687543000},"page":"1132-1145","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":212,"title":["Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation"],"prefix":"10.14778","volume":"17","author":[{"given":"Dawei","family":"Gao","sequence":"first","affiliation":[{"name":"Alibaba Group"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haibin","family":"Wang","sequence":"additional","affiliation":[{"name":"Alibaba Group"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yaliang","family":"Li","sequence":"additional","affiliation":[{"name":"Alibaba Group"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiuyu","family":"Sun","sequence":"additional","affiliation":[{"name":"Alibaba Group"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yichen","family":"Qian","sequence":"additional","affiliation":[{"name":"Alibaba Group"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bolin","family":"Ding","sequence":"additional","affiliation":[{"name":"Alibaba Group"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingren","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,5,2]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"A General Language Assistant as a Laboratory for Alignment. CoRR abs\/2112.00861","author":"Askell Amanda","year":"2021","unstructured":"Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A General Language Assistant as a Laboratory for Alignment. CoRR abs\/2112.00861 (2021)."},{"key":"e_1_2_1_2_1","unstructured":"LILY Group at Yale University. 2018. Spider 1.0 Yale Semantic Parsing and Text-to-SQL Challenge. https:\/\/yale-lily.github.io\/spider."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 2020 International Conference on Management of Data. 2319--2329","author":"Baik Christopher","unstructured":"Christopher Baik, Zhongjun Jin, Michael J. Cafarella, and H. V. Jagadish. 2020. Duoquest: A Dual-Specification System for Expressive SQL Queries. In Proceedings of the 2020 International Conference on Management of Data. 2319--2329."},{"key":"e_1_2_1_4_1","volume-title":"37th IEEE International Conference on Data Engineering. 2177--2182","author":"Brunner Ursin","year":"2021","unstructured":"Ursin Brunner and Kurt Stockinger. 2021. ValueNet: A Natural Language-to-SQL System that Learns from Database Information. In 37th IEEE International Conference on Data Engineering. 2177--2182."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. 3977--3983","author":"Cai Ruichu","year":"2018","unstructured":"Ruichu Cai, Boyan Xu, Zhenjie Zhang, Xiaoyan Yang, Zijian Li, and Zhihao Liang. 2018. An Encoder-Decoder Framework Translating Natural Language to Database Queries. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. 3977--3983."},{"key":"e_1_2_1_6_1","volume-title":"How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings. CoRR abs\/2305.11853","author":"Chang Shuaichen","year":"2023","unstructured":"Shuaichen Chang and Eric Fosler-Lussier. 2023. How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings. CoRR abs\/2305.11853 (2023)."},{"key":"e_1_2_1_7_1","volume-title":"Xing","author":"Chiang Wei-Lin","year":"2023","unstructured":"Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https:\/\/lmsys.org\/blog\/2023-03-30-vicuna\/"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics. 2166--2187","author":"Deng Naihao","year":"2022","unstructured":"Naihao Deng, Yulong Chen, and Yue Zhang. 2022. Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect. In Proceedings of the 29th International Conference on Computational Linguistics. 2166--2187."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.105"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171--4186","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171--4186."},{"key":"e_1_2_1_11_1","volume-title":"A Survey for In-context Learning. CoRR abs\/2301.00234","author":"Dong Qingxiu","year":"2023","unstructured":"Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A Survey for In-context Learning. CoRR abs\/2301.00234 (2023)."},{"key":"e_1_2_1_12_1","volume-title":"C3: Zero-shot Text-to-SQL with ChatGPT. CoRR abs\/2307.07306","author":"Dong Xuemei","year":"2023","unstructured":"Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. 2023. C3: Zero-shot Text-to-SQL with ChatGPT. CoRR abs\/2307.07306 (2023)."},{"key":"e_1_2_1_13_1","volume-title":"Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. CoRR abs\/2308.15363","author":"Gao Dawei","year":"2023","unstructured":"Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. CoRR abs\/2308.15363 (2023)."},{"key":"e_1_2_1_14_1","volume-title":"A Case-Based Reasoning Framework for Adaptive Prompting in Cross-Domain Text-to-SQL. CoRR abs\/2304.13301","author":"Guo Chunxi","year":"2023","unstructured":"Chunxi Guo, Zhiliang Tian, Jintao Tang, Pancheng Wang, Zhihua Wen, Kang Yang, and Ting Wang. 2023. A Case-Based Reasoning Framework for Adaptive Prompting in Cross-Domain Text-to-SQL. CoRR abs\/2304.13301 (2023)."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 57th Conference of the Association for Computational Linguistics. 4524--4535","author":"Guo Jiaqi","year":"2019","unstructured":"Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation. In Proceedings of the 57th Conference of the Association for Computational Linguistics. 4524--4535."},{"key":"e_1_2_1_16_1","volume-title":"LoRA: Low-Rank Adaptation of Large Language Models. In The 10th International Conference on Learning Representations.","author":"Hu Edward J.","year":"2022","unstructured":"Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The 10th International Conference on Learning Representations."},{"key":"e_1_2_1_17_1","unstructured":"Binyuan Hui Ruiying Geng Lihan Wang Bowen Qin Yanyang Li Bowen Li Jian Sun and Yongbin Li. 2022. S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers. In Findings of the Association for Computational Linguistics. 1254--1262."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-022-00776-8"},{"key":"e_1_2_1_19_1","volume-title":"From Words to Code: Harnessing Data for Program Synthesis from Natural Language. CoRR abs\/2305.01598","author":"Khatry Anirudh","year":"2023","unstructured":"Anirudh Khatry, Joyce Cahoon, Jordan Henkel, Shaleen Deep, K. Venkatesh Emani, Avrilia Floratou, Sumit Gulwani, Vu Le, Mohammad Raza, Sherry Shi, Mukul Singh, and Ashish Tiwari. 2023. From Words to Code: Harnessing Data for Program Synthesis from Natural Language. CoRR abs\/2305.01598 (2023)."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2261--2273","author":"Lee Chia-Hsuan","year":"2021","unstructured":"Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2261--2273."},{"key":"e_1_2_1_21_1","volume-title":"RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In 37th AAAI Conference on Artificial Intelligence. 13067--13075","author":"Li Haoyang","year":"2023","unstructured":"Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In 37th AAAI Conference on Artificial Intelligence. 13067--13075."},{"key":"e_1_2_1_22_1","volume-title":"37th AAAI Conference on Artificial Intelligence. 13076--13084","author":"Li Jinyang","year":"2023","unstructured":"Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023. Graphix-T5: Mixing Pre-trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing. In 37th AAAI Conference on Artificial Intelligence. 13076--13084."},{"key":"e_1_2_1_23_1","volume-title":"Fei Huang, Reynold Cheng, and Yongbin Li.","author":"Li Jinyang","year":"2023","unstructured":"Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen-Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. CoRR abs\/2305.03111 (2023)."},{"key":"e_1_2_1_24_1","volume-title":"Yu","author":"Liu Aiwei","year":"2023","unstructured":"Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A Comprehensive Evaluation of ChatGPT's Zero-Shot Text-to-SQL Capability. CoRR abs\/2303.13547 (2023)."},{"key":"e_1_2_1_25_1","volume-title":"Multi-hop Relational Graph Attention Network for Text-to-SQL Parsing. In International Joint Conference on Neural Networks. 1--8.","author":"Liu Hu","year":"2023","unstructured":"Hu Liu, Yuliang Shi, Jianlin Zhang, Xinjun Wang, Hui Li, and Fanyu Kong. 2023. Multi-hop Relational Graph Attention Network for Text-to-SQL Parsing. In International Joint Conference on Neural Networks. 1--8."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. 100--114","author":"Liu Jiachang","year":"2022","unstructured":"Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. 100--114."},{"key":"e_1_2_1_27_1","volume-title":"Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies. CoRR abs\/2305.12586","author":"Nan Linyong","year":"2023","unstructured":"Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies. CoRR abs\/2305.12586 (2023)."},{"key":"e_1_2_1_29_1","unstructured":"OpenAI. 2023. Introducing ChatGPT. https:\/\/openai.com\/blog\/chatgpt. Last accessed on 2023-07-24."},{"key":"e_1_2_1_30_1","unstructured":"OpenAI. 2023. Rate limits. https:\/\/platform.openai.com\/docs\/guides\/rate-limits\/overview. Last accessed on 2023-07-24."},{"key":"e_1_2_1_31_1","unstructured":"OpenAI. 2023. SQL translate. https:\/\/platform.openai.com\/examples\/default-sql-translate. Last accessed on 2023-07-24."},{"key":"e_1_2_1_32_1","unstructured":"Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll L. Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell Peter Welinder Paul F. Christiano Jan Leike and Ryan Lowe. 2022. Training Language Models to Follow Instructions with Human Feedback. In NeurIPS."},{"key":"e_1_2_1_33_1","volume-title":"The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. CoRR abs\/2306.01116","author":"Penedo Guilherme","year":"2023","unstructured":"Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. CoRR abs\/2306.01116 (2023)."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics. 1593--1603","author":"Popescu Octavian","year":"2022","unstructured":"Octavian Popescu, Irene Manotas, Ngoc Phuoc An Vo, Hangu Yeo, Elahe Khorashani, and Vadim Sheinin. 2022. Addressing Limitations of Encoder-Decoder Based Approach to Text-to-SQL. In Proceedings of the 29th International Conference on Computational Linguistics. 1593--1603."},{"key":"e_1_2_1_35_1","volume-title":"DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. CoRR abs\/2304.11015","author":"Pourreza Mohammadreza","year":"2023","unstructured":"Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. CoRR abs\/2304.11015 (2023)."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 3215--3229","author":"Qi Jiexing","year":"2022","unstructured":"Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. 2022. RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 3215--3229."},{"key":"e_1_2_1_37_1","volume-title":"A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions. CoRR abs\/2208.13629","author":"Qin Bowen","year":"2022","unstructured":"Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, Fei Huang, and Yongbin Li. 2022. A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions. CoRR abs\/2208.13629 (2022)."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1561\/1900000078"},{"key":"e_1_2_1_39_1","volume-title":"Evaluating the Text-to-SQL Capabilities of Large Language Models. CoRR abs\/2204.00498","author":"Rajkumar Nitarshan","year":"2022","unstructured":"Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the Text-to-SQL Capabilities of Large Language Models. CoRR abs\/2204.00498 (2022)."},{"key":"e_1_2_1_40_1","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin Artyom Kozhevnikov Ivan Evtimov Joanna Bitton Manish Bhatt Cristian Canton-Ferrer Aaron Grattafiori Wenhan Xiong Alexandre D\u00e9fossez Jade Copet Faisal Azhar Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom and Gabriel Synnaeve. 2023. Code Llama: Open Foundation Models for Code. CoRR abs\/2308.12950 (2023)."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.779"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407858"},{"key":"e_1_2_1_43_1","volume-title":"SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL. CoRR abs\/2306.00739","author":"Sun Ruoxi","year":"2023","unstructured":"Ruoxi Sun, Sercan \u00d6. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. 2023. SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL. CoRR abs\/2306.00739 (2023)."},{"key":"e_1_2_1_44_1","volume-title":"Hashimoto","author":"Taori Rohan","year":"2023","unstructured":"Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https:\/\/github.com\/tatsu-lab\/stanford_alpaca."},{"key":"e_1_2_1_45_1","volume-title":"LLaMA: Open and Efficient Foundation Language Models. CoRR abs\/2302.13971","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur\u00e9lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. CoRR abs\/2302.13971 (2023)."},{"key":"e_1_2_1_46_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov and Thomas Scialom. 2023. LLAMA2: Open Foundation and Fine-Tuned Chat Models. CoRR (2023)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/3551793.3551841"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.677"},{"key":"e_1_2_1_49_1","volume-title":"The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1889--1898","author":"Wang Lihan","year":"2022","unstructured":"Lihan Wang, Bowen Qin, Binyuan Hui, Bowen Li, Min Yang, Bailin Wang, Binhua Li, Jian Sun, Fei Huang, Luo Si, and Yongbin Li. 2022. Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing. In The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1889--1898."},{"key":"e_1_2_1_50_1","volume-title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.","author":"Wang Xuezhi","year":"2023","unstructured":"Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1112"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.745"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3911--3921","author":"Yu Tao","unstructured":"Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3911--3921."},{"key":"e_1_2_1_54_1","volume-title":"A Survey of Large Language Models. CoRR abs\/2303.18223","author":"Zhao Wayne Xin","year":"2023","unstructured":"Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A Survey of Large Language Models. CoRR abs\/2303.18223 (2023)."},{"key":"e_1_2_1_55_1","volume-title":"Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. CoRR abs\/2306.05685","author":"Zheng Lianmin","year":"2023","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. CoRR abs\/2306.05685 (2023)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"crossref","unstructured":"Yanzhao Zheng Haibin Wang Baohua Dong Xingjun Wang and Changshan Li. 2022. HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing. In Findings of the Association for Computational Linguistics. 2997--3007.","DOI":"10.18653\/v1\/2022.findings-acl.236"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 396--411","author":"Zhong Ruiqi","year":"2020","unstructured":"Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 396--411."},{"key":"e_1_2_1_58_1","volume-title":"Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR abs\/1709.00103","author":"Zhong Victor","year":"2017","unstructured":"Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR abs\/1709.00103 (2017)."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3641204.3641221","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,2]],"date-time":"2024-05-02T22:08:12Z","timestamp":1714687692000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3641204.3641221"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1]]},"references-count":57,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,1]]}},"alternative-id":["10.14778\/3641204.3641221"],"URL":"https:\/\/doi.org\/10.14778\/3641204.3641221","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2024,1]]},"assertion":[{"value":"2024-05-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}