{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:49:36Z","timestamp":1782848976873,"version":"3.54.5"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["Grant No. 62272400"],"award-info":[{"award-number":["Grant No. 62272400"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Fujian Provincial Natural Science Foundation of China","award":["Grant No. 2025J010002"],"award-info":[{"award-number":["Grant No. 2025J010002"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>\n                    Large Language Models (LLMs) can translate natural language (NL) into SQL, enabling non-experts to query databases via conversational interfaces. However, the generated SQL often contains\n                    <jats:italic toggle=\"yes\">intent-violating hallucinations<\/jats:italic>\n                    \u2014queries that are syntactically valid and executable, yet semantically misaligned with the user\u2019s question. These failures are especially risky in real-world settings where users cannot verify the correctness. \t \tIn this paper, we propose MRSQLGen, a framework for detecting intent-violating hallucinations, built on the metamorphic prompting paradigm. \tMRSQLGen rewrites the input prompt using task-specific transformation rules derived from a hallucination taxonomy, and validates the generated SQL by checking behavioral consistency across multiple executions. Each transformation is associated with a metamorphic relationship (MR) that defines the expected relation between results; discrepancies are aggregated through a majority-vote strategy to robustly flag hallucinations without ground-truth SQL. \tWe evaluate MRSQLGen on two benchmarks (Spider and Bird) using five representative LLMs, including GPT-4o. Experimental results demonstrate that MRSQLGen consistently outperforms state-of-the-art hallucination detection techniques, achieving higher precision and recall in detecting hallucinated SQL queries.\n                  <\/jats:p>","DOI":"10.1145\/3797146","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"411-434","source":"Crossref","is-referenced-by-count":0,"title":["Validating LLM-Generated SQL Queries through Metamorphic Prompting"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-9895-8360","authenticated-orcid":false,"given":"Li","family":"Lin","sequence":"first","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7759-9431","authenticated-orcid":false,"given":"Qinglin","family":"Zhu","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0110-3701","authenticated-orcid":false,"given":"Jintai","family":"Hong","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1424-6290","authenticated-orcid":false,"given":"Chong","family":"Wang","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7300-9215","authenticated-orcid":false,"given":"Yang","family":"Liu","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4648-3795","authenticated-orcid":false,"given":"Rongxin","family":"Wu","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2024. Qwen2.5-Coder-14B. https:\/\/huggingface.co\/Qwen\/Qwen2.5-Coder-14B Accessed: 2025-03-09."},{"key":"e_1_2_1_2_1","unstructured":"Explosion AI. 2015. spaCy: Industrial-Strength Natural Language Processing in Python. https:\/\/spacy.io\/. Accessed: 2025-08-06."},{"key":"e_1_2_1_3_1","volume-title":"ANTLR: ANother Tool for Language Recognition. https:\/\/www.antlr.org\/. Accessed: 2025-06-05.","author":"Project ANTLR","year":"2025","unstructured":"ANTLR Project. 2025. ANTLR: ANother Tool for Language Recognition. https:\/\/www.antlr.org\/. Accessed: 2025-06-05."},{"key":"e_1_2_1_4_1","volume-title":"Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543","author":"Chen Tsong Y","year":"2020","unstructured":"Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543 (2020)."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.07.037"},{"key":"e_1_2_1_6_1","volume-title":"Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, et al.","author":"Ding Hantian","year":"2023","unstructured":"Hantian Ding, Varun Kumar, Yuchen Tian, Zijian Wang, Rob Kwiatkowski, Xiaopeng Li, Murali Krishna Ramanathan, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, et al. 2023. A static evaluation of code completion by large language models. arXiv preprint arXiv:2306.03203 (2023)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3728902"},{"key":"e_1_2_1_8_1","volume-title":"Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363","author":"Gao Dawei","year":"2023","unstructured":"Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363 (2023)."},{"key":"e_1_2_1_9_1","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Yu Wu YK Li et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming-The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3703155"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383123"},{"key":"e_1_2_1_12_1","volume-title":"A comprehensive exploration on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01069","author":"Hwang Wonseok","year":"2019","unstructured":"Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01069 (2019)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST60714.2024.00019"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3571730"},{"key":"e_1_2_1_15_1","volume-title":"Siriusbi: Building end-to-end business intelligence enhanced by large language models. arXiv preprint arXiv:2411.06102","author":"Jiang Jie","year":"2024","unstructured":"Jie Jiang, Haining Xie, Yu Shen, Zihan Zhang, Meng Lei, Yifeng Zheng, Yide Fang, Chunyou Li, Danqing Huang, Wentao Zhang, et al. 2024. Siriusbi: Building end-to-end business intelligence enhanced by large language models. arXiv preprint arXiv:2411.06102 (2024)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555041.3589678"},{"key":"e_1_2_1_17_1","volume-title":"Experts don't cheat: learning what you don't know by predicting pairs. arXiv preprint arXiv:2402.08733","author":"Johnson Daniel D","year":"2024","unstructured":"Daniel D Johnson, Daniel Tarlow, David Duvenaud, and Chris J Maddison. 2024. Experts don't cheat: learning what you don't know by predicting pairs. arXiv preprint arXiv:2402.08733 (2024)."},{"key":"e_1_2_1_18_1","volume-title":"University of Calgary 23","author":"Khandkar S. H.","year":"2009","unstructured":"S. H. Khandkar. 2009. Open Coding. University of Calgary 23, 2009 (2009)."},{"key":"e_1_2_1_19_1","volume-title":"Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520","author":"Kocmi Tom","year":"2023","unstructured":"Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520 (2023)."},{"key":"e_1_2_1_20_1","volume-title":"Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664","author":"Kuhn Lorenz","year":"2023","unstructured":"Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 (2023)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3656340"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-short.118"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-1835"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3689776"},{"key":"e_1_2_1_25_1","volume-title":"ChatBI: Towards Natural Language to Complex Business Intelligence SQL. arXiv preprint arXiv:2405.00527","author":"Lian Jinqing","year":"2024","unstructured":"Jinqing Lian, Xinyi Liu, Yingxia Shao, Yang Dong, Ming Wang, Zhang Wei, Tianqi Wan, Ming Dong, and Hailin Yan. 2024. ChatBI: Towards Natural Language to Complex Business Intelligence SQL. arXiv preprint arXiv:2405.00527 (2024)."},{"key":"e_1_2_1_26_1","volume-title":"Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334","author":"Lin Stephanie","year":"2022","unstructured":"Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334 (2022)."},{"key":"e_1_2_1_27_1","volume-title":"Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. arXiv preprint arXiv:2012.12627","author":"Lin Xi Victoria","year":"2020","unstructured":"Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. arXiv preprint arXiv:2012.12627 (2020)."},{"key":"e_1_2_1_28_1","volume-title":"Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971","author":"Liu Fang","year":"2024","unstructured":"Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971 (2024)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/64"},{"key":"e_1_2_1_30_1","volume-title":"Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896","author":"Manakul Potsawee","year":"2023","unstructured":"Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896 (2023)."},{"key":"e_1_2_1_31_1","unstructured":"Toby Mao. 2022. SQLGlot: An open source SQL parser transpiler and optimizer. https:\/\/github.com\/tobymao\/sqlglot. Accessed: 2025-07-07."},{"key":"e_1_2_1_32_1","volume-title":"On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661","author":"Maynez Joshua","year":"2020","unstructured":"Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661 (2020)."},{"key":"e_1_2_1_33_1","unstructured":"MRSQLGen Authors. 2026. Detailed HKB Construction and MR Rules Usage for MRSQLGen. https: \/\/github.com\/MRSQLGen\/FSE26-Rebuttal-Material\/blob\/main\/Details%20of%20HKB%20Construction%20and%"},{"key":"e_1_2_1_34_1","unstructured":"md. Supplementary material."},{"key":"e_1_2_1_35_1","unstructured":"MRSQLGen Authors. 2026. Failure Case Analysis for MRSQLGen. https:\/\/github.com\/MRSQLGen\/FSE26-Rebuttal- Material\/blob\/main\/Failure%20Cases%20Analysis.md. Supplementary material."},{"key":"e_1_2_1_36_1","volume-title":"Clarifygpt: Empowering llm-based code generation with intention clarification. arXiv preprint arXiv:2310.10996","author":"Mu Fangwen","year":"2023","unstructured":"Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang, Shichao Liu, and Qing Wang. 2023. Clarifygpt: Empowering llm-based code generation with intention clarification. arXiv preprint arXiv:2310.10996 (2023)."},{"key":"e_1_2_1_37_1","unstructured":"OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https:\/\/openai.com\/blog\/chatgpt. Accessed: 2025-03-09."},{"key":"e_1_2_1_38_1","doi-asserted-by":"crossref","unstructured":"Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022) 27730-27744.","DOI":"10.52202\/068431-2011"},{"key":"e_1_2_1_39_1","volume-title":"Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in neural information processing systems 36","author":"Pourreza Mohammadreza","year":"2023","unstructured":"Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in neural information processing systems 36 (2023), 36339-36348."},{"key":"e_1_2_1_40_1","volume-title":"An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of llms. arXiv preprint arXiv:2406.12288","author":"Rai Daking","year":"2024","unstructured":"Daking Rai and Ziyu Yao. 2024. An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of llms. arXiv preprint arXiv:2406.12288 (2024)."},{"key":"e_1_2_1_41_1","volume-title":"Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al.","author":"Roziere Baptiste","year":"2023","unstructured":"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2764464"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3320248"},{"key":"e_1_2_1_45_1","unstructured":"Ezekiel J. Smith. 2021. SQLGlot: An Open Source Python SQL Parser Transpiler and Engine. https:\/\/sqlglot.com\/ sqlglot.html. Accessed: 2025-08-06."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639477.3639732"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2934848"},{"key":"e_1_2_1_48_1","volume-title":"Sql-palm: Improved large language model adaptation for text-to-sql (extended). arXiv preprint arXiv:2306.00739","author":"Sun Ruoxi","year":"2023","unstructured":"Ruoxi Sun, Sercan \u00d6 Arik, Alex Muzio, Lesly Miculicich, Satya Gundabathula, Pengcheng Yin, Hanjun Dai, Hootan Nakhost, Rajarishi Sinha, Zifeng Wang, et al. 2023. Sql-palm: Improved large language model adaptation for text-to-sql (extended). arXiv preprint arXiv:2306.00739 (2023)."},{"key":"e_1_2_1_49_1","volume-title":"Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755","author":"Talaei Shayan","year":"2024","unstructured":"Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755 (2024)."},{"key":"e_1_2_1_50_1","unstructured":"Mistral AI Team. 2023. Mistral-7B-Instruct-v0.2. https:\/\/huggingface.co\/mistralai\/Mistral-7B-Instruct-v0.2. Accessed: 2025-08-14."},{"key":"e_1_2_1_51_1","volume-title":"2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 678-682","author":"Tsigkanos Christos","year":"2023","unstructured":"Christos Tsigkanos, Pooja Rani, Sebastian M\u00fcller, and Timo Kehrer. 2023. Large language models: The next frontier for variable discovery within metamorphic testing?. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 678-682."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_2_1_53_1","volume-title":"A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987","author":"Varshney Neeraj","year":"2023","unstructured":"Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987 (2023)."},{"key":"e_1_2_1_54_1","volume-title":"Mac-sql: A multi-agent collaborative framework for text-to-sql. arXiv preprint arXiv:2312.11242","author":"Wang Bing","year":"2023","unstructured":"Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, et al. 2023. Mac-sql: A multi-agent collaborative framework for text-to-sql. arXiv preprint arXiv:2312.11242 (2023)."},{"key":"e_1_2_1_55_1","volume-title":"Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048","author":"Wang Jiaan","year":"2023","unstructured":"Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048 (2023)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3728963"},{"key":"e_1_2_1_57_1","volume-title":"Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz.","author":"Wang Tianlu","year":"2023","unstructured":"Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04592 (2023)."},{"key":"e_1_2_1_58_1","volume-title":"Validating LLM-Generated Programs with Metamorphic Prompt Testing. arXiv preprint arXiv:2406.06864","author":"Wang Xiaoyin","year":"2024","unstructured":"Xiaoyin Wang and Dakai Zhu. 2024. Validating LLM-Generated Programs with Metamorphic Prompt Testing. arXiv preprint arXiv:2406.06864 (2024)."},{"key":"e_1_2_1_59_1","volume-title":"Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087","author":"Wang Yidong","year":"2023","unstructured":"Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087 (2023)."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3715784"},{"key":"e_1_2_1_61_1","volume-title":"Language Models can Evaluate Themselves via Probability Discrepancy. arXiv preprint arXiv:2405.10516","author":"Xia Tingyu","year":"2024","unstructured":"Tingyu Xia, Bowen Yu, Yuan Wu, Yi Chang, and Chang Zhou. 2024. Language Models can Evaluate Themselves via Probability Discrepancy. arXiv preprint arXiv:2405.10516 (2024)."},{"key":"e_1_2_1_62_1","volume-title":"Decomposition for enhancing attention: Improving llm-based text-to-sql through workflow paradigm. arXiv preprint arXiv:2402.10671","author":"Xie Yuanzhen","year":"2024","unstructured":"Yuanzhen Xie, Xinzhou Jin, Tao Xie, MingXiong Lin, Liang Chen, Chenyun Yu, Lei Cheng, ChengXiang Zhuo, Bo Hu, and Zang Li. 2024. Decomposition for enhancing attention: Improving llm-based text-to-sql through workflow paradigm. arXiv preprint arXiv:2402.10671 (2024)."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3656340"},{"key":"e_1_2_1_64_1","volume-title":"Exploring and Lifting the Robustness of LLM-powered Automated Program Repair with Metamorphic Testing. arXiv preprint arXiv:2410.07516","author":"Xue Pengyu","year":"2024","unstructured":"Pengyu Xue, Linhao Wu, Zhen Yang, Xinyi Li, Zhongxing Yu, Zhi Jin, Ge Li, Yan Xiao, and Jingwen Wu. 2024. Exploring and Lifting the Robustness of LLM-powered Automated Program Repair with Metamorphic Testing. arXiv preprint arXiv:2410.07516 (2024)."},{"key":"e_1_2_1_65_1","doi-asserted-by":"crossref","first-page":"58077","DOI":"10.52202\/079017-1852","article-title":"To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty","volume":"37","author":"Yadkori Yasin A","year":"2024","unstructured":"Yasin A Yadkori, Ilja Kuzborskij, Andr\u00e1s Gy\u00f6rgy, and Csaba Szepesvari. 2024. To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty. Advances in Neural Information Processing Systems 37 (2024), 58077-58117.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3715735"},{"key":"e_1_2_1_67_1","first-page":"1696","volume-title":"Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society","volume":"7","author":"Yang Joshua C","year":"2024","unstructured":"Joshua C Yang, Damian Dailisan, Marcin Korecki, Carina I Hausladen, and Dirk Helbing. 2024. Llm voting: Human choices and ai collective decision-making. In Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, Vol. 7. 1696-1708."},{"key":"e_1_2_1_68_1","volume-title":"Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469","author":"Yao Jia-Yu","year":"2023","unstructured":"Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023)."},{"key":"e_1_2_1_69_1","volume-title":"Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887","author":"Yu Tao","year":"2018","unstructured":"Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887 (2018)."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3728894"},{"key":"e_1_2_1_71_1","doi-asserted-by":"crossref","unstructured":"Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36 (2023) 46595-46623.","DOI":"10.52202\/075280-2020"},{"key":"e_1_2_1_72_1","volume-title":"Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks. arXiv preprint arXiv:2408.15207","author":"Zhou Shide","year":"2024","unstructured":"Shide Zhou, Tianlin Li, Kailong Wang, Yihao Huang, Ling Shi, Yang Liu, and Haoyu Wang. 2024. Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks. arXiv preprint arXiv:2408.15207 (2024)."},{"key":"e_1_2_1_73_1","first-page":"346","volume-title":"Proceedings of the 8th International Symposium on Future Software Technology (ISFST","author":"Zhou Zhi Quan","year":"2004","unstructured":"Zhi Quan Zhou, DH Huang, TH Tse, Zongyuan Yang, Haitao Huang, and TY Chen. 2004. Metamorphic testing and its applications. In Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004). Software Engineers Association Xian, China, 346-351."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3797146","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:00:32Z","timestamp":1782846032000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3797146"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":74,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3797146"],"URL":"https:\/\/doi.org\/10.1145\/3797146","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}