{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T09:19:44Z","timestamp":1773825584574,"version":"3.50.1"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,4,16]],"date-time":"2025-04-16T00:00:00Z","timestamp":1744761600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2025,4,30]]},"abstract":"<jats:p>Tabular Representation Learning (TRL) and Large Language Models (LLMs) have become established for tackling Question Answering (QA) and Semantic Parsing (SP) tasks on tabular data. State-of-the-art models are pre-trained and evaluated on large open-domain datasets. However, the performance on existing QA and SP benchmarks is not necessarily representative of that achieved on proprietary data as the characteristics of the input and the complexity of the posed queries show high variability. To tackle this challenge, our goal is to allow end-users to evaluate TRL and LLM performance on their own proprietary data. We present Query-Aided TRL CHecklist (QATCH), a toolbox to automatically generate a testing checklist tailored to QA and SP. QATCH provides a testing suite highlighting models\u2019 strengths and weaknesses on relational tables unseen at training time. The proposed toolbox relies on a SQL query generator that crafts tests of varying types and complexity including, amongst others, tests on null values, projection, selections, joins, group by, and having clauses. QATCH also supports a set of general cross-task performance metrics providing more insights into SQL-related model capabilities than currently used metrics. The empirical results, achieved by state-of-the-art TRL models and LLMs, show substantial performance differences (1) between existing benchmarks and proprietary data, (2) across queries of different complexity.<\/jats:p>","DOI":"10.1145\/3712704","type":"journal-article","created":{"date-parts":[[2025,1,21]],"date-time":"2025-01-21T05:26:44Z","timestamp":1737437204000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["QATCH: Automatic Evaluation of SQL-Centric Tasks on Proprietary Data"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-5361-0042","authenticated-orcid":false,"given":"Simone","family":"Papicchio","sequence":"first","affiliation":[{"name":"Dipartimento di Automatica e Informatica, Politecnico di Torino, Turin, Italyand EURECOM, Sophia Antipolis, Alpes | Maritimes, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0651-4128","authenticated-orcid":false,"given":"Paolo","family":"Papotti","sequence":"additional","affiliation":[{"name":"EURECOM, Sophia Antipolis, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7185-5247","authenticated-orcid":false,"given":"Luca","family":"Cagliero","sequence":"additional","affiliation":[{"name":"Dipartimento di Automatica e Informatica, Politecnico di Torino, Turin, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,4,16]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00544"},{"key":"e_1_3_2_3_2","first-page":"1","article-title":"The need for tabular representation learning: An industry perspective","author":"Cahoon Joyce","year":"2022","unstructured":"Joyce Cahoon, Alexandra Savelieva, Andreas C. Mueller, Avrilia Floratou, Carlo Curino, Hiren Patel, Jordan Henkel, Markus Weimer, Nellie Gustafsson, Richard Wydrowski, et\u00a0al. 2022. The need for tabular representation learning: An industry perspective. In NeurIPS 2022 First Table Representation Workshop, 1\u20132.","journal-title":"NeurIPS 2022 First Table Representation Workshop"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/800105.803397"},{"key":"e_1_3_2_5_2","first-page":"1","volume-title":"the 11th International Conference on Learning Representations","author":"Chang Shuaichen","year":"2023","unstructured":"Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, et\u00a0al. 2023. Dr.Spider: A diagnostic evaluation benchmark towards text-to-SQL robustness. In the 11th International Conference on Learning Representations, 1\u201326. Retrieved from https:\/\/openreview.net\/forum?id=Wc5bmZZU9cy"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"e_1_3_2_7_2","unstructured":"Wenhu Chen Hongmin Wang Jianshu Chen Yunkai Zhang Hong Wang Shiyang Li Xiyou Zhou and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. arXiv:1909.02164. Retrieved from https:\/\/arxiv.org\/abs\/1909.02164"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00716"},{"key":"e_1_3_2_9_2","unstructured":"Xi Fang Weijie Xu Fiona Anting Tan Jiani Zhang Ziqing Hu Yanjun Qi Scott Nickleach Diego Socolinsky Srinivasan Sengamedu and Christos Faloutsos. 2024. Large language models on tabular data\u2013A survey. arXiv:2402.17944. Retrieved from https:\/\/arxiv.org\/abs\/2402.17944"},{"key":"e_1_3_2_10_2","first-page":"1","volume-title":"Conference on Innovative Data Systems Research","author":"Floratou Avrilia","year":"2024","unstructured":"Avrilia Floratou, Fotis Psallidas, Fuheng Zhao, Shaleen Deep, Gunther Hagleither, Wangda Tan, Joyce Cahoon, Rana Alotaibi, Jordan Henkel, Abhik Singla, et\u00a0al. 2024. NL2SQL is a solved problem\u2026 Not! In Conference on Innovative Data Systems Research, 1\u20138. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:266729311"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.195"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.702"},{"key":"e_1_3_2_13_2","unstructured":"Dawei Gao Haibin Wang Yaliang Li Xiuyu Sun Yichen Qian Bolin Ding and Jingren Zhou. 2023. Text-to-SQL empowered by large language models: A benchmark evaluation. arXiv:2308.15363. Retrieved from https:\/\/arxiv.org\/abs\/2308.15363"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.48786\/EDBT.2024.40"},{"key":"e_1_3_2_15_2","first-page":"13471","volume-title":"2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING \u201924)","author":"Grijalba Jorge Os\u00e9s","year":"2024","unstructured":"Jorge Os\u00e9s Grijalba, L. Alfonso Urena Lopez, Eugenio Mart\u00ednez-C\u00e1mara, and Jose Camacho-Collados. 2024. Question answering over tabular data with DataBench: A large-scale empirical evaluation of LLMs. In 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING \u201924), 13471\u201313488."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1188"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.398"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.68"},{"key":"e_1_3_2_19_2","first-page":"1","volume-title":"the 12th International Conference on Learning Representations","author":"Kong Kezhi","year":"2024","unstructured":"Kezhi Kong, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Chuan Lei, Christos Faloutsos, Huzefa Rangwala, and George Karypis. 2024. OpenTab: Advancing large language models as open-domain table reasoners. In the 12th International Conference on Learning Representations, 1\u201317. Retrieved from https:\/\/openreview.net\/forum?id=Qa0ULgosc9"},{"key":"e_1_3_2_20_2","first-page":"8285","article-title":"Open-WikiTable: Dataset for open domain question answering with complex reasoning over table","author":"Kweon Sunjun","year":"2023","unstructured":"Sunjun Kweon, Yeonsu Kwon, Seonhee Cho, Yohan Jo, and Edward Choi. 2023. Open-WikiTable: Dataset for open domain question answering with complex reasoning over table. In Annual Meeting of the Association for Computational Linguistics, 8285\u20138297. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:258676308","journal-title":"Annual Meeting of the Association for Computational Linguistics"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.176"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i11.26535"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i11.26536"},{"key":"e_1_3_2_24_2","volume-title":"37th International Conference on Neural Information Processing Systems (NIPS \u201923)","author":"Li Jinyang","year":"2024","unstructured":"Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et\u00a0al. 2024. Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs. In 37th International Conference on Neural Information Processing Systems (NIPS \u201923). Curran Associates Inc., Red Hook, NY, Article 1835, 28 pages."},{"key":"e_1_3_2_25_2","first-page":"1","volume-title":"International Conference on Learning Representations","author":"Liu Qian","year":"2021","unstructured":"Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2021. TAPEX: Table pre-training via learning a neural SQL executor. In International Conference on Learning Representations, 1\u201319."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.14778\/3494124.3494139"},{"key":"e_1_3_2_27_2","unstructured":"OpenAI. 2023. ChatGPT 3.5. Retrieved from https:\/\/openai.com\/blog\/chatgpt"},{"key":"e_1_3_2_28_2","first-page":"30898","volume-title":"37th Conference on Neural Information Processing Systems Datasets and Benchmarks Track","author":"Papicchio Simone","year":"2023","unstructured":"Simone Papicchio, Paolo Papotti, and Luca Cagliero. 2023. QATCH: Benchmarking SQL-centric tasks with table representation learning models on your data. In 37th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 30898\u201330917."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDEW61823.2024.00050"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1142"},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","unstructured":"Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. arXiv:1508.00305. Retrieved from https:\/\/arxiv.org\/abs\/1508.00305","DOI":"10.3115\/v1\/P15-1142"},{"key":"e_1_3_2_32_2","first-page":"36339","article-title":"DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction","author":"Pourreza Mohammadreza","year":"2024","unstructured":"Mohammadreza Pourreza and Davood Rafiei. 2024. DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction. In Advances in Neural Information Processing Systems, Vol. 36, 36339\u201336348.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_33_2","first-page":"140","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1\u201367. Retrieved from http:\/\/jmlr.org\/papers\/v21\/20-074.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_34_2","unstructured":"Baptiste Roziere Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin et\u00a0al. 2023. Code Llama: Open foundation models for code. arXiv:2308.12950. Retrieved from https:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.722"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i15.17627"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.560"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3616855.3635752"},{"key":"e_1_3_2_39_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et\u00a0al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE55515.2023.00041"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2629489"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380589"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.707"},{"key":"e_1_3_2_44_2","unstructured":"Tianbao Xie Chen Henry Wu Peng Shi Ruiqi Zhong Torsten Scholak Michihiro Yasunaga Chien-Sheng Wu Ming Zhong Pengcheng Yin Sida I. Wang et\u00a0al. 2022. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv:2201.05966. Retrieved from https:\/\/arxiv.org\/abs\/2201.05966"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1425"},{"key":"e_1_3_2_46_2","unstructured":"Victor Zhong Caiming Xiong and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv: 1709.00103. Retrieved from https:\/\/arxiv.org\/abs\/1709.00103"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3712704","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3712704","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:09Z","timestamp":1750295889000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3712704"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,16]]},"references-count":45,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,4,30]]}},"alternative-id":["10.1145\/3712704"],"URL":"https:\/\/doi.org\/10.1145\/3712704","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,16]]},"assertion":[{"value":"2024-08-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}