{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T15:36:27Z","timestamp":1772033787527,"version":"3.50.1"},"reference-count":68,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,10,2]],"date-time":"2023-10-02T00:00:00Z","timestamp":1696204800000},"content-version":"vor","delay-in-days":274,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,9,29]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Sequence generation models are increasingly being used to translate natural language into programs, i.e., to perform executable semantic parsing. The fact that semantic parsing aims to predict programs that can lead to executed actions in the real world motivates developing safe systems. This in turn makes measuring calibration\u2014a central component to safety\u2014particularly important. We investigate the calibration of popular generation models across four popular semantic parsing datasets, finding that it varies across models and datasets. We then analyze factors associated with calibration error and release new confidence-based challenge splits of two parsing datasets. To facilitate the inclusion of calibration in semantic parsing evaluations, we release a library for computing calibration metrics.1<\/jats:p>","DOI":"10.1162\/tacl_a_00598","type":"journal-article","created":{"date-parts":[[2023,10,2]],"date-time":"2023-10-02T15:13:22Z","timestamp":1696259602000},"page":"1213-1231","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["Calibrated Interpretation: Confidence Estimation in Semantic Parsing"],"prefix":"10.1162","volume":"11","author":[{"given":"Elias","family":"Stengel-Eskin","sequence":"first","affiliation":[{"name":"Johns Hopkins University, USA. elias@jhu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Van Durme","sequence":"additional","affiliation":[{"name":"Johns Hopkins University, USA. vandurme@jhu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,9,29]]},"reference":[{"key":"2023100215131002800_bib1","doi-asserted-by":"publisher","first-page":"2260","DOI":"10.1145\/3477495.3531841","article-title":"Modern baselines for sparql semantic parsing","volume-title":"Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Banerjee","year":"2022"},{"key":"2023100215131002800_bib2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023100215131002800_bib3","article-title":"Density-softmax: Scalable and distance-aware uncertainty estimation under distribution shifts","author":"Ha","year":"2023","journal-title":"arXiv preprint arXiv:2302.06495"},{"key":"2023100215131002800_bib4","first-page":"249","article-title":"Re-evaluating the role of BLEU in machine translation research","volume-title":"11th Conference of the European Chapter of the Association for Computational Linguistics","author":"Callison-Burch","year":"2006"},{"key":"2023100215131002800_bib5","article-title":"The calibration generalization gap","author":"Carrell","year":"2022","journal-title":"arXiv preprint arXiv:2210.01964"},{"key":"2023100215131002800_bib6","article-title":"Evaluating large language models trained on code","author":"Chen","year":"2021","journal-title":"arXiv preprint arXiv:2107.03374"},{"key":"2023100215131002800_bib7","article-title":"Error detection for interactive text-to-sql semantic parsing","volume-title":"Proceedings of the Second Workshop on Interactive Learning for Natural Language Processing","author":"Chen","year":"2022"},{"key":"2023100215131002800_bib8","doi-asserted-by":"publisher","first-page":"8107","DOI":"10.18653\/v1\/2020.emnlp-main.651","article-title":"Conversational semantic parsing for dialog state tracking","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Cheng","year":"2020"},{"key":"2023100215131002800_bib9","doi-asserted-by":"publisher","first-page":"57","DOI":"10.18653\/v1\/2021.findings-acl.5","article-title":"Diagnosing transformers in task-oriented semantic parsing","volume-title":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","author":"Desai","year":"2021"},{"key":"2023100215131002800_bib10","doi-asserted-by":"publisher","first-page":"295","DOI":"10.18653\/v1\/2020.emnlp-main.21","article-title":"Calibration of pre-trained transformers","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Desai","year":"2020"},{"key":"2023100215131002800_bib11","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/CVPRW50498.2020.00010","article-title":"Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops","author":"Ding","year":"2020"},{"key":"2023100215131002800_bib12","doi-asserted-by":"publisher","first-page":"743","DOI":"10.18653\/v1\/P18-1069","article-title":"Confidence modeling for neural semantic parsing","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Li","year":"2018"},{"key":"2023100215131002800_bib13","article-title":"Deep biaffine attention for neural dependency parsing","volume-title":"5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24\u201326, 2017, Conference Track Proceedings","author":"Dozat","year":"2017"},{"key":"2023100215131002800_bib14","first-page":"1321","article-title":"On calibration of modern neural networks","volume-title":"International Conference on Machine Learning","author":"Guo","year":"2017"},{"key":"2023100215131002800_bib15","doi-asserted-by":"publisher","first-page":"2787","DOI":"10.18653\/v1\/D18-1300","article-title":"Semantic parsing for task oriented dialog using hierarchical representations","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Gupta","year":"2018"},{"key":"2023100215131002800_bib16","article-title":"Measuring massive multitask language understanding","volume-title":"International Conference on Learning Representations","author":"Hendrycks","year":"2021"},{"key":"2023100215131002800_bib17","article-title":"A baseline for detecting misclassified and out-of-distribution examples in neural networks","author":"Hendrycks","year":"2016","journal-title":"arXiv preprint arXiv:1610.02136"},{"issue":"8","key":"2023100215131002800_bib18","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2023100215131002800_bib19","doi-asserted-by":"publisher","first-page":"2078","DOI":"10.18653\/v1\/2020.acl-main.188","article-title":"Calibrating structured output predictors for natural language processing","volume-title":"Proceedings of the Association for Computational Linguistics Meeting","author":"Jagannatha","year":"2020"},{"key":"2023100215131002800_bib20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1147","article-title":"Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","author":"Joshi","year":"2017"},{"key":"2023100215131002800_bib21","article-title":"Language models (mostly) know what they know","author":"Kadavath","year":"2022","journal-title":"arXiv preprint arXiv:2207.05221"},{"key":"2023100215131002800_bib22","doi-asserted-by":"crossref","first-page":"582","DOI":"10.18653\/v1\/2022.sigdial-1.54","article-title":"Evaluating n-best calibration of natural language understanding for dialogue systems","volume-title":"Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue","author":"Khojah","year":"2022"},{"key":"2023100215131002800_bib23","article-title":"Adam: A method for stochastic optimization","volume-title":"3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, Conference Track Proceedings","author":"Kingma","year":"2015"},{"key":"2023100215131002800_bib24","article-title":"Calibration of encoder decoder models for neural machine translation","author":"Kumar","year":"2019","journal-title":"arXiv preprint arXiv:1903.00802"},{"key":"2023100215131002800_bib25","doi-asserted-by":"publisher","first-page":"7871","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Lewis","year":"2020"},{"key":"2023100215131002800_bib26","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01464","article-title":"Calibrating concepts and operations: Towards symbolic reasoning on real images","volume-title":"Proceedings of the IEEE\/ CVF International Conference on Computer Vision (ICCV)","author":"Li","year":"2021"},{"key":"2023100215131002800_bib27","article-title":"Teaching models to express their uncertainty in words","author":"Lin","year":"2022","journal-title":"Transactions on Machine Learning Research"},{"key":"2023100215131002800_bib28","article-title":"Roberta: A robustly optimized bert pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"2023100215131002800_bib29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.448","article-title":"Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics","author":"Mathur","year":"2020","journal-title":"arXiv preprint arXiv:2006.06264"},{"key":"2023100215131002800_bib30","article-title":"Augmented language models: A survey","author":"Mialon","year":"2023","journal-title":"arXiv preprint arXiv:2302.07842"},{"key":"2023100215131002800_bib31","doi-asserted-by":"publisher","first-page":"857","DOI":"10.1162\/tacl_a_00494","article-title":"Reducing conversational agents\u2019 overconfidence through linguistic calibration","volume":"10","author":"Mielke","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023100215131002800_bib32","first-page":"15682","article-title":"Revisiting the calibration of modern neural networks","volume":"34","author":"Minderer","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023100215131002800_bib33","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v29i1.9602","article-title":"Obtaining well calibrated probabilities using bayesian binning","volume-title":"Twenty-Ninth AAAI Conference on Artificial Intelligence","author":"Naeini","year":"2015"},{"key":"2023100215131002800_bib34","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1145\/1102351.1102430","article-title":"Predicting good probabilities with supervised learning","volume-title":"Proceedings of the 22nd International Conference on Machine Learning","author":"Niculescu-Mizil","year":"2005"},{"key":"2023100215131002800_bib35","article-title":"Codegen: An open large language model for code with multi-turn program synthesis","author":"Nijkamp","year":"2022","journal-title":"arXiv preprint arXiv:2203.13474"},{"key":"2023100215131002800_bib36","article-title":"Can you trust your model\u2019s uncertainty? Evaluating predictive uncertainty under dataset shift","volume":"32","author":"Ovadia","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023100215131002800_bib37","doi-asserted-by":"publisher","first-page":"3666","DOI":"10.18653\/v1\/2021.acl-long.284","article-title":"Value-agnostic conversational semantic parsing","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Platanios","year":"2021"},{"issue":"140","key":"2023100215131002800_bib38","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"issue":"4","key":"2023100215131002800_bib39","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1561\/1500000019","article-title":"The probabilistic relevance framework: Bm25 and beyond","volume":"3","author":"Robertson","year":"2009","journal-title":"Foundations and Trends\u00ae in Information Retrieval"},{"key":"2023100215131002800_bib40","article-title":"Benchclamp: A benchmark for evaluating language models on semantic parsing","author":"Roy","year":"2022","journal-title":"arXiv preprint arXiv: 2206.10668"},{"key":"2023100215131002800_bib41","article-title":"Toolformer: Language models can teach themselves to use tools","author":"Schick","year":"2023","journal-title":"arXiv preprint arXiv:2302.04761"},{"key":"2023100215131002800_bib42","doi-asserted-by":"publisher","first-page":"9895","DOI":"10.18653\/v1\/2021.emnlp-main.779","article-title":"PICARD: Parsing incrementally for constrained auto-regressive decoding from language models","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Scholak","year":"2021"},{"key":"2023100215131002800_bib43","doi-asserted-by":"publisher","first-page":"556","DOI":"10.1162\/tacl_a_00333","article-title":"Task-oriented dialogue as dataflow synthesis","volume":"8","author":"Machines","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023100215131002800_bib44","doi-asserted-by":"publisher","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural machine translation of rare words with subword units","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"2023100215131002800_bib45","doi-asserted-by":"publisher","first-page":"922","DOI":"10.18653\/v1\/2021.acl-long.75","article-title":"Compositional generalization and natural language variation: Can a semantic parsing approach handle both?","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Shaw","year":"2021"},{"key":"2023100215131002800_bib46","doi-asserted-by":"publisher","first-page":"7699","DOI":"10.18653\/v1\/2021.emnlp-main.608","article-title":"Constrained language models yield few-shot semantic parsers","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Shin","year":"2021"},{"key":"2023100215131002800_bib47","doi-asserted-by":"publisher","first-page":"5417","DOI":"10.18653\/v1\/2022.naacl-main.396","article-title":"Few-shot semantic parsing with language models trained on code","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Shin","year":"2022"},{"key":"2023100215131002800_bib48","article-title":"Prompting GPT-3 to be reliable","author":"Si","year":"2023","journal-title":"International Conference on Learning Representations"},{"key":"2023100215131002800_bib49","doi-asserted-by":"crossref","first-page":"2814","DOI":"10.18653\/v1\/2022.findings-emnlp.204","article-title":"Re-examining calibration: The case of question answering","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022","author":"Si","year":"2022"},{"key":"2023100215131002800_bib50","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00396","article-title":"Joint universal syntactic and semantic parsing","author":"Stengel-Eskin","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023100215131002800_bib51","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.789","article-title":"When more data hurts: A troubling quirk in developing broad-coverage natural language understanding systems","author":"Stengel-Eskin","year":"2022","journal-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)"},{"key":"2023100215131002800_bib52","doi-asserted-by":"publisher","first-page":"8427","DOI":"10.18653\/v1\/2020.acl-main.746","article-title":"Universal decompositional semantic parsing","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Stengel-Eskin","year":"2020"},{"key":"2023100215131002800_bib53","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1146\/annurev-control-101119-071628","article-title":"Robots that use language","volume":"3","author":"Tellex","year":"2020","journal-title":"Annual Review of Control, Robotics, and Autonomous Systems"},{"key":"2023100215131002800_bib54","doi-asserted-by":"publisher","first-page":"1995","DOI":"10.18653\/v1\/2022.findings-acl.158","article-title":"Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Varshney","year":"2022"},{"key":"2023100215131002800_bib55","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\u20139, 2017, Long Beach, CA, USA","author":"Vaswani","year":"2017"},{"key":"2023100215131002800_bib56","doi-asserted-by":"publisher","first-page":"3070","DOI":"10.18653\/v1\/2020.acl-main.278","article-title":"On the inference calibration of neural machine translation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Wang","year":"2020"},{"key":"2023100215131002800_bib57","doi-asserted-by":"publisher","first-page":"8696","DOI":"10.18653\/v1\/2021.emnlp-main.685","article-title":"Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Wang","year":"2021"},{"key":"2023100215131002800_bib58","article-title":"Inverse scaling can become u-shaped","author":"Wei","year":"2022","journal-title":"arXiv preprint arXiv:2211.02011"},{"key":"2023100215131002800_bib59","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 conference on empirical methods in natural language processing: System demonstrations","author":"Wolf","year":"2020"},{"key":"2023100215131002800_bib60","doi-asserted-by":"publisher","first-page":"1962","DOI":"10.18653\/v1\/D19-1204","article-title":"Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Yu","year":"2019"},{"key":"2023100215131002800_bib61","doi-asserted-by":"publisher","first-page":"3911","DOI":"10.18653\/v1\/D18-1425","article-title":"Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Yu","year":"2018"},{"key":"2023100215131002800_bib62","doi-asserted-by":"publisher","first-page":"694","DOI":"10.1145\/775047.775151","article-title":"Transforming classifier scores into accurate multiclass probability estimates","volume-title":"Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Zadrozny","year":"2002"},{"key":"2023100215131002800_bib63","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1009","article-title":"Swag: A large-scale adversarial dataset for grounded commonsense inference","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Zellers","year":"2018"},{"key":"2023100215131002800_bib64","unstructured":"Sheng Zhang . 2020. Transductive Semantic Parsing. Ph.D. thesis, The Johns Hopkins University."},{"key":"2023100215131002800_bib65","doi-asserted-by":"publisher","first-page":"80","DOI":"10.18653\/v1\/P19-1009","article-title":"AMR parsing as sequence-to-graph transduction","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Zhang","year":"2019"},{"key":"2023100215131002800_bib66","doi-asserted-by":"publisher","first-page":"3786","DOI":"10.18653\/v1\/D19-1392","article-title":"Broad-coverage semantic parsing as transduction","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Zhang","year":"2019"},{"key":"2023100215131002800_bib67","doi-asserted-by":"publisher","first-page":"396","DOI":"10.18653\/v1\/2020.emnlp-main.29","article-title":"Semantic evaluation for text-to-sql with distilled test suites","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Zhong","year":"2020"},{"key":"2023100215131002800_bib68","article-title":"Navigating the grey area: Expressions of overconfidence and uncertainty in language models","author":"Zhou","year":"2023","journal-title":"arXiv preprint arXiv: 2302.13439"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00598\/2161218\/tacl_a_00598.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00598\/2161218\/tacl_a_00598.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,23]],"date-time":"2023-12-23T06:29:09Z","timestamp":1703312949000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00598\/117737\/Calibrated-Interpretation-Confidence-Estimation-in"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":68,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00598","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}