{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T10:04:26Z","timestamp":1768644266645,"version":"3.49.0"},"reference-count":55,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2022,6,15]],"date-time":"2022-06-15T00:00:00Z","timestamp":1655251200000},"content-version":"vor","delay-in-days":165,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,15]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>State-of-the-art classification and regression models are often not well calibrated, and cannot reliably provide uncertainty estimates, limiting their utility in safety-critical applications such as clinical decision-making. While recent work has focused on calibration of classifiers, there is almost no work in NLP on calibration in a regression setting. In this paper, we quantify the calibration of pre- trained language models for text regression, both intrinsically and extrinsically. We further apply uncertainty estimates to augment training data in low-resource domains. Our experiments on three regression tasks in both self-training and active-learning settings show that uncertainty estimation can be used to increase overall performance and enhance model generalization.<\/jats:p>","DOI":"10.1162\/tacl_a_00483","type":"journal-article","created":{"date-parts":[[2022,6,15]],"date-time":"2022-06-15T14:28:11Z","timestamp":1655303291000},"page":"680-696","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":16,"title":["Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression"],"prefix":"10.1162","volume":"10","author":[{"given":"Yuxia","family":"Wang","sequence":"first","affiliation":[{"name":"The University of Melbourne, Melbourne, Victoria, Australia. d.beck@unimelb.edu.au"}]},{"given":"Daniel","family":"Beck","sequence":"additional","affiliation":[{"name":"The University of Melbourne, Melbourne, Victoria, Australia. d.beck@unimelb.edu.au"}]},{"given":"Timothy","family":"Baldwin","sequence":"additional","affiliation":[{"name":"The University of Melbourne, Melbourne, Victoria, Australia. tb@ldwin.net"}]},{"given":"Karin","family":"Verspoor","sequence":"additional","affiliation":[{"name":"The University of Melbourne, Melbourne, Victoria, Australia"},{"name":"RMIT University, Melbourne, Victoria, Australia. karin.verspoor@rmit.edu.au"}]}],"member":"281","published-online":{"date-parts":[[2022,6,15]]},"reference":[{"key":"2022061514280103500_bib1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2021.05.008","article-title":"A review of uncertainty quantification in deep learning: Techniques, applications and challenges","author":"Abdar","year":"2020","journal-title":"arXiv preprint arXiv:2011.06225"},{"key":"2022061514280103500_bib2","doi-asserted-by":"publisher","first-page":"1798","DOI":"10.3115\/v1\/D14-1190","article-title":"Joint emotion analysis via multi-task Gaussian processes","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Beck","year":"2014"},{"key":"2022061514280103500_bib3","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1145\/1553374.1553380","article-title":"Curriculum learning","volume-title":"Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009","author":"Bengio","year":"2009"},{"key":"2022061514280103500_bib4","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v29i1.9515","article-title":"Predicting peer-to-peer loan rates using Bayesian non-linear regression","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Bitvai","year":"2015"},{"key":"2022061514280103500_bib5","first-page":"1613","article-title":"Weight uncertainty in neural networks","volume-title":"International Conference on Machine Learning","author":"Blundell","year":"2015"},{"key":"2022061514280103500_bib6","article-title":"Estimation of accurate and calibrated uncertainties in deterministic models","author":"Camporeale","year":"2020","journal-title":"CoRR"},{"key":"2022061514280103500_bib7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18653\/v1\/S17-2001","article-title":"SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation","volume-title":"Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 2017)","author":"Cer","year":"2017"},{"key":"2022061514280103500_bib8","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450087","article-title":"Unite: Uncertainty-based health risk prediction leveraging multi-sourced data","author":"Chen","year":"2020","journal-title":"arXiv preprint arXiv:2010.11389"},{"key":"2022061514280103500_bib9","article-title":"Rethinking embedding coupling in pre- trained language models","author":"Chung","year":"2020","journal-title":"arXiv preprint arXiv: 2010.12821"},{"key":"2022061514280103500_bib10","doi-asserted-by":"publisher","first-page":"13","DOI":"10.3115\/1631862.1631865","article-title":"Measuring the semantic similarity of texts","volume-title":"Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment","author":"Corley","year":"2005"},{"issue":"6","key":"2022061514280103500_bib11","doi-asserted-by":"publisher","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","article-title":"Calibration drift in regression and machine learning models for acute kidney injury","volume":"24","author":"Davis","year":"2017","journal-title":"Journal of the American Medical Informatics Association"},{"key":"2022061514280103500_bib12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.21","article-title":"Calibration of pre-trained transformers","author":"Desai","year":"2020","journal-title":"arXiv preprint arXiv:2003.07892"},{"key":"2022061514280103500_bib13","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2022061514280103500_bib14","article-title":"BLiTZ \u2013 Bayesian Layers in Torch Zoo (a Bayesian deep learing library for Torch)","author":"Esposito","year":"2020"},{"key":"2022061514280103500_bib15","first-page":"1050","article-title":"Dropout as a Bayesian approximation: Representing model uncertainty in deep learning","volume-title":"International Conference on Machine Learning","author":"Gal","year":"2016"},{"key":"2022061514280103500_bib16","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2021.emnlp-main.552","article-title":"SimCSE: Simple contrastive learning of sentence embeddings","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Gao","year":"2021"},{"key":"2022061514280103500_bib17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.330","article-title":"Uncertainty- aware machine translation evaluation","author":"Glushkova","year":"2021","journal-title":"CoRR"},{"issue":"2","key":"2022061514280103500_bib18","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1111\/j.1467-9868.2007.00587.x","article-title":"Probabilistic forecasts, calibration and sharpness","volume":"69","author":"Gneiting","year":"2007","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"1","key":"2022061514280103500_bib19","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1017\/S1351324915000339","article-title":"Can machine translation systems be evaluated by the crowd alone?","volume":"23","author":"Graham","year":"2017","journal-title":"Natural Language Engineering"},{"key":"2022061514280103500_bib20","first-page":"1321","article-title":"On calibration of modern neural networks","volume-title":"International Conference on Machine Learning","author":"Guo","year":"2017"},{"key":"2022061514280103500_bib21","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2019.103321","article-title":"Quantifying semantic similarity of clinical evidence in the biomedical literature to facilitate related evidence synthesis","author":"Hassanzadeh","year":"2019","journal-title":"Journal of Biomedical Informatics"},{"key":"2022061514280103500_bib22","first-page":"1861","article-title":"Probabilistic backpropagation for scalable learning of Bayesian neural networks","volume-title":"International Conference on Machine Learning","author":"Miguel Hern\u00e1ndez-Lobato","year":"2015"},{"key":"2022061514280103500_bib23","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1162\/tacl_a_00324","article-title":"How can we know what language models know?","volume":"8","author":"Jiang","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022061514280103500_bib24","doi-asserted-by":"publisher","first-page":"1647","DOI":"10.18653\/v1\/N18-1149","article-title":"A dataset of peer reviews (PeerRead): Collection, insights and NLP applications","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Kang","year":"2018"},{"key":"2022061514280103500_bib25","article-title":"A hybrid approach for sentiment analysis applied to paper","volume-title":"Proceedings of ACM SIGKDD Conference","author":"Keith","year":"2017"},{"key":"2022061514280103500_bib26","first-page":"5574","article-title":"What uncertainties do we need in Bayesian deep learning for computer vision?","volume-title":"Advances in Neural Information Processing Systems","author":"Kendall","year":"2017"},{"key":"2022061514280103500_bib27","first-page":"2796","article-title":"Accurate uncertainties for deep learning using calibrated regression","volume-title":"International Conference on Machine Learning","author":"Kuleshov","year":"2018"},{"key":"2022061514280103500_bib28","article-title":"Self-paced learning for latent variable models.","volume-title":"Advances in Neural Information Processing Systems","author":"Pawan Kumar","year":"2010"},{"key":"2022061514280103500_bib29","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume-title":"Advances in Neural Information Processing Systems","author":"Lakshminarayanan","year":"2017"},{"key":"2022061514280103500_bib30","first-page":"393","article-title":"Well-calibrated regression uncertainty in medical imaging with deep learning","volume-title":"Medical Imaging with Deep Learning","author":"Laves","year":"2020"},{"key":"2022061514280103500_bib31","article-title":"A tutorial on sparse gaussian processes and variational inference","author":"Leibfried","year":"2020","journal-title":"arXiv preprint arXiv:2012.13962"},{"key":"2022061514280103500_bib32","article-title":"WMT 2020 quality estimation dataset","author":"Lucia","year":"2020"},{"key":"2022061514280103500_bib33","article-title":"Bayesian methods for semi- supervised text annotation","author":"Miok","year":"2020","journal-title":"arXiv preprint arXiv:2010.14872"},{"key":"2022061514280103500_bib34","first-page":"6359","article-title":"Bayesian batch active learning as sparse subset approximation","volume-title":"Advances in Neural Information Processing Systems","author":"Pinsler","year":"2019"},{"key":"2022061514280103500_bib35","first-page":"977","article-title":"A temporal model of text periodicities using Gaussian processes","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Preo\u0163iuc-Pietro","year":"2013"},{"key":"2022061514280103500_bib36","doi-asserted-by":"publisher","first-page":"4310","DOI":"10.18653\/v1\/2021.acl-long.332","article-title":"Subsequence based deep active learning for named entity recognition","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL\/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1\u20136, 2021","author":"Radmard","year":"2021"},{"key":"2022061514280103500_bib37","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence- BERT: Sentence embeddings using Siamese BERT-networks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Reimers","year":"2019"},{"key":"2022061514280103500_bib38","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/3206.001.0001","article-title":"Gaussian process for machine learning","author":"Rusmassen","year":"2005"},{"key":"2022061514280103500_bib39","article-title":"Yelp review dataset","author":"Sabnis","year":"2018"},{"key":"2022061514280103500_bib40","article-title":"Active learning literature survey","author":"Settles","year":"2009"},{"key":"2022061514280103500_bib41","doi-asserted-by":"publisher","first-page":"1070","DOI":"10.3115\/1613715.1613855","article-title":"An analysis of active learning strategies for sequence labeling tasks","volume-title":"2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25\u201327 October 2008, Honolulu, Hawaii, USA, A meeting of SIGDAT, a Special Interest Group of the ACL","author":"Settles","year":"2008"},{"key":"2022061514280103500_bib42","doi-asserted-by":"publisher","first-page":"191","DOI":"10.18653\/v1\/D19-5525","article-title":"Modelling uncertainty in collaborative document quality assessment","volume-title":"Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)","author":"Shen","year":"2019"},{"key":"2022061514280103500_bib43","article-title":"A novel regression loss for non-parametric uncertainty optimization","author":"Sicking","year":"2021","journal-title":"arXiv preprint arXiv:2101.02726"},{"issue":"14","key":"2022061514280103500_bib44","doi-asserted-by":"publisher","first-page":"i49\u2013i58","DOI":"10.1093\/bioinformatics\/btx238","article-title":"BIOSSES: A semantic sentence similarity estimation system for the biomedical domain","volume":"33","author":"So\u011fanc\u0131o\u011flu","year":"2017","journal-title":"Bioinformatics"},{"key":"2022061514280103500_bib45","first-page":"5897","article-title":"Distribution calibration for regression","volume-title":"International Conference on Machine Learning","author":"Song","year":"2019"},{"key":"2022061514280103500_bib46","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.28","article-title":"Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks","author":"Thakur","year":"2020","journal-title":"arXiv preprint arXiv:2010.08240"},{"key":"2022061514280103500_bib47","first-page":"567","article-title":"Variational learning of inducing variables in sparse Gaussian processes","volume-title":"Artificial intelligence and statistics","author":"Titsias","year":"2009"},{"key":"2022061514280103500_bib48","first-page":"3459","article-title":"Evaluating model calibration in classification","volume-title":"The 22nd International Conference on Artificial Intelligence and Statistics","author":"Vaicenavicius","year":"2019"},{"key":"2022061514280103500_bib49","doi-asserted-by":"publisher","first-page":"1074","DOI":"10.18653\/v1\/2020.emnlp-main.80","article-title":"Self-paced learning for neural machine translation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16\u201320, 2020","author":"Wan","year":"2020"},{"key":"2022061514280103500_bib50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10579-018-9431-1","article-title":"MedSTS: A resource for clinical semantic textual similarity","author":"Wang","year":"2018","journal-title":"Language Resources and Evaluation"},{"issue":"11","key":"2022061514280103500_bib51","doi-asserted-by":"publisher","DOI":"10.2196\/23375","article-title":"The 2019 n2c2\/OHNLP track on clinical semantic textual similarity: Overview","volume":"8","author":"Wang","year":"2020","journal-title":"JMIR Medical Informatics"},{"key":"2022061514280103500_bib52","doi-asserted-by":"publisher","first-page":"105","DOI":"10.18653\/v1\/2020.bionlp-1.11","article-title":"Evaluating the utility of model configurations and data augmentation on clinical semantic textual similarity","volume-title":"Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing","author":"Wang","year":"2020"},{"key":"2022061514280103500_bib53","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.clinicalnlp-1.25","article-title":"Learning from unlabeled data for clinical semantic textual similarity","volume-title":"Proceedings of the 3rd Clinical NLP Workshop","author":"Wang","year":"2020"},{"key":"2022061514280103500_bib54","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP39728.2021.9414046","article-title":"Bayesian transformer language models for speech recognition","author":"Xue","year":"2021","journal-title":"arXiv preprint arXiv:2102.04754"},{"key":"2022061514280103500_bib55","article-title":"Crude: Calibrating regression uncertainty distributions empirically","author":"Zelikman","year":"2020","journal-title":"arXiv preprint arXiv:2005.12496"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00483\/2029951\/tacl_a_00483.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00483\/2029951\/tacl_a_00483.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,8]],"date-time":"2023-02-08T10:25:41Z","timestamp":1675851941000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00483\/111592\/Uncertainty-Estimation-and-Reduction-of-Pre"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":55,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00483","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}