{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,16]],"date-time":"2026-05-16T15:44:16Z","timestamp":1778946256475,"version":"3.51.4"},"reference-count":52,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,2,14]],"date-time":"2024-02-14T00:00:00Z","timestamp":1707868800000},"content-version":"vor","delay-in-days":44,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,2,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Computational linguistics models commonly target the prediction of discrete\u2014categorical\u2014labels. When assessing how well-calibrated these model predictions are, popular evaluation schemes require practitioners to manually determine a binning scheme: grouping labels into bins to approximate true label posterior. The problem is that these metrics are sensitive to binning decisions. We consider two solutions to the binning problem that apply at the stage of data annotation: collecting either distributed (redundant) labels or direct scalar value assignment.<\/jats:p>\n               <jats:p>In this paper, we show that although both approaches address the binning problem by evaluating instance-level calibration, direct scalar assignment is significantly more cost-effective. We provide theoretical analysis and empirical evidence to support our proposal for dataset creators to adopt scalar annotation protocols to enable a higher-quality assessment of model calibration.<\/jats:p>","DOI":"10.1162\/tacl_a_00636","type":"journal-article","created":{"date-parts":[[2024,2,15]],"date-time":"2024-02-15T14:13:55Z","timestamp":1708006435000},"page":"120-136","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["Addressing the Binning Problem in Calibration Assessment through Scalar Annotations"],"prefix":"10.1162","volume":"12","author":[{"given":"Zhengping","family":"Jiang","sequence":"first","affiliation":[{"name":"Johns Hopkins University, USA. zjiang31@jhu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anqi","family":"Liu","sequence":"additional","affiliation":[{"name":"Johns Hopkins University, USA. aliu@cs.jhu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamnin Van","family":"Durme","sequence":"additional","affiliation":[{"name":"Johns Hopkins University, USA. vandurme@jhu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,2,16]]},"reference":[{"key":"2024021514134897900_bib1","doi-asserted-by":"publisher","first-page":"1892","DOI":"10.18653\/v1\/2022.emnlp-main.124","article-title":"Stop measuring calibration when humans disagree","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Baan","year":"2022"},{"key":"2024021514134897900_bib2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.bigscience-1.9","article-title":"GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow","author":"Black","year":"2021"},{"key":"2024021514134897900_bib3","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024021514134897900_bib4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3758\/s13428-021-01566-w","article-title":"Beyond likert ratings: Improving the robustness of developmental research measurement using best\u2013worst scaling","author":"Burton","year":"2021","journal-title":"Behavior Research Methods"},{"key":"2024021514134897900_bib5","article-title":"Evaluating large language models trained on code","author":"Chen","year":"2021","journal-title":"arXiv preprint arXiv:2107.03374"},{"key":"2024021514134897900_bib6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.774","article-title":"Uncertain natural language inference","author":"Chen","year":"2019","journal-title":"arXiv preprint arXiv:1909.03042"},{"key":"2024021514134897900_bib7","article-title":"Boolq: Exploring the surprising difficulty of natural yes\/no questions","author":"Clark","year":"2019","journal-title":"arXiv preprint arXiv:1905.10044"},{"issue":"2","key":"2024021514134897900_bib8","doi-asserted-by":"publisher","first-page":"844","DOI":"10.1214\/009052607000000910","article-title":"Ranking and empirical minimization of u-statistics","volume":"36","author":"Cl\u00e9men\u00e7on","year":"2008","journal-title":"The Annals of Statistics"},{"key":"2024021514134897900_bib9","first-page":"441","article-title":"Minimax learning rates for bipartite ranking and plug-in rules","volume-title":"International Conference on Machine Learning","author":"Cl\u00e9men\u00e7on","year":"2011"},{"issue":"1","key":"2024021514134897900_bib10","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1007\/s10994-012-5325-4","article-title":"Ranking data with ordinal labels: Optimality and pairwise aggregation","volume":"91","author":"Cl\u00e9men\u00e7on","year":"2013","journal-title":"Machine Learning"},{"key":"2024021514134897900_bib11","doi-asserted-by":"publisher","DOI":"10.1145\/3600211.3604692","article-title":"Human uncertainty in concept-based AI systems","author":"Collins","year":"2023","journal-title":"arXiv preprint arXiv:2303.12872"},{"key":"2024021514134897900_bib12","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v10i1.21986","article-title":"Eliciting and learning with soft labels from every annotator","author":"Collins","year":"2022","journal-title":"arXiv preprint arXiv:2207.00810"},{"key":"2024021514134897900_bib13","doi-asserted-by":"publisher","first-page":"2096","DOI":"10.18653\/v1\/2021.findings-emnlp.180","article-title":"On the effects of transformer size on in-and out-of-domain calibration","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Dan","year":"2021"},{"key":"2024021514134897900_bib14","doi-asserted-by":"publisher","first-page":"4040","DOI":"10.18653\/v1\/2020.acl-main.372","article-title":"GoEmotions: A dataset of fine-grained emotions","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Demszky","year":"2020"},{"key":"2024021514134897900_bib15","doi-asserted-by":"publisher","first-page":"3450","DOI":"10.1109\/CVPR.2012.6248086","article-title":"Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition","volume-title":"2012 IEEE Conference on Computer Vision and Pattern Recognition","author":"Deng","year":"2012"},{"key":"2024021514134897900_bib16","doi-asserted-by":"publisher","first-page":"295","DOI":"10.18653\/v1\/2020.emnlp-main.21","article-title":"Calibration of pre-trained transformers","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Desai","year":"2020"},{"key":"2024021514134897900_bib17","first-page":"1321","article-title":"On calibration of modern neural networks","volume-title":"International Conference on Machine Learning","author":"Guo","year":"2017"},{"key":"2024021514134897900_bib18","article-title":"Measuring massive multitask language understanding","author":"Hendrycks","year":"2020","journal-title":"arXiv preprint arXiv:2009.03300"},{"issue":"7","key":"2024021514134897900_bib19","article-title":"Distilling the knowledge in a neural network","volume":"2","author":"Hinton","year":"2015","journal-title":"arXiv preprint arXiv:1503.02531"},{"key":"2024021514134897900_bib20","first-page":"1120","article-title":"Learning whom to trust with MACE","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Hovy","year":"2013"},{"key":"2024021514134897900_bib21","doi-asserted-by":"publisher","first-page":"2648","DOI":"10.18653\/v1\/2022.emnlp-main.170","article-title":"Calibrating zero-shot cross-lingual (un-)structured predictions","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Jiang","year":"2022"},{"key":"2024021514134897900_bib22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.470","article-title":"Wice: Real-world entailment for claims in wikipedia","author":"Kamoi","year":"2023","journal-title":"arXiv preprint arXiv:2303.01432"},{"key":"2024021514134897900_bib23","article-title":"Large language models are zero-shot reasoners","author":"Kojima","year":"2022","journal-title":"arXiv preprint arXiv:2205.11916"},{"key":"2024021514134897900_bib24","article-title":"Generalized sliced Wasserstein distances","volume":"32","author":"Kolouri","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024021514134897900_bib25","article-title":"Calibrated structured prediction","volume-title":"Advances in Neural Information Processing Systems","author":"Kuleshov","year":"2015"},{"key":"2024021514134897900_bib26","article-title":"Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration","volume":"32","author":"Kull","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024021514134897900_bib27","article-title":"Verified uncertainty calibration","volume":"32","author":"Kumar","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024021514134897900_bib28","unstructured":"Aitor\n              Lewkowycz\n            , AmbroseSlone, AndersAndreassen, DanielFreeman, Ethan S.Dyer, GauravMishra, GuyGur-Ari, JaehoonLee, JaschaSohl-dickstein, KristenChiafullo, Liam B.Fedus, NoahFiedel, RosanneLiu, VedantMisra, and Vinay VenkateshRamasesh. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Technical report."},{"key":"2024021514134897900_bib29","doi-asserted-by":"publisher","first-page":"4818","DOI":"10.18653\/v1\/P19-1475","article-title":"Learning to rank for plausible plausibility","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Li","year":"2019"},{"key":"2024021514134897900_bib30","article-title":"Roberta: A robustly optimized bert pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"2024021514134897900_bib31","doi-asserted-by":"publisher","first-page":"7411","DOI":"10.18653\/v1\/2020.emnlp-main.601","article-title":"\u201cI\u2019d rather just go to bed\u201d: Understanding indirect answers","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Louis","year":"2020"},{"key":"2024021514134897900_bib32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-short.109","article-title":"Embracing ambiguity: Shifting the training target of NLI models","author":"Meissner","year":"2021","journal-title":"arXiv preprint arXiv:2106 .03020"},{"key":"2024021514134897900_bib33","article-title":"On the relationship between binary classification, bipartite ranking, and binary class probability estimation","volume":"26","author":"Narasimhan","year":"2013","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024021514134897900_bib34","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.441","article-title":"Adversarial NLI: A new benchmark for natural language understanding","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Nie","year":"2020"},{"key":"2024021514134897900_bib35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.734","article-title":"What can we learn from collective human opinions on natural language inference data?","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Nie","year":"2020"},{"key":"2024021514134897900_bib36","article-title":"Measuring calibration in deep learning.","volume-title":"CVPR Workshops","author":"Nixon","year":"2019"},{"key":"2024021514134897900_bib37","article-title":"Capabilities of GPT-4 on medical challenge problems","author":"Nori","year":"2023","journal-title":"arXiv preprint arXiv:2303 .13375"},{"key":"2024021514134897900_bib38","unstructured":"OpenAI. 2023. GPT-4 technical report."},{"key":"2024021514134897900_bib39","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024021514134897900_bib40","volume-title":"Can You Trust Your Model\u2019s Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift","author":"Ovadia","year":"2019"},{"key":"2024021514134897900_bib41","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1162\/tacl_a_00293","article-title":"Inherent disagreements in human textual inferences","volume":"7","author":"Pavlick","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024021514134897900_bib42","doi-asserted-by":"publisher","first-page":"9617","DOI":"10.1109\/ICCV.2019.00971","article-title":"Human uncertainty makes classification more robust","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Peterson","year":"2019"},{"key":"2024021514134897900_bib43","article-title":"On fairness and calibration","volume-title":"Advances in Neural Information Processing Systems","author":"Pleiss","year":"2017"},{"issue":"3","key":"2024021514134897900_bib44","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1002\/ejsp.2420100303","article-title":"A comparison of ranking and rating procedures for value system measurement","volume":"10","author":"Rankin","year":"1980","journal-title":"European Journal of Social Psychology"},{"issue":"1","key":"2024021514134897900_bib45","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1111\/j.2044-8295.1994.tb02509.x","article-title":"Ranking or rating? some data and their implications for the measurement of evaluative response","volume":"85","author":"Russell","year":"1994","journal-title":"British Journal of Psychology"},{"key":"2024021514134897900_bib46","doi-asserted-by":"publisher","first-page":"2818","DOI":"10.1109\/CVPR.2016.308","article-title":"Rethinking the inception architecture for computer vision","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"Szegedy","year":"2016"},{"issue":"4481","key":"2024021514134897900_bib47","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1126\/science.7455683","article-title":"The framing of decisions and the psychology of choice","volume":"211","author":"Tversky","year":"1981","journal-title":"Science"},{"issue":"5","key":"2024021514134897900_bib48","doi-asserted-by":"publisher","first-page":"1080","DOI":"10.1109\/TPAMI.2014.2360397","article-title":"Statistical optimality in multipartite ranking and ordinal regression","volume":"37","author":"Uematsu","year":"2014","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2024021514134897900_bib49","first-page":"3459","article-title":"Evaluating model calibration in classification","volume-title":"The 22nd International Conference on Artificial Intelligence and Statistics","author":"Vaicenavicius","year":"2019"},{"key":"2024021514134897900_bib50","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.190","article-title":"Generating data to mitigate spurious correlations in natural language inference datasets","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","author":"Yuxiang","year":"2022"},{"key":"2024021514134897900_bib51","first-page":"22313","article-title":"Calibrating predictions to decisions: A novel approach to multi-class calibration","volume":"34","author":"Zhao","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024021514134897900_bib52","first-page":"11387","article-title":"Individual calibration with randomized forecasting","volume-title":"International Conference on Machine Learning","author":"Zhao","year":"2020"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00636\/2334456\/tacl_a_00636.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00636\/2334456\/tacl_a_00636.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,15]],"date-time":"2024-02-15T14:14:10Z","timestamp":1708006450000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00636\/119541\/Addressing-the-Binning-Problem-in-Calibration"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":52,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00636","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}