{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T18:33:59Z","timestamp":1785004439513,"version":"3.55.0"},"reference-count":30,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2022,12,14]],"date-time":"2022-12-14T00:00:00Z","timestamp":1670976000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,8,14]]},"abstract":"<jats:title>Summary<\/jats:title>\n               <jats:p>Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability $x$, it should materialize with approximately that frequency, which means that the so-called calibration curve $p(\\cdot)$ should equal the identity, i.e., $p(x) = x$ for all $x$ in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid subject to only the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well-specified model. We show that our bands have a finite-sample coverage guarantee, are narrower than those of existing approaches, and adapt to the local smoothness of the calibration curve $p$ and the local variance of the binary observations. In an application to modelling predictions of an infant having low birth weight, the bounds give informative insights into model calibration.<\/jats:p>","DOI":"10.1093\/biomet\/asac068","type":"journal-article","created":{"date-parts":[[2022,12,13]],"date-time":"2022-12-13T14:10:19Z","timestamp":1670940619000},"page":"663-680","source":"Crossref","is-referenced-by-count":17,"title":["Honest calibration assessment for binary outcome predictions"],"prefix":"10.1093","volume":"110","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8322-360X","authenticated-orcid":false,"given":"Timo","family":"Dimitriadis","sequence":"first","affiliation":[{"name":"Heidelberg University Alfred Weber Institute of Economics, , Bergheimer Str. 58, 69115 Heidelberg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lutz","family":"D\u00fcmbgen","sequence":"additional","affiliation":[{"name":"University of Bern Institute of Mathematical Statistics and Actuarial Science, , Alpeneggstrasse 22, 3012 Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Henzi","sequence":"additional","affiliation":[{"name":"Seminar for Statistics, ETH Z\u00fcrich, R\u00e4mistrasse 101, 8092 Z\u00fcrich , Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marius","family":"Puke","sequence":"additional","affiliation":[{"name":"University of Hohenheim Institute of Economics and Statistics, , Schloss Hohenheim 1 C, 70599 Stuttgart, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5916-9746","authenticated-orcid":false,"given":"Johanna","family":"Ziegel","sequence":"additional","affiliation":[{"name":"University of Bern Institute of Mathematical Statistics and Actuarial Science, , Alpeneggstrasse 22, 3012 Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,12,14]]},"reference":[{"key":"2023081420143900600_B1","article-title":"Measures of fit for logistic regression","author":"Allison,","year":"2014"},{"key":"2023081420143900600_B2","first-page":"251","article-title":"One model, several results: The paradox of the Hosmer-Lemeshow goodness-of-fit test for the logistic regression model","volume":"5","author":"Bertolini,","year":"2000","journal-title":"J. Epidemiol. Biostatist."},{"key":"2023081420143900600_B3","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1093\/biomet\/26.4.404","article-title":"The use of confidence or fiducial limits illustrated in the case of the binomial","volume":"26","author":"Clopper,","year":"1934","journal-title":"Biometrika"},{"issue":"3","key":"2023081420143900600_B4","article-title":"Stable reliability diagrams for probabilistic classifiers","volume":"118","author":"Dimitriadis,","year":"2021","journal-title":"Proc. Nat. Acad. Sci."},{"key":"2023081420143900600_B5","doi-asserted-by":"crossref","first-page":"568","DOI":"10.1214\/18-STS665","article-title":"Nonparametric shape-restricted regression","volume":"33","author":"Guntuboyina,","year":"2018","journal-title":"Statist. Sci."},{"key":"2023081420143900600_B6","doi-asserted-by":"crossref","first-page":"1892","DOI":"10.1214\/13-AOS1137","article-title":"A simple bootstrap method for constructing nonparametric confidence bands for functions","volume":"41","author":"Hall,","year":"2013","journal-title":"Ann. Statist."},{"key":"2023081420143900600_B7","doi-asserted-by":"crossref","first-page":"2633","DOI":"10.1007\/s11009-022-09937-2","article-title":"Accelerating the pool-adjacent-violators algorithm for isotonic distributional regression","volume":"24","author":"Henzi,","year":"2022","journal-title":"Methodol. Comp. Appl. Prob."},{"key":"2023081420143900600_B8","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1080\/01621459.1963.10500830","article-title":"Probability inequalities for sums of bounded random variables","volume":"58","author":"Hoeffding,","year":"1963","journal-title":"J. Am. Statist. Assoc."},{"key":"2023081420143900600_B9","doi-asserted-by":"crossref","first-page":"1043","DOI":"10.1080\/03610928008827941","article-title":"Goodness of fit tests for the multiple logistic regression model","volume":"9","author":"Hosmer,","year":"1980","journal-title":"Commun. Statist. A"},{"key":"2023081420143900600_B10","doi-asserted-by":"crossref","DOI":"10.1002\/9781118548387","volume-title":"Applied Logistic Regression","author":"Hosmer,","year":"2013","edition":"3rd ed"},{"key":"2023081420143900600_B11","doi-asserted-by":"crossref","DOI":"10.1002\/0471715816","volume-title":"Univariate Discrete Distributions","author":"Johnson,","year":"2005","edition":"3rd ed"},{"key":"2023081420143900600_B12","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.jeconom.2009.01.009","article-title":"Parametric links for binary choice models: A Fisherian--Bayesian colloquy","volume":"152","author":"Koenker,","year":"2009","journal-title":"J. Economet."},{"key":"2023081420143900600_B13","doi-asserted-by":"crossref","first-page":"2052","DOI":"10.1097\/01.CCM.0000275267.64078.B0","article-title":"Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited","volume":"35","author":"Kramer,","year":"2007","journal-title":"Critical Care Med."},{"key":"2023081420143900600_B14","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1214\/19-EJS1659","article-title":"Monotone least squares and isotonic quantiles","volume":"14","author":"M\u00f6sching,","year":"2020","journal-title":"Electron. J. Statist."},{"key":"2023081420143900600_B15","volume-title":"NCHS\u2019 Vital Statistics Natality Birth Data","year":"2017"},{"key":"2023081420143900600_B16","doi-asserted-by":"crossref","first-page":"2390","DOI":"10.1002\/sim.6100","article-title":"A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes","volume":"33","author":"Nattino,","year":"2014","journal-title":"Statist. Med."},{"key":"2023081420143900600_B17","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1111\/biom.13249","article-title":"Assessing the goodness of fit of logistic regression models in large samples: A modification of the Hosmer-Lemeshow test","volume":"76","author":"Nattino,","year":"2020","journal-title":"Biometrics"},{"key":"2023081420143900600_B18","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1111\/biom.13250","article-title":"Rejoinder to \u201cAssessing the goodness of fit of logistic regression models in large samples: A modification of the Hosmer-Lemeshow test\u201d","volume":"76","author":"Nattino,","year":"2020","journal-title":"Biometrics"},{"key":"2023081420143900600_B19","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1002\/sim.5525","article-title":"Standardizing the power of the Hosmer--Lemeshow goodness of fit test in large data sets","volume":"32","author":"Paul,","year":"2013","journal-title":"Statist. Med."},{"key":"2023081420143900600_B20","doi-asserted-by":"crossref","first-page":"6047","DOI":"10.1016\/j.vaccine.2016.03.045","article-title":"Preterm birth: Case definition & guidelines for data collection, analysis, and presentation of immunisation safety data","volume":"34","author":"Quinn,","year":"2016","journal-title":"Vaccine"},{"key":"2023081420143900600_B21","article-title":"R Foundation for Statistical Computing, Vienna, Austria","volume-title":"R: A Language and Environment for Statistical Computing","year":"2023"},{"key":"2023081420143900600_B22","article-title":"Mitigating bias in calibration error estimation","author":"Roelofs,","year":"2022","journal-title":"arXiv: 2012.08668v3"},{"key":"2023081420143900600_B23","doi-asserted-by":"crossref","first-page":"1953","DOI":"10.1214\/09-AOS777","article-title":"Inconsistency of bootstrap: The Grenander estimator","volume":"38","author":"Sen,","year":"2010","journal-title":"Ann. Statist."},{"key":"2023081420143900600_B24","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-34675-5","volume-title":"Stochastic Orders","author":"Shaked,","year":"2007"},{"key":"2023081420143900600_B25","doi-asserted-by":"crossref","first-page":"1240","DOI":"10.1126\/science.aah6168","article-title":"Enhancing reproducibility for computational methods","volume":"354","author":"Stodden,","year":"2016","journal-title":"Science"},{"key":"2023081420143900600_B26","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511842061","volume-title":"Regression for Categorical Data","author":"Tutz,","year":"2011"},{"key":"2023081420143900600_B27","volume-title":"International Statistical Classification of Diseases and Related Health Problems","year":"2015","edition":"5th ed"},{"key":"2023081420143900600_B28","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1214\/aos\/1176345411","article-title":"The asymptotic behavior of monotone regression estimates","volume":"9","author":"Wright,","year":"1981","journal-title":"Ann. Statist."},{"key":"2023081420143900600_B29","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1214\/18-EJS1520","article-title":"Contraction and uniform convergence of isotonic regression","volume":"13","author":"Yang,","year":"2019","journal-title":"Electron. J. Statist."},{"key":"2023081420143900600_B30","doi-asserted-by":"crossref","first-page":"3920","DOI":"10.1073\/pnas.1901326117","article-title":"Veridical data science","volume":"117","author":"Yu,","year":"2020","journal-title":"Proc. Nat. Acad. Sci."}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/biomet\/advance-article-pdf\/doi\/10.1093\/biomet\/asac068\/48936426\/asac068.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/110\/3\/663\/51111472\/asac068.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/110\/3\/663\/51111472\/asac068.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,14]],"date-time":"2023-08-14T20:35:23Z","timestamp":1692045323000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/110\/3\/663\/6900929"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,14]]},"references-count":30,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,12,14]]},"published-print":{"date-parts":[[2023,8,14]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asac068","relation":{},"ISSN":["0006-3444","1464-3510"],"issn-type":[{"value":"0006-3444","type":"print"},{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,9,1]]},"published":{"date-parts":[[2022,12,14]]}}}