{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T07:42:27Z","timestamp":1784101347140,"version":"3.55.0"},"reference-count":60,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2020,2,27]],"date-time":"2020-02-27T00:00:00Z","timestamp":1582761600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01GM118609"],"award-info":[{"award-number":["R01GM118609"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01HL136835"],"award-info":[{"award-number":["R01HL136835"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Our primary objective is to provide the clinical informatics community with an introductory tutorial on calibration measurements and calibration models for predictive models using existing R packages and custom implemented code in R on real and simulated data. Clinical predictive model performance is commonly published based on discrimination measures, but use of models for individualized predictions requires adequate model calibration. This tutorial is intended for clinical researchers who want to evaluate predictive models in terms of their applicability to a particular population. It is also for informaticians and for software engineers who want to understand the role that calibration plays in the evaluation of a clinical predictive model, and to provide them with a solid starting point to consider incorporating calibration evaluation and calibration models in their work.<\/jats:p><jats:p>Covered topics include (1) an introduction to the importance of calibration in the clinical setting, (2) an illustration of the distinct roles that discrimination and calibration play in the assessment of clinical predictive models, (3) a tutorial and demonstration of selected calibration measurements, (4) a tutorial and demonstration of selected calibration models, and (5) a brief discussion of limitations of these methods and practical suggestions on how to use them in practice.<\/jats:p>","DOI":"10.1093\/jamia\/ocz228","type":"journal-article","created":{"date-parts":[[2020,1,2]],"date-time":"2020-01-02T20:08:47Z","timestamp":1577995727000},"page":"621-633","source":"Crossref","is-referenced-by-count":425,"title":["A tutorial on calibration measurements and calibration models for clinical prediction models"],"prefix":"10.1093","volume":"27","author":[{"given":"Yingxiang","family":"Huang","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, UC San Diego Health, University of California, San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wentao","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, UC San Diego Health, University of California, San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fima","family":"Macheret","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, UC San Diego Health, University of California, San Diego, La Jolla, California, USA"},{"name":"Division of Hospital Medicine, Department of Medicine, University of California, San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rodney A","family":"Gabriel","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, University of California, San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucila","family":"Ohno-Machado","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, UC San Diego Health, University of California, San Diego, La Jolla, California, USA"},{"name":"Division of Health Services Research & Development, VA San Diego Healthcare System, San Diego, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,2,27]]},"reference":[{"issue":"1","key":"2020110613080044100_ocz228-B1","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","article-title":"Assessing the performance of prediction models: a framework for traditional and novel measures","volume":"21","author":"Steyerberg","year":"2010","journal-title":"Epidemiology"},{"issue":"14","key":"2020110613080044100_ocz228-B2","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1001\/jama.2017.12126","article-title":"Discrimination and calibration of clinical prediction models: users\u2019 guides to the medical literature","volume":"318","author":"Alba","year":"2017","journal-title":"JAMA"},{"key":"2020110613080044100_ocz228-B3","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-77244-8","volume-title":"Clinical Prediction Models","author":"Steyerberg","year":"2009"},{"issue":"14","key":"2020110613080044100_ocz228-B4","doi-asserted-by":"crossref","first-page":"1326","DOI":"10.1056\/NEJMsa1204629","article-title":"Monetary costs of dementia in the United States","volume":"368","author":"Hurd","year":"2013","journal-title":"N Engl J Med"},{"issue":"7","key":"2020110613080044100_ocz228-B5","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1007\/s10654-018-0403-y","article-title":"External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from the Rotterdam Study","volume":"33","author":"Licher","year":"2018","journal-title":"Eur J Epidemiol"},{"issue":"9","key":"2020110613080044100_ocz228-B6","first-page":"580","article-title":"Estimating cardiovascular risk","volume":"95","author":"Firnhaber","year":"2017","journal-title":"Am Fam Physician"},{"key":"2020110613080044100_ocz228-B7"},{"key":"2020110613080044100_ocz228-B8","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.prevetmed.2017.11.018","article-title":"A discussion of calibration techniques for evaluating binary and categorical predictive models","volume":"149","author":"Fenlon","year":"2018","journal-title":"Prev Vet Med"},{"key":"2020110613080044100_ocz228-B9","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.jbi.2017.10.008","article-title":"Beyond discrimination: a comparison of calibration methods and clinical usefulness of predictive models of readmission risk","volume":"76","author":"Walsh","year":"2017","journal-title":"J Biomed Inform"},{"issue":"29","key":"2020110613080044100_ocz228-B10","doi-asserted-by":"crossref","first-page":"1925","DOI":"10.1093\/eurheartj\/ehu207","article-title":"Towards better clinical prediction models: seven steps for development and an ABCD for validation","volume":"35","author":"Steyerberg","year":"2014","journal-title":"Eur Heart J"},{"issue":"4","key":"2020110613080044100_ocz228-B11","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1161\/CIRCOUTCOMES.115.001693","article-title":"Clinical prediction models for cardiovascular disease: tufts predictive analytics and comparative effectiveness clinical prediction model database","volume":"8","author":"Wessler","year":"2015","journal-title":"Circ Cardiovasc Qual Outcomes"},{"issue":"2","key":"2020110613080044100_ocz228-B12","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1002\/sim.4780030207","article-title":"Regression modelling strategies for improved prognostic prediction","volume":"3","author":"Harrell","year":"1984","journal-title":"Stat Med"},{"issue":"18","key":"2020110613080044100_ocz228-B13","doi-asserted-by":"crossref","first-page":"2543","DOI":"10.1001\/jama.1982.03320430047030","article-title":"Evaluating the yield of medical tests","volume":"247","author":"Harrell","year":"1982","journal-title":"JAMA"},{"issue":"9","key":"2020110613080044100_ocz228-B14","doi-asserted-by":"crossref","first-page":"2052","DOI":"10.1097\/01.CCM.0000275267.64078.B0","article-title":"Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited","volume":"35","author":"Kramer","year":"2007","journal-title":"Crit Care Med"},{"key":"2020110613080044100_ocz228-B15","first-page":"625","author":"Niculescu-Mizil","year":"2005"},{"key":"2020110613080044100_ocz228-B16","doi-asserted-by":"crossref","DOI":"10.1201\/b11031","volume-title":"Statistical Evaluation of Diagnostic Performance: Topics in ROC Analysis","author":"Zou","year":"2016"},{"issue":"8","key":"2020110613080044100_ocz228-B17","doi-asserted-by":"crossref","first-page":"938","DOI":"10.1016\/j.jclinepi.2009.11.009","article-title":"Use of Brier score to assess binary predictions","volume":"63","author":"Rufibach","year":"2010","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2020110613080044100_ocz228-B18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","article-title":"Verification of forecasts expressed in terms of probability","volume":"78","author":"Brier","year":"1950","journal-title":"Mon Weather Rev"},{"key":"2020110613080044100_ocz228-B19"},{"issue":"10","key":"2020110613080044100_ocz228-B20","doi-asserted-by":"crossref","first-page":"1043","DOI":"10.1080\/03610928008827941","article-title":"Goodness of fit tests for the multiple logistic regression model","volume":"9","author":"Hosmer","year":"1980","journal-title":"Commun Stat Theory Methods"},{"key":"2020110613080044100_ocz228-B21","doi-asserted-by":"crossref","DOI":"10.1002\/0471722146","volume-title":"Applied Logistic Regression","author":"Hosmer","year":"2000","edition":"2nd ed."},{"issue":"1","key":"2020110613080044100_ocz228-B22","doi-asserted-by":"crossref","first-page":"122","DOI":"10.2193\/2007-535","article-title":"A new method for estimation of resource selection probability function","volume":"73","author":"Lele","year":"2009","journal-title":"J Wildl Manag"},{"issue":"11","key":"2020110613080044100_ocz228-B23","doi-asserted-by":"crossref","DOI":"10.18637\/jss.v023.i11","article-title":"PresenceAbsence: an R package for presence absence analysis","volume":"23","author":"Freeman","year":"2008","journal-title":"J Stat Softw"},{"issue":"8","key":"2020110613080044100_ocz228-B24","doi-asserted-by":"crossref","first-page":"1823","DOI":"10.1109\/TPAMI.2019.2895794","article-title":"Calibrating classification probabilities with shape-restricted polynomial regression","volume":"41","author":"Wang","year":"2019","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2020110613080044100_ocz228-B25","author":"Guo","year":"2017"},{"key":"2020110613080044100_ocz228-B26","first-page":"2901","article-title":"Obtaining well calibrated probabilities using Bayesian binning","volume":"2015","author":"Naeini","year":"2015","journal-title":"Proc Conf AAAI Artif Intell"},{"key":"2020110613080044100_ocz228-B27"},{"issue":"2","key":"2020110613080044100_ocz228-B28","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1590\/s1679-45082017ao3913","article-title":"Spontaneous breathing test in the prediction of extubation failure in the pediatric population","volume":"15","author":"Nascimento","year":"2017","journal-title":"Einstein (Sao Paulo)"},{"issue":"21","key":"2020110613080044100_ocz228-B29","doi-asserted-by":"crossref","first-page":"4051","DOI":"10.1002\/sim.8281","article-title":"The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models","volume":"38","author":"Austin","year":"2019","journal-title":"Stat Med"},{"issue":"11","key":"2020110613080044100_ocz228-B30","doi-asserted-by":"crossref","first-page":"e48823","DOI":"10.1371\/journal.pone.0048823","article-title":"Doubly Optimized Calibrated Support Vector Machine (DOC-SVM): an algorithm for joint optimization of discrimination and calibration","volume":"7","author":"Jiang","year":"2012","journal-title":"PLoS One"},{"issue":"3","key":"2020110613080044100_ocz228-B31","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1177\/2167702617691560","article-title":"Predicting risk of suicide attempts over time through machine learning","volume":"5","author":"Walsh","year":"2017","journal-title":"Clin Psychol Sci"},{"issue":"12","key":"2020110613080044100_ocz228-B32","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1111\/jcpp.12916","article-title":"Predicting suicide attempts in adolescents with longitudinal clinical data and machine learning","volume":"59","author":"Walsh","year":"2018","journal-title":"J Child Psychol Psychiatry"},{"key":"2020110613080044100_ocz228-B33","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.jclinepi.2015.12.005","article-title":"A calibration hierarchy for risk models was defined: from utopia to empirical data","volume":"74","author":"Van Calster","year":"2016","journal-title":"J Clin Epidemiol"},{"key":"2020110613080044100_ocz228-B34","doi-asserted-by":"crossref","first-page":"i3140","DOI":"10.1136\/bmj.i3140","article-title":"External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges","volume":"353","author":"Riley","year":"2016","journal-title":"BMJ"},{"key":"2020110613080044100_ocz228-B35","volume-title":"Advances in Large Margin Classifiers","author":"Platt","year":"2000"},{"issue":"5","key":"2020110613080044100_ocz228-B36","doi-asserted-by":"crossref","DOI":"10.18637\/jss.v032.i05","article-title":"Isotone optimization inR: Pool-Adjacent-Violators Algorithm (PAVA) and active set methods","volume":"32","author":"Leeuw","year":"2009","journal-title":"J Stat Softw"},{"key":"2020110613080044100_ocz228-B37","first-page":"609","author":"Zadrozny","year":"2001"},{"issue":"3","key":"2020110613080044100_ocz228-B38","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1007\/BF00994016","article-title":"Learning Bayesian networks: the combination of knowledge and statistical data","volume":"20","author":"Heckerman","year":"1995","journal-title":"Mach Learn"},{"key":"2020110613080044100_ocz228-B39","first-page":"623","article-title":"Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers","volume":"54","author":"Kull","year":"2017","journal-title":"Proc Int Conf Artif Intell Stat"},{"key":"2020110613080044100_ocz228-B40","volume-title":"HCUP Nationwide Inpatient Sample","year":"2012"},{"issue":"1","key":"2020110613080044100_ocz228-B41","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1002\/sim.5525","article-title":"Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets","volume":"32","author":"Paul","year":"2013","journal-title":"Stat Med"},{"issue":"23","key":"2020110613080044100_ocz228-B42","doi-asserted-by":"crossref","first-page":"11813","DOI":"10.1080\/03610926.2017.1285922","article-title":"A modified Hosmer\u2013Lemeshow test for large data sets","volume":"46","author":"Yu","year":"2017","journal-title":"Commun Stat Theory Methods"},{"issue":"13","key":"2020110613080044100_ocz228-B43","doi-asserted-by":"crossref","first-page":"2463","DOI":"10.1080\/00949655.2018.1467912","article-title":"A simple test procedure in standardizing the power of Hosmer\u2013Lemeshow test in large data sets","volume":"88","author":"Lai","year":"2018","journal-title":"J Stat Comput Simul"},{"issue":"9","key":"2020110613080044100_ocz228-B44","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1161\/CIRCRESAHA.117.311312","article-title":"Cardiovascular event prediction by machine learning: the multi-ethnic study of atherosclerosis","volume":"121","author":"Ambale-Venkatesh","year":"2017","journal-title":"Circ Res"},{"issue":"5","key":"2020110613080044100_ocz228-B45","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1016\/S1470-2045(17)30155-9","article-title":"DNA methylation-based classification and grading system for meningioma: a multicentre, retrospective analysis","volume":"18","author":"Sahm","year":"2017","journal-title":"Lancet Oncol"},{"issue":"4","key":"2020110613080044100_ocz228-B46","doi-asserted-by":"crossref","first-page":"e157","DOI":"10.1016\/S2352-3026(17)30026-1","article-title":"Derivation and external validation of the PLASMIC score for rapid assessment of adults with thrombotic microangiopathies: a cohort study","volume":"4","author":"Bendapudi","year":"2017","journal-title":"Lancet Haematol"},{"issue":"1","key":"2020110613080044100_ocz228-B47","doi-asserted-by":"crossref","first-page":"F9","DOI":"10.1136\/adc.2008.148015","article-title":"Predicting neonatal mortality among very preterm infants: a comparison of three versions of the CRIB score","volume":"95","author":"Manktelow","year":"2010","journal-title":"Arch Dis Child Fetal Neonatal Ed"},{"issue":"5","key":"2020110613080044100_ocz228-B48","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1002\/sim.4780050506","article-title":"Probabilistic prediction in patient management and clinical trials","volume":"5","author":"Spiegelhalter","year":"1986","journal-title":"Stat Med"},{"issue":"7","key":"2020110613080044100_ocz228-B49","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1016\/j.bjps.2017.04.010","article-title":"External validation of the breast reconstruction risk assessment calculator","volume":"70","author":"Khavanin","year":"2017","journal-title":"J Plast Reconstr Aesthet Surg"},{"issue":"3","key":"2020110613080044100_ocz228-B50","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1175\/WAF993.1","article-title":"Increasing the reliability of reliability diagrams","volume":"22","author":"Br\u00f6cker","year":"2007","journal-title":"Weather Forecast"},{"issue":"5","key":"2020110613080044100_ocz228-B51","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/MC.2018.2381131","article-title":"Deep learning for the internet of things","volume":"51","author":"Yao","year":"2018","journal-title":"Computer"},{"key":"2020110613080044100_ocz228-B52","author":"Lee"},{"key":"2020110613080044100_ocz228-B53","author":"Maddox","year":"2019"},{"issue":"22","key":"2020110613080044100_ocz228-B54","doi-asserted-by":"crossref","first-page":"4290","DOI":"10.1002\/sim.8296","article-title":"Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration","volume":"38","author":"Steyerberg","year":"2019","journal-title":"Stat Med"},{"issue":"7","key":"2020110613080044100_ocz228-B55","doi-asserted-by":"crossref","first-page":"879","DOI":"10.1002\/bjs.11124","article-title":"Mortality prediction following non-traumatic amputation of the lower extremity","volume":"106","author":"Norvell","year":"2019","journal-title":"Br J Surg"},{"issue":"2","key":"2020110613080044100_ocz228-B56","doi-asserted-by":"crossref","first-page":"e20154477","DOI":"10.1542\/peds.2015-4477","article-title":"Predictors of poor school readiness in children without developmental delay at age 2","volume":"138","author":"Nelson","year":"2016","journal-title":"Pediatrics"},{"key":"2020110613080044100_ocz228-B57","first-page":"694","author":"Zadrozny","year":"2002"},{"issue":"2","key":"2020110613080044100_ocz228-B58","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1136\/amiajnl-2011-000291","article-title":"Calibrating predictive model estimates to support personalized medicine","volume":"19","author":"Jiang","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"10","key":"2020110613080044100_ocz228-B59","doi-asserted-by":"crossref","first-page":"1659","DOI":"10.1002\/sim.6428","article-title":"Tests of calibration and goodness-of-fit in the survival setting","volume":"34","author":"Demler","year":"2015","journal-title":"Stat Med"},{"key":"2020110613080044100_ocz228-B60","first-page":"16","article-title":"Smooth isotonic regression: a new method to calibrate predictive models","volume":"2011","author":"Jiang","year":"2011","journal-title":"AMIA Jt Summits Transl Sci Proc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/4\/621\/34153143\/ocz228.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/4\/621\/34153143\/ocz228.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,2,14]],"date-time":"2021-02-14T22:35:55Z","timestamp":1613342155000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/4\/621\/5762806"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,27]]},"references-count":60,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,2,27]]},"published-print":{"date-parts":[[2020,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz228","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,4]]},"published":{"date-parts":[[2020,2,27]]}}}