{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T21:16:49Z","timestamp":1770844609585,"version":"3.50.1"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2020,11,24]],"date-time":"2020-11-24T00:00:00Z","timestamp":1606176000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Clalit Research Institute"},{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010663","name":"European Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Binational Science Foundation"},{"name":"Amazon Research Award"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To illustrate the problem of subpopulation miscalibration, to adapt an algorithm for recalibration of the predictions, and to validate its performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>In this retrospective cohort study, we evaluated the calibration of predictions based on the Pooled Cohort Equations (PCE) and the fracture risk assessment tool (FRAX) in the overall population and in subpopulations defined by the intersection of age, sex, ethnicity, socioeconomic status, and immigration history. We next applied the recalibration algorithm and assessed the change in calibration metrics, including calibration-in-the-large.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>1\u00a0021\u00a0041 patients were included in the PCE population, and 1\u00a0116\u00a0324 patients were included in the FRAX population. Baseline overall model calibration of the 2 tested models was good, but calibration in a substantial portion of the subpopulations was poor. After applying the algorithm, subpopulation calibration statistics were greatly improved, with the variance of the calibration-in-the-large values across all subpopulations reduced by 98.8% and 94.3% in the PCE and FRAX models, respectively.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Prediction models in medicine are increasingly common. Calibration, the agreement between predicted and observed risks, is commonly poor for subpopulations that were underrepresented in the development set of the models, resulting in bias and reduced performance for these subpopulations. In this work, we empirically evaluated an adapted version of the fairness algorithm designed by Hebert-Johnson et al. (2017) and demonstrated its use in improving subpopulation miscalibration.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>A postprocessing and model-independent fairness algorithm for recalibration of predictive models greatly decreases the bias of subpopulation miscalibration and thus increases fairness and equality.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa283","type":"journal-article","created":{"date-parts":[[2020,10,26]],"date-time":"2020-10-26T20:39:23Z","timestamp":1603744763000},"page":"549-558","source":"Crossref","is-referenced-by-count":40,"title":["Addressing bias in prediction models by improving subpopulation calibration"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3400-235X","authenticated-orcid":false,"given":"Noam","family":"Barda","sequence":"first","affiliation":[{"name":"Clalit Research Institute, Clalit Health Services, Tel-Aviv, Israel"},{"name":"School of Public Health, Ben-Gurion University, Beer-Sheba, Israel"},{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA"}]},{"given":"Gal","family":"Yona","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel"}]},{"given":"Guy N","family":"Rothblum","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel"}]},{"given":"Philip","family":"Greenland","sequence":"additional","affiliation":[{"name":"Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA"}]},{"given":"Morton","family":"Leibowitz","sequence":"additional","affiliation":[{"name":"Clalit Research Institute, Clalit Health Services, Tel-Aviv, Israel"}]},{"given":"Ran","family":"Balicer","sequence":"additional","affiliation":[{"name":"Clalit Research Institute, Clalit Health Services, Tel-Aviv, Israel"},{"name":"School of Public Health, Ben-Gurion University, Beer-Sheba, Israel"}]},{"given":"Eitan","family":"Bachmat","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Ben-Gurion University, Beer-Sheba, Israel"}]},{"given":"Noa","family":"Dagan","sequence":"additional","affiliation":[{"name":"Clalit Research Institute, Clalit Health Services, Tel-Aviv, Israel"},{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA"},{"name":"Department of Computer Science, Ben-Gurion University, Beer-Sheba, Israel"}]}],"member":"286","published-online":{"date-parts":[[2020,11,24]]},"reference":[{"issue":"2","key":"2021030612313028700_ocaa283-B1","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.ijmedinf.2006.11.006","article-title":"Predictive data mining in clinical medicine: current issues and guidelines","volume":"77","author":"Bellazzi","year":"2008","journal-title":"Int J Med Inform"},{"issue":"1","key":"2021030612313028700_ocaa283-B2","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/0002-9149(76)90061-8","article-title":"A general cardiovascular risk profile: the Framingham Study","volume":"38","author":"Kannel","year":"1976","journal-title":"Am J Cardiol"},{"issue":"25","key":"2021030612313028700_ocaa283-B3","doi-asserted-by":"crossref","first-page":"2935","DOI":"10.1016\/j.jacc.2013.11.005","article-title":"2013 ACC\/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology\/American Heart Association Task Force on Practice Guidelines","volume":"63","author":"Goff","year":"2014","journal-title":"J Am Coll Cardiol"},{"issue":"4","key":"2021030612313028700_ocaa283-B4","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1007\/s00198-007-0543-5","article-title":"FRAX and the assessment of fracture probability in men and women from the UK","volume":"19","author":"Kanis","year":"2008","journal-title":"Osteoporos Int"},{"key":"2021030612313028700_ocaa283-B5","doi-asserted-by":"crossref","first-page":"i6755","DOI":"10.1136\/bmj.i6755","article-title":"External validation and comparison of three prediction tools for risk of osteoporotic fractures using data from population based electronic health records: retrospective cohort study","volume":"356","author":"Dagan","year":"2017","journal-title":"BMJ"},{"issue":"2","key":"2021030612313028700_ocaa283-B6","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1038\/sj.bjc.6604158","article-title":"The LLP risk model: an individual risk prediction model for lung cancer","volume":"98","author":"Cassidy","year":"2008","journal-title":"Br J Cancer"},{"issue":"8","key":"2021030612313028700_ocaa283-B7","first-page":"666","article-title":"Unequal treatment: confronting racial and ethnic disparities in health care","volume":"94","author":"Nelson","year":"2002","journal-title":"J Natl Med Assoc"},{"issue":"4","key":"2021030612313028700_ocaa283-B8","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1016\/S0033-3549(04)50253-4","article-title":"Defining cultural competence: a practical framework for addressing racial\/ethnic disparities in health and health care","volume":"118","author":"Betancourt","year":"2003","journal-title":"Public Health Rep"},{"issue":"19","key":"2021030612313028700_ocaa283-B9","doi-asserted-by":"crossref","first-page":"2579","DOI":"10.1001\/jama.283.19.2579","article-title":"Inequality in quality: addressing socioeconomic, racial, and ethnic disparities in health care","volume":"283","author":"Fiscella","year":"2000","journal-title":"JAMA"},{"issue":"7714","key":"2021030612313028700_ocaa283-B10","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1038\/d41586-018-05707-8","article-title":"AI can be sexist and racist\u2014it\u2019s time to make it fair","volume":"559","author":"Zou","year":"2018","journal-title":"Nature"},{"issue":"12","key":"2021030612313028700_ocaa283-B11","doi-asserted-by":"crossref","first-page":"866","DOI":"10.7326\/M18-1990","article-title":"Ensuring fairness in machine learning to advance health equity","volume":"169","author":"Rajkomar","year":"2018","journal-title":"Ann Intern Med"},{"issue":"6464","key":"2021030612313028700_ocaa283-B12","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"issue":"11","key":"2021030612313028700_ocaa283-B13","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","article-title":"Potential biases in machine learning algorithms using electronic health record data","volume":"178","author":"Gianfrancesco","year":"2018","journal-title":"JAMA Intern Med"},{"key":"2021030612313028700_ocaa283-B14","first-page":"1","article-title":"Counterfactual reasoning for fair clinical risk prediction","volume":"106","author":"Pfohl","year":"2019","journal-title":"Proc Mach Learn Res"},{"key":"2021030612313028700_ocaa283-B15","doi-asserted-by":"crossref","first-page":"g7873","DOI":"10.1136\/bmj.g7873","article-title":"High sensitivity cardiac troponin and the under-diagnosis of myocardial infarction in women: prospective cohort study","volume":"350","author":"Shah","year":"2015","journal-title":"BMJ"},{"issue":"2","key":"2021030612313028700_ocaa283-B16","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1001\/jama.286.2.180","article-title":"Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation","volume":"286","author":"D\u2019Agostino","year":"2001","journal-title":"JAMA"},{"issue":"4","key":"2021030612313028700_ocaa283-B17","doi-asserted-by":"crossref","first-page":"266","DOI":"10.7326\/M14-1281","article-title":"An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort","volume":"162","author":"DeFilippis","year":"2015","journal-title":"Ann Intern Med"},{"issue":"9","key":"2021030612313028700_ocaa283-B18","doi-asserted-by":"crossref","first-page":"871","DOI":"10.1093\/aje\/kwf113","article-title":"Multi-ethnic study of atherosclerosis: objectives and design","volume":"156","author":"Bild","year":"2002","journal-title":"Am J Epidemiol"},{"issue":"11","key":"2021030612313028700_ocaa283-B19","doi-asserted-by":"crossref","first-page":"786","DOI":"10.7326\/M16-1739","article-title":"Calibration of the pooled cohort equations for atherosclerotic cardiovascular disease: an update","volume":"165","author":"Cook","year":"2016","journal-title":"Ann Intern Med"},{"issue":"14","key":"2021030612313028700_ocaa283-B20","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1001\/jama.2017.12126","article-title":"Discrimination and calibration of clinical prediction models: users' guides to the medical literature","volume":"318","author":"Alba","year":"2017","journal-title":"JAMA"},{"key":"2021030612313028700_ocaa283-B21","author":"H\u00e9bert-Johnson"},{"issue":"3","key":"2021030612313028700_ocaa283-B22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v045.i03","article-title":"mice: Multivariate imputation by chained equations in R","volume":"45","author":"Buuren","year":"2011","journal-title":"J Stat Soft"},{"key":"2021030612313028700_ocaa283-B23","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-77244-8","volume-title":"Clinical Prediction Models","author":"Steyerberg","year":"2009"},{"issue":"18","key":"2021030612313028700_ocaa283-B24","doi-asserted-by":"crossref","first-page":"1837","DOI":"10.1161\/01.CIR.97.18.1837","article-title":"Prediction of coronary heart disease using risk factor categories","volume":"97","author":"Wilson","year":"1998","journal-title":"Circulation"},{"issue":"6","key":"2021030612313028700_ocaa283-B25","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1161\/CIRCULATIONAHA.107.699579","article-title":"General cardiovascular risk profile for use in primary care: the Framingham Heart Study","volume":"117","author":"D\u2019Agostino","year":"2008","journal-title":"Circulation"},{"key":"2021030612313028700_ocaa283-B26","volume-title":"Theory of Games and Economic Behavior (Commemorative Edition)","author":"Von Neumann","year":"2007"},{"key":"2021030612313028700_ocaa283-B27","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1038\/s41746-019-0156-3","article-title":"Translating clinical trial results into personalized recommendations by considering multiple outcomes and subjective views","volume":"2","author":"Dagan","year":"2019","journal-title":"NPJ Digit Med"},{"key":"2021030612313028700_ocaa283-B28","first-page":"3323","article-title":"Equality of opportunity in supervised learning","author":"Hardt","year":"2016","journal-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS'16)."},{"key":"2021030612313028700_ocaa283-B29","author":"Pfohl"},{"key":"2021030612313028700_ocaa283-B30","first-page":"3543","article-title":"Why is my classifier discriminatory?","author":"Chen","year":"2018","journal-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS'18)"},{"issue":"2","key":"2021030612313028700_ocaa283-B31","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1089\/big.2016.0047","article-title":"Fair prediction with disparate impact: a study of bias in recidivism prediction instruments","volume":"5","author":"Chouldechova","year":"2017","journal-title":"Big Data"},{"issue":"9","key":"2021030612313028700_ocaa283-B32","doi-asserted-by":"crossref","first-page":"874","DOI":"10.1056\/NEJMms2004740","article-title":"Hidden in plain sight - reconsidering the use of race correction in clinical algorithms","volume":"383","author":"Vyas","year":"2020","journal-title":"N Engl J Med"},{"issue":"4","key":"2021030612313028700_ocaa283-B33","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1002\/1096-8628(20010201)98:4<336::AID-AJMG1100>3.0.CO;2-7","article-title":"Consent to the use of stored DNA for genetics research: a survey of attitudes in the Jewish population","volume":"98","author":"Schwartz","year":"2001","journal-title":"Am J Med Genet"},{"issue":"3","key":"2021030612313028700_ocaa283-B34","first-page":"148","article-title":"Toward a framework of mutualism: the Jewish community in genetics research","volume":"1","author":"Rothenberg","year":"1998","journal-title":"Community Genet"},{"issue":"1","key":"2021030612313028700_ocaa283-B35","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1073\/pnas.88.1.310","article-title":"Evolution of a genetic disease in an ethnic isolate: beta-thalassemia in the Jews of Kurdistan","volume":"88","author":"Rund","year":"1991","journal-title":"Proc Natl Acad Sci USA"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocaa283\/34490967\/ocaa283.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/3\/549\/36428833\/ocaa283.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/3\/549\/36428833\/ocaa283.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,6]],"date-time":"2021-03-06T12:33:00Z","timestamp":1615033980000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/3\/549\/6000400"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,24]]},"references-count":35,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,11,24]]},"published-print":{"date-parts":[[2021,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa283","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,3,1]]},"published":{"date-parts":[[2020,11,24]]}}}