{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T20:17:11Z","timestamp":1776889031088,"version":"3.51.2"},"reference-count":50,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2022,5,2]],"date-time":"2022-05-02T00:00:00Z","timestamp":1651449600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>The increasing translation of artificial intelligence (AI)\/machine learning (ML) models into clinical practice brings an increased risk of direct harm from modeling bias; however, bias remains incompletely measured in many medical AI applications. This article aims to provide a framework for objective evaluation of medical AI from multiple aspects, focusing on binary classification models.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Using data from over 56\u00a0000 Mass General Brigham (MGB) patients with confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), we evaluate unrecognized bias in 4 AI models developed during the early months of the pandemic in Boston, Massachusetts that predict risks of hospital admission, ICU admission, mechanical ventilation, and death after a SARS-CoV-2 infection purely based on their pre-infection longitudinal medical records. Models were evaluated both retrospectively and prospectively using model-level metrics of discrimination, accuracy, and reliability, and a novel individual-level metric for error.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We found inconsistent instances of model-level bias in the prediction models. From an individual-level aspect, however, we found most all models performing with slightly higher error rates for older patients.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>While a model can be biased against certain protected groups (ie, perform worse) in certain tasks, it can be at the same time biased towards another protected group (ie, perform better). As such, current bias evaluation studies may lack a full depiction of the variable effects of a model on its subpopulations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Only a holistic evaluation, a diligent search for unrecognized bias, can provide enough information for an unbiased judgment of AI bias that can invigorate follow-up investigations on identifying the underlying roots of bias and ultimately make a change.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocac070","type":"journal-article","created":{"date-parts":[[2022,4,27]],"date-time":"2022-04-27T15:32:51Z","timestamp":1651073571000},"page":"1334-1341","source":"Crossref","is-referenced-by-count":29,"title":["An objective framework for evaluating unrecognized bias in medical AI models predicting COVID-19 outcomes"],"prefix":"10.1093","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0204-8978","authenticated-orcid":false,"given":"Hossein","family":"Estiri","sequence":"first","affiliation":[{"name":"Laboratory of Computer Science, Massachusetts General Hospital , Boston, Massachusetts, USA"},{"name":"Department of Medicine, Massachusetts General Hospital , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4846-6059","authenticated-orcid":false,"given":"Zachary H","family":"Strasser","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Massachusetts General Hospital , Boston, Massachusetts, USA"},{"name":"Department of Medicine, Massachusetts General Hospital , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1210-2939","authenticated-orcid":false,"given":"Sina","family":"Rashidian","sequence":"additional","affiliation":[{"name":"Verily Life Sciences , Boston, Massachusetts, USA"},{"name":"Massachusetts General Hospital, Boston, MA 02114, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2043-1601","authenticated-orcid":false,"given":"Jeffrey G","family":"Klann","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Massachusetts General Hospital , Boston, Massachusetts, USA"},{"name":"Department of Medicine, Massachusetts General Hospital , Boston, Massachusetts, USA"},{"name":"Research Information Science and Computing, Mass General Brigham , Somerville, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6219-861X","authenticated-orcid":false,"given":"Kavishwar B","family":"Wagholikar","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Massachusetts General Hospital , Boston, Massachusetts, USA"},{"name":"Department of Medicine, Massachusetts General Hospital , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"suffix":"Jr","given":"Thomas H","family":"McCoy","sequence":"additional","affiliation":[{"name":"Center for Quantitative Health, Massachusetts General Hospital , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shawn N","family":"Murphy","sequence":"additional","affiliation":[{"name":"Laboratory of Computer Science, Massachusetts General Hospital , Boston, Massachusetts, USA"},{"name":"Research Information Science and Computing, Mass General Brigham , Somerville, Massachusetts, USA"},{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, Massachusetts, USA"},{"name":"Department of Neurology, Massachusetts General Hospital , Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2022,5,12]]},"reference":[{"issue":"11","key":"2022071310041807100_ocac070-B1","doi-asserted-by":"crossref","first-page":"e1002689","DOI":"10.1371\/journal.pmed.1002689","article-title":"Machine learning in medicine: addressing ethical challenges","volume":"15","author":"Vayena","year":"2018","journal-title":"PLoS Med"},{"issue":"11","key":"2022071310041807100_ocac070-B2","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1056\/NEJMp1714229","article-title":"Implementing machine learning in health care\u2014addressing ethical challenges","volume":"378","author":"Char","year":"2018","journal-title":"N Engl J Med"},{"issue":"5","key":"2022071310041807100_ocac070-B3","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1016\/j.medin.2019.11.003","article-title":"Big data analysis and machine learning in intensive care medicine: identifying new ethical and legal challenges","volume":"44","author":"Moratinos","year":"2020","journal-title":"Med Intensiva (Engl Ed)"},{"key":"2022071310041807100_ocac070-B4","doi-asserted-by":"publisher","author":"Hajjo","year":"2018","DOI":"10.1109\/cancercare.2018.8618186"},{"issue":"12","key":"2022071310041807100_ocac070-B5","doi-asserted-by":"crossref","first-page":"2020","DOI":"10.1093\/jamia\/ocaa094","article-title":"Latent bias and the implementation of artificial intelligence in medicine","volume":"27","author":"DeCamp","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2022071310041807100_ocac070-B6","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1145\/3376898","article-title":"A snapshot of the frontiers of fairness in machine learning","volume":"63","author":"Chouldechova","year":"2020","journal-title":"Commun ACM"},{"issue":"6464","key":"2022071310041807100_ocac070-B7","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"key":"2022071310041807100_ocac070-B8","doi-asserted-by":"crossref","first-page":"m363","DOI":"10.1136\/bmj.m363","article-title":"Can we trust AI not to further embed racial bias and prejudice?","volume":"368","author":"Noor","year":"2020","journal-title":"BMJ"},{"issue":"11","key":"2022071310041807100_ocac070-B9","doi-asserted-by":"crossref","first-page":"1247","DOI":"10.1001\/jamadermatol.2018.2348","article-title":"Machine learning and health care disparities in dermatology","volume":"154","author":"Adamson","year":"2018","journal-title":"JAMA Dermatol"},{"key":"2022071310041807100_ocac070-B10","author":"Clarke","year":"2019"},{"issue":"4","key":"2022071310041807100_ocac070-B11","doi-asserted-by":"crossref","first-page":"689","DOI":"10.1007\/s11023-018-9482-5","article-title":"AI4People\u2014an ethical framework for a good AI society: opportunities, risks, principles, and recommendations","volume":"28","author":"Floridi","year":"2018","journal-title":"Minds Mach (Dordr)"},{"issue":"6","key":"2022071310041807100_ocac070-B12","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1109\/TIFS.2012.2214212","article-title":"Face recognition performance: role of demographic information","volume":"7","author":"Klare","year":"2012","journal-title":"IEEE Trans Inf Forensic Secur"},{"key":"2022071310041807100_ocac070-B13","first-page":"169","author":"O\u2019Toole","year":"2012"},{"key":"2022071310041807100_ocac070-B14","doi-asserted-by":"publisher","author":"Hupont","year":"2019","DOI":"10.1109\/fg.2019.8756625"},{"issue":"6334","key":"2022071310041807100_ocac070-B15","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1126\/science.aal4230","article-title":"Semantics derived automatically from language corpora contain human-like biases","volume":"356","author":"Caliskan","year":"2017","journal-title":"Science"},{"key":"2022071310041807100_ocac070-B16","author":"Aran","year":"2019"},{"key":"2022071310041807100_ocac070-B17","author":"Rice","year":"2017"},{"issue":"12","key":"2022071310041807100_ocac070-B18","doi-asserted-by":"crossref","first-page":"866","DOI":"10.7326\/M18-1990","article-title":"Ensuring fairness in machine learning to advance health equity","volume":"169","author":"Rajkomar","year":"2018","journal-title":"Ann Intern Med"},{"issue":"17","key":"2022071310041807100_ocac070-B19","doi-asserted-by":"crossref","first-page":"1907","DOI":"10.1001\/archinte.166.17.1907","article-title":"Ethnic differences among patients with cutaneous melanoma","volume":"166","author":"Cormier","year":"2006","journal-title":"Arch Intern Med"},{"issue":"17","key":"2022071310041807100_ocac070-B20","doi-asserted-by":"crossref","first-page":"e012788","DOI":"10.1161\/JAHA.119.012788","article-title":"Artificial intelligence: practical primer for clinical research in cardiovascular disease","volume":"8","author":"Kagiyama","year":"2019","journal-title":"J Am Heart Assoc"},{"issue":"5","key":"2022071310041807100_ocac070-B21","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1016\/j.mayocp.2020.01.038","article-title":"Artificial intelligence in cardiology: present and future","volume":"95","author":"Lopez-Jimenez","year":"2020","journal-title":"Mayo Clin Proc"},{"issue":"12","key":"2022071310041807100_ocac070-B22","doi-asserted-by":"crossref","first-page":"e635","DOI":"10.1016\/S2589-7500(20)30249-1","article-title":"Addressing bias: artificial intelligence in cardiovascular medicine","volume":"2","author":"Tat","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"24","key":"2022071310041807100_ocac070-B23","doi-asserted-by":"crossref","first-page":"2377","DOI":"10.1001\/jama.2019.18058","article-title":"Addressing bias in artificial intelligence in health care","volume":"322","author":"Parikh","year":"2019","journal-title":"JAMA"},{"issue":"1","key":"2022071310041807100_ocac070-B24","doi-asserted-by":"crossref","first-page":"5322","DOI":"10.1038\/s41598-021-84781-x","article-title":"Individualized prediction of COVID-19 adverse outcomes with MLHO","volume":"11","author":"Estiri","year":"2021","journal-title":"Sci Rep"},{"issue":"1","key":"2022071310041807100_ocac070-B25","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1038\/s41746-021-00383-x","article-title":"Predicting COVID-19 mortality with electronic medical records","volume":"4","author":"Estiri","year":"2021","journal-title":"NPJ Digit Med"},{"key":"2022071310041807100_ocac070-B26","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1007\/978-3-030-59137-3_37","volume-title":"Artificial Intelligence in Medicine","author":"Estiri","year":"2020"},{"issue":"4","key":"2022071310041807100_ocac070-B27","doi-asserted-by":"crossref","first-page":"100051","DOI":"10.1016\/j.patter.2020.100051","article-title":"Transitive sequencing medical records for mining predictive and interpretable temporal representations","volume":"1","author":"Estiri","year":"2020","journal-title":"Patterns (N Y)"},{"issue":"6","key":"2022071310041807100_ocac070-B28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi","year":"2021","journal-title":"ACM Comput Surv"},{"issue":"2","key":"2022071310041807100_ocac070-B29","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1089\/big.2016.0047","article-title":"Fair prediction with disparate impact: a study of bias in recidivism prediction instruments","volume":"5","author":"Chouldechova","year":"2017","journal-title":"Big Data"},{"key":"2022071310041807100_ocac070-B30","first-page":"1","author":"Verma","year":"2018"},{"issue":"1","key":"2022071310041807100_ocac070-B31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","article-title":"Verification of forecasts expressed in terms of probability","volume":"78","author":"Brier","year":"1950","journal-title":"Mon Wea Rev"},{"key":"2022071310041807100_ocac070-B32","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.jbi.2017.10.008","article-title":"Beyond discrimination: a comparison of calibration methods and clinical usefulness of predictive models of readmission risk","volume":"76","author":"Walsh","year":"2017","journal-title":"J Biomed Inform"},{"key":"2022071310041807100_ocac070-B33","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.1214\/aos\/1013699998","article-title":"The control of the false discovery rate in multiple testing under dependency","volume":"29","author":"Benjamini","year":"2001","journal-title":"Ann. Stat"},{"issue":"4","key":"2022071310041807100_ocac070-B34","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1093\/jamia\/ocz228","article-title":"A tutorial on calibration measurements and calibration models for clinical prediction models","volume":"27","author":"Huang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2022071310041807100_ocac070-B35","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","article-title":"Calibration: the Achilles heel of predictive analytics","volume":"17","author":"Van Calster","year":"2019","journal-title":"BMC Med"},{"key":"2022071310041807100_ocac070-B36","doi-asserted-by":"crossref","DOI":"10.1201\/9781315370279","volume-title":"Generalized Additive Models: An Introduction with R","author":"Wood","year":"2017","edition":"2nd ed"},{"key":"2022071310041807100_ocac070-B37","author":"Wood","year":"2021"},{"issue":"1","key":"2022071310041807100_ocac070-B38","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1373\/clinchem.2007.096529","article-title":"Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve","volume":"54","author":"Cook","year":"2008","journal-title":"Clin Chem"},{"key":"2022071310041807100_ocac070-B39","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1056\/NEJMoa2021436","article-title":"Dexamethasone in hospitalized patients with Covid-19","volume":"384","author":"Horby","year":"2021","journal-title":"N Engl J Med"},{"issue":"19","key":"2022071310041807100_ocac070-B40","doi-asserted-by":"crossref","first-page":"1813","DOI":"10.1056\/NEJMoa2007764","article-title":"Remdesivir for the treatment of Covid-19\u2014final report","volume":"383","author":"Beigel","year":"2020","journal-title":"N Engl J Med"},{"key":"2022071310041807100_ocac070-B41","year":"2020"},{"key":"2022071310041807100_ocac070-B42","author":"Markos","year":"2021"},{"issue":"24","key":"2022071310041807100_ocac070-B43","doi-asserted-by":"crossref","first-page":"5709","DOI":"10.1093\/bioinformatics\/btaa943","article-title":"Calibrating variant-scoring methods for clinical decision making","volume":"36","author":"Benevenuta","year":"2021","journal-title":"Bioinformatics"},{"issue":"14","key":"2022071310041807100_ocac070-B44","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1001\/jama.2017.12126","article-title":"Discrimination and calibration of clinical prediction models: users\u2019 guides to the medical literature","volume":"318","author":"Alba","year":"2017","journal-title":"JAMA"},{"issue":"2","key":"2022071310041807100_ocac070-B45","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1177\/0272989X14547233","article-title":"Calibration of risk prediction models: impact on decision-analytic performance","volume":"35","author":"Van Calster","year":"2015","journal-title":"Med Decis Making"},{"issue":"7","key":"2022071310041807100_ocac070-B46","doi-asserted-by":"crossref","first-page":"e1001491","DOI":"10.1371\/journal.pmed.1001491","article-title":"Evaluation of prediction models for decision-making: beyond calibration and discrimination","volume":"10","author":"Holmberg","year":"2013","journal-title":"PLoS Med"},{"key":"2022071310041807100_ocac070-B47","first-page":"1","article-title":"Isotone optimization in R: pool-adjacent-violators algorithm (PAVA) and active set methods","volume":"32","author":"Mair","year":"2009","journal-title":"J Stat Softw"},{"key":"2022071310041807100_ocac070-B48","author":"Kaushal","year":"2020"},{"issue":"4\/5","key":"2022071310041807100_ocac070-B49","doi-asserted-by":"crossref","first-page":"4:1","DOI":"10.1147\/JRD.2019.2942287","article-title":"AI fairness 360: an extensible toolkit for detecting and mitigating algorithmic bias","volume":"63","author":"Bellamy","year":"2019","journal-title":"IBM J Res Dev"},{"key":"2022071310041807100_ocac070-B50","author":"Bird","year":"2020"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocac070\/43517281\/ocac070.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/8\/1334\/44829588\/ocac070.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/8\/1334\/44829588\/ocac070.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,13]],"date-time":"2022-07-13T06:10:10Z","timestamp":1657692610000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/8\/1334\/6576634"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,12]]},"references-count":50,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2022,5,12]]},"published-print":{"date-parts":[[2022,7,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac070","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.10.28.21265629","asserted-by":"object"}]},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,8,1]]},"published":{"date-parts":[[2022,5,12]]}}}