{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T03:24:52Z","timestamp":1781839492872,"version":"3.54.5"},"reference-count":31,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>While performance drift of clinical prediction models is well-documented, the potential for algorithmic biases to emerge post-deployment has had limited characterization. A better understanding of how temporal model performance may shift across subpopulations is required to incorporate fairness drift into model maintenance strategies.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We explore fairness drift in a national population over 11 years, with and without model maintenance aimed at sustaining population-level performance. We trained random forest models predicting 30-day post-surgical readmission, mortality, and pneumonia using 2013 data from US Department of Veterans Affairs facilities. We evaluated performance quarterly from 2014 to 2023 by self-reported race and sex. We estimated discrimination, calibration, and accuracy, and operationalized fairness using metric parity measured as the gap between disadvantaged and advantaged groups.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Our cohort included 1\u00a0739\u00a0666 surgical cases. We observed fairness drift in both the original and temporally updated models. Model updating had a larger impact on overall performance than fairness gaps. During periods of stable fairness, updating models at the population level increased, decreased, or did not impact fairness gaps. During periods of fairness drift, updating models restored fairness in some cases and exacerbated fairness gaps in others.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>This exploratory study highlights that algorithmic fairness cannot be assured through one-time assessments during model development. Temporal changes in fairness may take multiple forms and interact with model updating strategies in unanticipated ways.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Equitable and sustainable clinical artificial intelligence deployments will require novel methods to monitor algorithmic fairness, detect emerging bias, and adopt model updates that promote fairness.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf039","type":"journal-article","created":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T14:44:48Z","timestamp":1741877088000},"page":"845-854","source":"Crossref","is-referenced-by-count":29,"title":["Emerging algorithmic bias: fairness drift as the next dimension of model maintenance and sustainability"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0792-8867","authenticated-orcid":false,"given":"Sharon E","family":"Davis","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chad","family":"Dorn","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel J","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael E","family":"Matheny","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]},{"name":"Department of Biostatistics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]},{"name":"Tennessee Valley Healthcare System VA Medical Center, Veterans Health Administration , Nashville, TN 37212,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"key":"2025042203524855400_ocaf039-B1","doi-asserted-by":"publisher","first-page":"e074819","DOI":"10.1136\/bmj-2023-074819","article-title":"Evaluation of clinical prediction models (part 1): from development to external validation","volume":"384","author":"Collins","year":"2024","journal-title":"BMJ"},{"key":"2025042203524855400_ocaf039-B2","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1038\/s41746-020-0304-9","article-title":"Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities","volume":"3","author":"Paulus","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2025042203524855400_ocaf039-B3","first-page":"1","author":"Verma","year":"2018"},{"key":"2025042203524855400_ocaf039-B4","doi-asserted-by":"publisher","first-page":"1397","DOI":"10.1001\/jama.2020.9371","article-title":"The case for algorithmic stewardship for artificial intelligence and machine learning technologies","volume":"324","author":"Eaneff","year":"2020","journal-title":"JAMA"},{"key":"2025042203524855400_ocaf039-B5","doi-asserted-by":"publisher","first-page":"103621","DOI":"10.1016\/j.jbi.2020.103621","article-title":"An empirical characterization of fair machine learning for clinical risk prediction","volume":"113","author":"Pfohl","year":"2021","journal-title":"J Biomed Inform"},{"key":"2025042203524855400_ocaf039-B6","first-page":"453","author":"Beutel","year":"2019"},{"key":"2025042203524855400_ocaf039-B7","doi-asserted-by":"crossref","first-page":"4:1-4:15","DOI":"10.1147\/JRD.2019.2942287","article-title":"AI Fairness 360: an extensible toolkit for detecting and mitigating algorithmic bias","volume":"63","author":"Bellamy","year":"2019","journal-title":"IBM J Res Dev"},{"key":"2025042203524855400_ocaf039-B8","doi-asserted-by":"publisher","first-page":"549","DOI":"10.1093\/jamia\/ocaa283","article-title":"Addressing bias in prediction models by improving subpopulation calibration","volume":"28","author":"Barda","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203524855400_ocaf039-B9","doi-asserted-by":"publisher","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","article-title":"Potential biases in machine learning algorithms using electronic health record data","volume":"178","author":"Gianfrancesco","year":"2018","journal-title":"JAMA Intern Med"},{"key":"2025042203524855400_ocaf039-B10","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1353\/hpu.2021.0065","article-title":"A proposed framework on integrating health equity and racial justice into the artificial intelligence development lifecycle","volume":"32","author":"Dankwa-Mullan","year":"2021","journal-title":"J Health Care Poor Underserved"},{"key":"2025042203524855400_ocaf039-B11","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1093\/jamia\/ocac221","article-title":"Performance drift in a mortality prediction algorithm among patients with cancer during the SARS-CoV-2 pandemic","volume":"30","author":"Parikh","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203524855400_ocaf039-B12","doi-asserted-by":"publisher","first-page":"1162","DOI":"10.1016\/j.jacr.2022.05.030","article-title":"Machine learning model drift: predicting diagnostic imaging follow-up as a case example","volume":"19","author":"Lacson","year":"2022","journal-title":"J Am Coll Radiol"},{"key":"2025042203524855400_ocaf039-B13","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","article-title":"Calibration drift in regression and machine learning models for acute kidney injury","volume":"24","author":"Davis","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203524855400_ocaf039-B14","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1056\/NEJMc2104626","article-title":"The clinician and dataset shift in artificial intelligence","volume":"385","author":"Finlayson","year":"2021","journal-title":"N Engl J Med"},{"key":"2025042203524855400_ocaf039-B15","doi-asserted-by":"publisher","first-page":"e2135286","DOI":"10.1001\/jamanetworkopen.2021.35286","article-title":"Quantification of sepsis model alerts in 24 US hospitals before and during the COVID-19 pandemic","volume":"4","author":"Wong","year":"2021","journal-title":"JAMA Netw Open"},{"key":"2025042203524855400_ocaf039-B16","doi-asserted-by":"publisher","first-page":"1146","DOI":"10.1093\/ejcts\/ezs584","article-title":"Dynamic trends in cardiac surgery: why the logistic euroscore is no longer suitable for contemporary cardiac surgery and implications for future risk models","volume":"43","author":"Hickey","year":"2013","journal-title":"Eur J Cardio-Thoracic Surg"},{"key":"2025042203524855400_ocaf039-B17","author":"Davis","year":"2017"},{"key":"2025042203524855400_ocaf039-B18","doi-asserted-by":"crossref","first-page":"353","DOI":"10.3414\/ME11-02-0044","article-title":"Statistical process control for monitoring standardized mortality ratios of a classification tree model","volume":"51","author":"Minne","year":"2012","journal-title":"Meth Inform Med"},{"key":"2025042203524855400_ocaf039-B19","doi-asserted-by":"publisher","first-page":"e008635","DOI":"10.1161\/CIRCOUTCOMES.121.008635","article-title":"Maintaining a national acute kidney injury risk prediction model to support local quality benchmarking","volume":"15","author":"Davis","year":"2022","journal-title":"Circ Cardiovasc Qual Outcomes"},{"key":"2025042203524855400_ocaf039-B20","doi-asserted-by":"crossref","DOI":"10.17226\/27111","volume-title":"Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril","author":"Matheny","year":"2019"},{"key":"2025042203524855400_ocaf039-B21","doi-asserted-by":"crossref","DOI":"10.1787\/eedfee77-en","volume-title":"Artificial Intelligence in Society","author":"OECD","year":"2019"},{"key":"2025042203524855400_ocaf039-B22","doi-asserted-by":"publisher","first-page":"833","DOI":"10.1016\/j.jamcollsurg.2013.07.385","article-title":"Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons","volume":"217","author":"Bilimoria","year":"2013","journal-title":"J Am Coll Surg"},{"key":"2025042203524855400_ocaf039-B23","author":"American College of Surgeons National Surgical Quality Improvement Program","year":"2023"},{"key":"2025042203524855400_ocaf039-B24","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1016\/j.jamcollsurg.2013.02.027","article-title":"Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus","volume":"217","author":"Cohen","year":"2013","journal-title":"J Am Coll Surg"},{"key":"2025042203524855400_ocaf039-B25","doi-asserted-by":"publisher","first-page":"1024","DOI":"10.1097\/XCS.0000000000000556","article-title":"American College of Surgeons NSQIP Risk Calculator Accuracy using a machine learning algorithm compared with regression","volume":"236","author":"Liu","year":"2023","journal-title":"J Am Coll Surg"},{"key":"2025042203524855400_ocaf039-B26","author":"Yale New Haven Health Services Corporation\/Center for Outcomes Research & Evaluation","year":"2015"},{"key":"2025042203524855400_ocaf039-B27","doi-asserted-by":"publisher","first-page":"103611","DOI":"10.1016\/j.jbi.2020.103611","article-title":"Detection of calibration drift in clinical prediction models to inform model updating","volume":"112","author":"Davis","year":"2020","journal-title":"J Biomed Inform"},{"key":"2025042203524855400_ocaf039-B28","doi-asserted-by":"publisher","first-page":"1448","DOI":"10.1093\/jamia\/ocz127","article-title":"A nonparametric updating method to correct clinical prediction model drift","volume":"26","author":"Davis","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2025042203524855400_ocaf039-B29","doi-asserted-by":"publisher","first-page":"4581","DOI":"10.1038\/s41467-022-32186-3","article-title":"Addressing fairness in artificial intelligence for medical imaging","volume":"13","author":"Ricci Lara","year":"2022","journal-title":"Nat Commun"},{"key":"2025042203524855400_ocaf039-B30","first-page":"1039","author":"Pfohl","year":"2022"},{"key":"2025042203524855400_ocaf039-B31","doi-asserted-by":"publisher","first-page":"1195","DOI":"10.1093\/jamia\/ocae036","article-title":"Sustainable deployment of clinical prediction tools\u2014a 360 degrees approach to model maintenance","volume":"31","author":"Davis","year":"2024","journal-title":"J Am Med Inform Assoc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/5\/845\/62400687\/ocaf039.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/5\/845\/62400687\/ocaf039.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,22]],"date-time":"2025-04-22T07:52:59Z","timestamp":1745308379000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/5\/845\/8074959"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,13]]},"references-count":31,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,3,13]]},"published-print":{"date-parts":[[2025,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf039","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,5]]},"published":{"date-parts":[[2025,3,13]]}}}