{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T05:06:57Z","timestamp":1777093617223,"version":"3.51.4"},"reference-count":50,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T00:00:00Z","timestamp":1596067200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T00:00:00Z","timestamp":1596067200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The machine learning community has become alert to the ways that predictive algorithms can inadvertently introduce unfairness in decision-making. Herein, we discuss how concepts of algorithmic fairness might apply in healthcare, where predictive algorithms are being increasingly used to support decision-making. Central to our discussion is the distinction between algorithmic fairness and algorithmic bias. Fairness concerns apply specifically when algorithms are used to support polar decisions (i.e., where one pole of prediction leads to decisions that are generally more desired than the other), such as when predictions are used to allocate scarce health care resources to a group of patients that could all benefit. We review different fairness criteria and demonstrate their mutual incompatibility. Even when models are used to balance benefits-harms to make optimal decisions for individuals (i.e., for non-polar decisions)\u2013and fairness concerns are not germane\u2013model, data or sampling issues can lead to biased predictions that support decisions that are differentially harmful\/beneficial across groups. We review these potential sources of bias, and also discuss ways to diagnose and remedy algorithmic bias. We note that remedies for algorithmic fairness may be more problematic, since we lack agreed upon definitions of fairness. Finally, we propose a provisional framework for the evaluation of clinical prediction models offered for further elaboration and refinement. Given the proliferation of prediction models used to guide clinical decisions, developing consensus for how these concerns can be addressed should be prioritized.<\/jats:p>","DOI":"10.1038\/s41746-020-0304-9","type":"journal-article","created":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T10:24:33Z","timestamp":1596104673000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":227,"title":["Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities"],"prefix":"10.1038","volume":"3","author":[{"given":"Jessica K.","family":"Paulus","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9205-5070","authenticated-orcid":false,"given":"David M.","family":"Kent","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,7,30]]},"reference":[{"key":"304_CR1","unstructured":"Pearl, J. & Mackenzie, D. The Book of Why: The New Science of Cause and Effect. 1st edn. (Basic Books, 2018)."},{"key":"304_CR2","doi-asserted-by":"publisher","first-page":"1529","DOI":"10.1001\/jama.2018.11944","volume":"320","author":"K Wailoo","year":"2018","unstructured":"Wailoo, K. Historical aspects of race and medicine: the case of J. Marion Sims. JAMA 320, 1529\u20131530 (2018).","journal-title":"JAMA"},{"key":"304_CR3","doi-asserted-by":"publisher","first-page":"1417","DOI":"10.1001\/jama.2015.2434","volume":"313","author":"WF Wong","year":"2015","unstructured":"Wong, W. F., LaVeist, T. A. & Sharfstein, J. M. Achieving health equity by design. JAMA 313, 1417\u20131418 (2015).","journal-title":"JAMA"},{"key":"304_CR4","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1001\/jama.2015.9260","volume":"314","author":"DR Williams","year":"2015","unstructured":"Williams, D. R. & Wyatt, R. Racial bias in health care and health: challenges and opportunities. JAMA 314, 555\u2013556 (2015).","journal-title":"JAMA"},{"key":"304_CR5","doi-asserted-by":"publisher","first-page":"95","DOI":"10.18865\/ed.27.2.95","volume":"27","author":"X Zhang","year":"2017","unstructured":"Zhang, X. et al. Big data science: opportunities and challenges to address minority health and health disparities in the 21st century. Ethn. Dis. 27, 95\u2013106 (2017).","journal-title":"Ethn. Dis."},{"key":"304_CR6","unstructured":"116th Congress. Meeting of the legislative branch of the United States federal government (116th United States Congress, Washington, 2019)."},{"key":"304_CR7","doi-asserted-by":"publisher","first-page":"866","DOI":"10.7326\/M18-1990","volume":"169","author":"A Rajkomar","year":"2018","unstructured":"Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G. & Chin, M. H. Ensuring fairness in machine learning to advance health equity. Ann. Intern Med. 169, 866\u2013872 (2018).","journal-title":"Ann. Intern Med."},{"key":"304_CR8","doi-asserted-by":"publisher","first-page":"981","DOI":"10.1056\/NEJMp1714229","volume":"378","author":"DS Char","year":"2018","unstructured":"Char, D. S., Shah, N. H. & Magnus, D. Implementing machine learning in health care\u2013addressing ethical challenges. New Engl. J. Med. 378, 981\u2013983 (2018).","journal-title":"New Engl. J. Med."},{"key":"304_CR9","first-page":"56","volume":"81","author":"D Lovallo","year":"2003","unstructured":"Lovallo, D. & Kahneman, D. Delusions of success: how optimism undermines executives' decisions. Harv. Bus. Rev. 81, 56\u201363 (2003).","journal-title":"Harv. Bus. Rev."},{"key":"304_CR10","doi-asserted-by":"publisher","first-page":"1585","DOI":"10.1001\/jama.2012.3568","volume":"307","author":"DM Kent","year":"2012","unstructured":"Kent, D. M. & Shah, N. D. Risk models and patient-centered evidence: Should physicians expect one right answer? JAMA 307, 1585\u20131586 (2012).","journal-title":"JAMA"},{"key":"304_CR11","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1111\/j.1751-7176.2012.00592.x","volume":"14","author":"RH Stern","year":"2012","unstructured":"Stern, R. H. Individual Risk. J. Clin. Hypertens. 14, 261\u2013264 (2012).","journal-title":"J. Clin. Hypertens."},{"key":"304_CR12","unstructured":"Sowell, T. Discrimination and Disparities. 2nd edn. (Basic Books, 2019)."},{"key":"304_CR13","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1089\/big.2016.0047","volume":"5","author":"A Chouldechova","year":"2017","unstructured":"Chouldechova, A. Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data 5, 153\u2013163 (2017).","journal-title":"Big Data"},{"key":"304_CR14","unstructured":"Title VII of the Civil Rights Act of 1964. 7 241. https:\/\/legcounsel.house.gov\/Comps\/Civil%20Rights%20Act%20Of%201964.pdf (2015)."},{"key":"304_CR15","doi-asserted-by":"publisher","unstructured":"Barocas, S. & Selbst, A. D. Big Data\u2019s Disparate Impact. Calif. Law Rev. https:\/\/doi.org\/10.2139\/ssrn.2477899 (2016).","DOI":"10.2139\/ssrn.2477899"},{"key":"304_CR16","unstructured":"Kleinberg, J., Mullainathan, S. & Raghavan, M. Inherent trade-offs in the fair determination of risk scores. arXiv. arXiv:1609.05807v2 [cs. LG] (2016)."},{"key":"304_CR17","unstructured":"Verma, S. & Rubin, J. In ACM\/IEEE International Workshop on Software Fairness (ACM, Gothenburg, Sweden, 2018)."},{"key":"304_CR18","unstructured":"Mitchell, S., Potash, E. & Barocas, S. Prediction-based decisions and fairness: a catalogue of choices, assumptions, and definitions. arXiv. arXiv:1811.07867 [stat AP] (2018)."},{"key":"304_CR19","unstructured":"Corbett-Davis, S. & Goel, S. The measure and mismeasure of fairness: a critical review of fair machine learning. arXiv. arXIv:1808.00023 [stat AP] (2018)."},{"key":"304_CR20","unstructured":"Loftus, J. R., Russell, C., Kusner, M. J. & Silva, R. Causal reasoning for algorithmic fairness. arXiv. arXiv:1805.05859v1 [cs.AI] (2018)."},{"key":"304_CR21","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1080\/01621459.2000.10473924","volume":"95","author":"S Greenland","year":"2000","unstructured":"Greenland, S. Causal Analysis in the Health Sciences. J. Am. Stat. Assoc. 95, 286\u2013289 (2000).","journal-title":"J. Am. Stat. Assoc."},{"key":"304_CR22","unstructured":"Miller, D. In Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford, CA, 2017)."},{"key":"304_CR23","doi-asserted-by":"publisher","first-page":"1060","DOI":"10.1007\/s10618-017-0506-1","volume":"31","author":"I \u017dliobait\u0117","year":"2017","unstructured":"\u017dliobait\u0117, I. Measuring discrimination in algorithmic decision making. Data Min. Knowl. Discov. 31, 1060\u20131089 (2017).","journal-title":"Data Min. Knowl. Discov."},{"key":"304_CR24","doi-asserted-by":"publisher","first-page":"2208","DOI":"10.1056\/NEJMp1615069","volume":"376","author":"HG Welch","year":"2017","unstructured":"Welch, H. G. & Fisher, E. S. Income and cancer overdiagnosis - when too much care is harmful. New Engl. J. Med. 376, 2208\u20132209 (2017).","journal-title":"New Engl. J. Med."},{"key":"304_CR25","doi-asserted-by":"publisher","first-page":"180","DOI":"10.1001\/jama.286.2.180","volume":"286","author":"RB D\u2019Agostino","year":"2001","unstructured":"D\u2019Agostino, R. B., Grundy, S., Sullivan, L. M., Wilson, P. & Group, C. R. P. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA 286, 180\u2013187 (2001).","journal-title":"JAMA"},{"key":"304_CR26","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1136\/heartjnl-2013-304474","volume":"100","author":"T Tillin","year":"2014","unstructured":"Tillin, T. et al. Ethnicity and prediction of cardiovascular disease: performance of QRISK2 and Framingham scores in a UK tri-ethnic prospective cohort study (SABRE\u2014Southall And Brent REvisited). Heart 100, 60\u201367 (2014).","journal-title":"Heart"},{"key":"304_CR27","doi-asserted-by":"crossref","unstructured":"Martin, A. R. et al. Current Clinical Use Of Polygenic Scores Will Risk Exacerbating Health Disparities (Cold Spring Harbor Laboratory, bioRxiv, 2019).","DOI":"10.1101\/441261"},{"key":"304_CR28","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1002\/hast.736","volume":"47","author":"M Sabatello","year":"2017","unstructured":"Sabatello, M. & Appelbaum, P. S. The precision medicine nation. Hastings Cent. Rep. 47, 19\u201329 (2017).","journal-title":"Hastings Cent. Rep."},{"key":"304_CR29","doi-asserted-by":"publisher","unstructured":"Paulus, J. K. & Kent, D. M. Race and ethnicity: a part of the equation for personalized clinical decision making? Circ. Cardiovasc. Qual. Outcomes. https:\/\/doi.org\/10.1161\/CIRCOUTCOMES.117.003823 (2017).","DOI":"10.1161\/CIRCOUTCOMES.117.003823"},{"key":"304_CR30","doi-asserted-by":"publisher","first-page":"1429","DOI":"10.1007\/s11606-018-4475-x","volume":"33","author":"JK Paulus","year":"2018","unstructured":"Paulus, J. K., Wessler, B. S., Lundquist, C. M. & Kent, D. M. Effects of race are rarely included in clinical prediction models for cardiovascular disease. J. Gen. Intern. Med. 33, 1429\u20131430 (2018).","journal-title":"J. Gen. Intern. Med."},{"key":"304_CR31","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/S0895-4356(01)00427-9","volume":"55","author":"DH Solomon","year":"2002","unstructured":"Solomon, D. H. et al. Which patients with unstable angina or non-Q-wave myocardial infarction should have immediate cardiac catheterization? A clinical decision rule for predicting who will fail medical therapy. J. Clin. Epidemiol. 55, 121\u2013128 (2002).","journal-title":"J. Clin. Epidemiol."},{"key":"304_CR32","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","volume":"21","author":"EW Steyerberg","year":"2010","unstructured":"Steyerberg, E. W. et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21, 128\u2013138 (2010).","journal-title":"Epidemiology"},{"key":"304_CR33","unstructured":"WHO. Centre for Metabolic Bone Diseases (University of Sheffield, 2008)."},{"key":"304_CR34","doi-asserted-by":"publisher","first-page":"djw215","DOI":"10.1093\/jnci\/djw215","volume":"109","author":"MP Banegas","year":"2017","unstructured":"Banegas, M. P. et al. Projecting individualized absolute invasive breast cancer risk in US hispanic women. J. Natl. Cancer Inst. 109, djw215 (2017).","journal-title":"J. Natl. Cancer Inst."},{"key":"304_CR35","doi-asserted-by":"publisher","first-page":"1879","DOI":"10.1093\/jnci\/81.24.1879","volume":"81","author":"MH Gail","year":"1989","unstructured":"Gail, M. H. et al. Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J. Natl. Cancer Inst. 81, 1879\u20131886 (1989).","journal-title":"J. Natl. Cancer Inst."},{"key":"304_CR36","doi-asserted-by":"publisher","first-page":"1782","DOI":"10.1093\/jnci\/djm223","volume":"99","author":"MH Gail","year":"2007","unstructured":"Gail, M. H. et al. Projecting individualized absolute invasive breast cancer risk in African American women. J. Natl. Cancer Inst. 99, 1782\u20131792 (2007).","journal-title":"J. Natl. Cancer Inst."},{"key":"304_CR37","doi-asserted-by":"publisher","first-page":"951","DOI":"10.1093\/jnci\/djr154","volume":"103","author":"RK Matsuno","year":"2011","unstructured":"Matsuno, R. K. et al. Projecting individualized absolute invasive breast cancer risk in Asian and Pacific Islander American women. J. Natl. Cancer Inst. 103, 951\u2013961 (2011).","journal-title":"J. Natl. Cancer Inst."},{"key":"304_CR38","doi-asserted-by":"publisher","first-page":"1362","DOI":"10.1016\/j.urology.2014.02.035","volume":"83","author":"DP Ankerst","year":"2014","unstructured":"Ankerst, D. P. et al. Prostate cancer prevention trial risk calculator 2.0 for the prediction of low-versus high-grade prostate cancer. Urology 83, 1362\u20131368 (2014).","journal-title":"Urology"},{"key":"304_CR39","unstructured":"National Cancer Institute. Prostate Cancer Prevention Trial Investigators (UT Health San Antonio, 2012)."},{"key":"304_CR40","doi-asserted-by":"publisher","first-page":"2935","DOI":"10.1016\/j.jacc.2013.11.005","volume":"63","author":"DC Goff Jr.","year":"2014","unstructured":"Goff, D. C. Jr. et al. 2013 ACC\/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology\/American Heart Association Task Force on Practice Guidelines. J. Am. Coll. Cardiol. 63, 2935\u20132959 (2014).","journal-title":"J. Am. Coll. Cardiol."},{"key":"304_CR41","unstructured":"ACC. ASCVD Risk Estimator Investigators & The Million Heart Longitudinal ASCVD Risk Assessment Tool Investigators. (American College of Cardiology, 2019)."},{"key":"304_CR42","doi-asserted-by":"publisher","first-page":"S49","DOI":"10.1161\/01.cir.0000437741.48606.98","volume":"129","author":"DC Goff Jr.","year":"2014","unstructured":"Goff, D. C. Jr. et al. 2013 ACC\/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology\/American Heart Association Task Force on Practice Guidelines. Circulation 129, S49\u2013S73 (2014).","journal-title":"Circulation"},{"key":"304_CR43","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1146\/annurev.pharmtox.41.1.347","volume":"41","author":"LJ Lesko","year":"2001","unstructured":"Lesko, L. J. & Atkinson, A. J. Jr. Use of biomarkers and surrogate endpoints in drug development and regulatory decision making: criteria, validation, strategies. Annu. Rev. Pharmacol. Toxicol. 41, 347\u2013366 (2001).","journal-title":"Annu. Rev. Pharmacol. Toxicol."},{"key":"304_CR44","doi-asserted-by":"publisher","first-page":"605","DOI":"10.7326\/0003-4819-125-7-199610010-00011","volume":"125","author":"TR Fleming","year":"1996","unstructured":"Fleming, T. R. & DeMets, D. L. Surrogate end points in clinical trials: are we being misled? Ann. Intern. Med. 125, 605\u2013613 (1996).","journal-title":"Ann. Intern. Med."},{"key":"304_CR45","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","volume":"366","author":"Z Obermeyer","year":"2019","unstructured":"Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447\u2013453 (2019).","journal-title":"Science"},{"key":"304_CR46","doi-asserted-by":"crossref","unstructured":"Doshi-Velez, F. et al. Accountability of AI under the law: the role of explanation. arXiv. arXiv:1711.01134v2 [cs.AI] (2017).","DOI":"10.2139\/ssrn.3064761"},{"key":"304_CR47","unstructured":"Zafar, M. B., Valera, I., Rodriguez, M. G. & Gummadi, K. P. Fairness constraints: mechanisms for fair classification. arXiv:1507.05259 [stat.ML] https:\/\/arxiv.org\/pdf\/1507.05259 (2015)."},{"key":"304_CR48","doi-asserted-by":"publisher","unstructured":"Eneanya, N. D., Yang, W. & Reese, P. P. Reconsidering the consequences of using race to estimate kidney function. JAMA. https:\/\/doi.org\/10.1001\/jama.2019.5774 (2019).","DOI":"10.1001\/jama.2019.5774"},{"key":"304_CR49","doi-asserted-by":"publisher","first-page":"564","DOI":"10.1126\/science.aac4951","volume":"351","author":"M Yudell","year":"2016","unstructured":"Yudell, M., Roberts, D., DeSalle, R. & Tishkoff, S. Science and society: taking race out of human genetics. Science 351, 564\u2013565 (2016).","journal-title":"Science"},{"key":"304_CR50","unstructured":"Diakopoulos, N. et al. Principles for Accountable Algorithms and a Social Impact Statement for Algorithms:: FAT ML, https:\/\/www.fatml.org\/resources\/principles-for-accountable-algorithms."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0304-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0304-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0304-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:25:33Z","timestamp":1670379933000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0304-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,30]]},"references-count":50,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["304"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0304-9","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,30]]},"assertion":[{"value":"19 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 June 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 July 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"99"}}