{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T06:48:15Z","timestamp":1775890095244,"version":"3.50.1"},"reference-count":62,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2023,2,24]],"date-time":"2023-02-24T00:00:00Z","timestamp":1677196800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Stanford Human-Centered Artificial Intelligence Seed"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,4,19]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Identifying ethical concerns with ML applications to healthcare (ML-HCA) before problems arise is now a stated goal of ML design oversight groups and regulatory agencies. Lack of accepted standard methodology for ethical analysis, however, presents challenges. In this case study, we evaluate use of a stakeholder \u201cvalues-collision\u201d approach to identify consequential ethical challenges associated with an ML-HCA for advanced care planning (ACP). Identification of ethical challenges could guide revision and improvement of the ML-HCA.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>We conducted semistructured interviews of the designers, clinician-users, affiliated administrators, and patients, and inductive qualitative analysis of transcribed interviews using modified grounded theory.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Seventeen stakeholders were interviewed. Five \u201cvalues-collisions\u201d\u2014where stakeholders disagreed about decisions with ethical implications\u2014were identified: (1) end-of-life workflow and how model output is introduced; (2) which stakeholders receive predictions; (3) benefit-harm trade-offs; (4) whether the ML design team has a fiduciary relationship to patients and clinicians; and, (5) how and if to protect early deployment research from external pressures, like news scrutiny, before research is completed.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>From these findings, the ML design team prioritized: (1) alternative workflow implementation strategies; (2) clarification that prediction was only evaluated for ACP need, not other mortality-related ends; and (3) shielding research from scrutiny until endpoint driven studies were completed.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>In this case study, our ethical analysis of this ML-HCA for ACP was able to identify multiple sites of intrastakeholder disagreement that mark areas of ethical and value tension. These findings provided a useful initial ethical screening.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocad022","type":"journal-article","created":{"date-parts":[[2023,2,24]],"date-time":"2023-02-24T15:09:58Z","timestamp":1677251398000},"page":"819-827","source":"Crossref","is-referenced-by-count":14,"title":["A framework to identify ethical concerns with ML-guided care workflows: a case study of mortality prediction to guide advance care planning"],"prefix":"10.1093","volume":"30","author":[{"given":"Diana","family":"Cagliero","sequence":"first","affiliation":[{"name":"Faculty of Medicine, University of Toronto , Toronto, Ontario, Canada"}]},{"given":"Natalie","family":"Deuitch","sequence":"additional","affiliation":[{"name":"Department of Genetics, Stanford University School of Medicine , Stanford, California, USA"},{"name":"National Institutes of Health, National Human Genome Research Institute Present address: , Bethesda, Maryland, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9385-7158","authenticated-orcid":false,"given":"Nigam","family":"Shah","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University School of Medicine , Palo Alto, California, USA"}]},{"given":"Chris","family":"Feudtner","sequence":"additional","affiliation":[{"name":"The Department of Medical Ethics, The Children\u2019s Hospital of Philadelphia , Philadelphia, Pennsylvania, USA"},{"name":"Departments of Pediatrics, Medical Ethics and Healthcare Policy, The Perelman School of Medicine, The University of Pennsylvania , Philadelphia, Pennsylvania, USA"}]},{"given":"Danton","family":"Char","sequence":"additional","affiliation":[{"name":"Division of Pediatric Cardiac Anesthesia, Department of Anesthesiology, Stanford University School of Medicine , Stanford, California, USA"},{"name":"Center for Biomedical Ethics, Stanford University School of Medicine , Stanford, California, USA"}]}],"member":"286","published-online":{"date-parts":[[2023,2,24]]},"reference":[{"key":"2023041909002515900_ocad022-B1","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1038\/s41746-018-0040-6","article-title":"Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices","volume":"1","author":"Abr\u00e0moff","year":"2018","journal-title":"NPJ Digital Med"},{"issue":"11","key":"2023041909002515900_ocad022-B2","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1016\/j.diii.2018.10.003","article-title":"Artificial intelligence and medical imaging 2018: French Radiology Community White Paper","volume":"99","author":"SFR-IA Group","year":"2018","journal-title":"Diagn Intervent Imaging"},{"key":"2023041909002515900_ocad022-B3","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.ajo.2020.02.022","article-title":"Autonomous AI: finding a safe, efficacious, and ethical path through the development process","volume":"214","author":"Abr\u00e0moff","year":"2020","journal-title":"Am J Ophthalmol"},{"key":"2023041909002515900_ocad022-B4","author":"Office of the Commissioner"},{"issue":"11","key":"2023041909002515900_ocad022-B5","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1056\/NEJMp1714229","article-title":"Implementing machine learning in health care - addressing ethical challenges","volume":"378","author":"Char","year":"2018","journal-title":"N Engl J Med"},{"issue":"1","key":"2023041909002515900_ocad022-B6","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1002\/hast.977","article-title":"Deep ethical learning: taking the interplay of human and artificial intelligence seriously","volume":"49","author":"Ho","year":"2019","journal-title":"Hastings Cent Rep"},{"issue":"2","key":"2023041909002515900_ocad022-B7","doi-asserted-by":"crossref","first-page":"E121","DOI":"10.1001\/amajethics.2019.121","article-title":"Ethical dimensions of using artificial intelligence in health care","volume":"21","author":"Rigby","year":"2019","journal-title":"AMA J Ethics"},{"issue":"3","key":"2023041909002515900_ocad022-B8","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1136\/bmjqs-2018-008370","article-title":"Artificial intelligence, bias and clinical safety","volume":"28","author":"Challen","year":"2019","journal-title":"BMJ Qual Saf"},{"issue":"6","key":"2023041909002515900_ocad022-B9","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1001\/jama.2019.21579","article-title":"Artificial intelligence in health care: a report from the National Academy of Medicine","volume":"323","author":"Matheny","year":"2020","journal-title":"JAMA"},{"issue":"1","key":"2023041909002515900_ocad022-B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12910-021-00577-8","article-title":"Artificial intelligence for good health: a scoping review of the ethics literature","volume":"22","author":"Murphy","year":"2021","journal-title":"BMC Med Ethics"},{"key":"2023041909002515900_ocad022-B11","author":"Adler-Milstein,","year":"2022"},{"key":"2023041909002515900_ocad022-B12","author":"European Commission Ethics Guidelines for Trustworthy AI"},{"issue":"4","key":"2023041909002515900_ocad022-B13","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1093\/jamia\/ocac006","article-title":"Defining AMIA\u2019s artificial intelligence principles","volume":"29","author":"Solomonides","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2023041909002515900_ocad022-B14","doi-asserted-by":"crossref","first-page":"113172","DOI":"10.1016\/j.socscimed.2020.113172","article-title":"The ethics of AI in health care: a mapping review","volume":"260","author":"Morley","year":"2020","journal-title":"Soc Sci Med"},{"issue":"11","key":"2023041909002515900_ocad022-B15","doi-asserted-by":"crossref","first-page":"e1002689","DOI":"10.1371\/journal.pmed.1002689","article-title":"Machine learning in medicine: addressing ethical challenges","volume":"15","author":"Vayena","year":"2018","journal-title":"PLoS Med"},{"issue":"8","key":"2023041909002515900_ocad022-B16","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1093\/jamia\/ocac095","article-title":"The imperative of applying ethical perspectives to biomedical and health informatics","volume":"29","author":"Bakken","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2023041909002515900_ocad022-B17","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1525\/jer.2011.6.1.25","article-title":"Consequences identification in forecasting and ethical decision-making","volume":"6","author":"Stenmark","year":"2011","journal-title":"J Emp Res Hum Res Ethics"},{"issue":"6464","key":"2023041909002515900_ocad022-B18","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"issue":"12","key":"2023041909002515900_ocad022-B19","doi-asserted-by":"crossref","first-page":"866","DOI":"10.7326\/M18-1990","article-title":"Ensuring fairness in machine learning to advance health equity","volume":"169","author":"Rajkomar","year":"2018","journal-title":"Ann Intern Med"},{"issue":"14","key":"2023041909002515900_ocad022-B20","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1001\/jama.2019.10306","article-title":"Making machine learning models clinically useful","volume":"322","author":"Shah","year":"2019","journal-title":"JAMA"},{"key":"2023041909002515900_ocad022-B21","author":"Miller","year":"2022"},{"issue":"11","key":"2023041909002515900_ocad022-B22","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1080\/15265161.2020.1819469","article-title":"Identifying ethical considerations for machine learning healthcare applications","volume":"20","author":"Char","year":"2020","journal-title":"Am J Bioethics"},{"issue":"2","key":"2023041909002515900_ocad022-B23","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1561\/1100000073","article-title":"Values and ethics in human-computer interaction","volume":"12","author":"Shilton","year":"2018","journal-title":"Found Trends Hum Comput Interact"},{"key":"2023041909002515900_ocad022-B24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11569-012-0141-7","article-title":"Anticipatory ethics for emerging technologies","volume":"6","author":"Brey","year":"2012","journal-title":"Nanoethics"},{"issue":"1","key":"2023041909002515900_ocad022-B25","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1089\/jpm.2015.0351","article-title":"Special report the growth of palliative care in U.S. hospitals: a status report","volume":"19","author":"Dumanovsky","year":"2016","journal-title":"J Palliat Med"},{"key":"2023041909002515900_ocad022-B26","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1377\/hlthaff.2020.01895","article-title":"Advance care planning for Medicare beneficiaries increased substantially, but prevalence remained low","volume":"40","author":"Palmer","year":"2021","journal-title":"Health Affairs"},{"key":"2023041909002515900_ocad022-B27","first-page":"20200911.401376","article-title":"Ethical considerations in the use of AI mortality predictions in the care of people with serious illness","author":"Lindvall","year":"2020","journal-title":"Health Affairs"},{"key":"2023041909002515900_ocad022-B28","doi-asserted-by":"crossref","first-page":"1143","DOI":"10.1111\/jgs.16539","article-title":"Rationing limited healthcare resources in the COVID19 era an beyond: ethical considerations regarding older adults","volume":"68","author":"Farrell","year":"2020","journal-title":"JAGS"},{"key":"2023041909002515900_ocad022-B29","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1186\/s12911-018-0677-8","article-title":"Improving palliative care with deep learning","volume":"18","author":"Avati","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"issue":"8","key":"2023041909002515900_ocad022-B30","doi-asserted-by":"crossref","first-page":"562","DOI":"10.7326\/M15-2970","article-title":"Two ways of knowing: big data and evidence-based medicine","volume":"164","author":"Sim","year":"2016","journal-title":"Ann Intern Med"},{"key":"2023041909002515900_ocad022-B31","doi-asserted-by":"crossref","first-page":"1216","DOI":"10.1056\/NEJMp1606181","article-title":"Artificial intelligence and the augmentation of health care decision-making","volume":"375","author":"Obermeyer","year":"2016","journal-title":"N Engl J Med"},{"key":"2023041909002515900_ocad022-B32","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1186\/s12916-017-0907-4","article-title":"How accurate is the \u2018Surprise Question\u2019 at identifying patients at the end of life? A systematic review and meta-analysis","volume":"15","author":"White","year":"2017","journal-title":"BMC Med"},{"key":"2023041909002515900_ocad022-B33","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1136\/bmjqs-2013-002035","article-title":"Suffering in silence: a qualitative study of second victims of adverse events","volume":"23","author":"Ullstr\u00f6m","year":"2014","journal-title":"BMJ Qual Saf"},{"key":"2023041909002515900_ocad022-B34","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1001\/jamasurgery.2013.403","article-title":"Surgeon-reported conflict with intensivists about postoperative goals of care","volume":"148","author":"Olson","year":"2013","journal-title":"JAMA Surg"},{"issue":"4","key":"2023041909002515900_ocad022-B35","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1007\/BF02599161","article-title":"The heart of darkness: the impact of perceived mistakes on physicians","volume":"7","author":"Christensen","year":"1992","journal-title":"J Gen Intern Med"},{"issue":"12","key":"2023041909002515900_ocad022-B36","doi-asserted-by":"crossref","first-page":"731","DOI":"10.1136\/jme.2010.037762","article-title":"Conflict and emotional exhaustion in obstetricians-gynaecologists: a national survey","volume":"36","author":"Yoon","year":"2010","journal-title":"J Med Ethics"},{"key":"2023041909002515900_ocad022-B37","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1186\/1472-6963-10-208","article-title":"Not all coping strategies are created equal: a mixed methods study exploring physicians' self reported coping strategies","volume":"10","author":"Lemaire","year":"2010","journal-title":"BMC Health Serv Res"},{"issue":"11","key":"2023041909002515900_ocad022-B38","doi-asserted-by":"crossref","first-page":"237","DOI":"10.4054\/DemRes.2010.22.11","article-title":"Reporting on first sexual experience: the importance of interviewer-respondent interaction","volume":"22","author":"Poulin","year":"2010","journal-title":"Demogr Res"},{"key":"2023041909002515900_ocad022-B39","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1186\/1471-2288-7-27","article-title":"A randomized trial of mailed questionnaires versus telephone interviews: response patterns in a survey","volume":"7","author":"Feveile","year":"2007","journal-title":"BMC Med Res Methodol"},{"key":"2023041909002515900_ocad022-B40","volume-title":"Basics of Qualitative Research","author":"Strauss","year":"1990"},{"key":"2023041909002515900_ocad022-B41","doi-asserted-by":"crossref","DOI":"10.4135\/9781412985833","volume-title":"Situational Analysis: Grounded Theory after the Postmodern Turn","author":"Clarke","year":"2005"},{"key":"2023041909002515900_ocad022-B42","volume-title":"Constructing Grounded Theory","author":"Charmaz","year":"2014","edition":"2nd ed"},{"issue":"1","key":"2023041909002515900_ocad022-B43","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1177\/1525822X02239569","article-title":"Techniques to identify themes","volume":"15","author":"Ryan","year":"2003","journal-title":"Field Methods"},{"issue":"12","key":"2023041909002515900_ocad022-B44","doi-asserted-by":"crossref","first-page":"2743","DOI":"10.1093\/jamia\/ocab195","article-title":"Guidance for publishing qualitative research in informatics","volume":"28","author":"Ancker","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"13","key":"2023041909002515900_ocad022-B45","doi-asserted-by":"crossref","first-page":"1216","DOI":"10.1056\/NEJMp1606181","article-title":"Predicting the future \u2014 big data, machine learning, and clinical medicine","volume":"375","author":"Obermeyer","year":"2016","journal-title":"N Engl J Med"},{"issue":"14","key":"2023041909002515900_ocad022-B46","doi-asserted-by":"crossref","first-page":"1347","DOI":"10.1056\/NEJMra1814259","article-title":"Machine learning in medicine","volume":"380","author":"Rajkomar","year":"2019","journal-title":"N Engl J Med"},{"issue":"1","key":"2023041909002515900_ocad022-B47","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1001\/jama.2018.18932","article-title":"Questions for artificial intelligence in health care","volume":"321","author":"Maddox","year":"2019","journal-title":"JAMA"},{"key":"2023041909002515900_ocad022-B48","doi-asserted-by":"crossref","DOI":"10.17226\/27111","volume-title":"Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril","author":"Matheny","year":"2019"},{"key":"2023041909002515900_ocad022-B49","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1001\/jama.2019.21579","article-title":"Artificial intelligence in health care: a report from the National Academy of Medicine","author":"Matheny","year":"2020","journal-title":"JAMA"},{"key":"2023041909002515900_ocad022-B50","first-page":"316","volume-title":"The Cambridge Handbook of Artificial Intelligence","author":"Bostrom","year":"2011"},{"key":"2023041909002515900_ocad022-B51","author":"Rosenberg","year":"2018"},{"key":"2023041909002515900_ocad022-B52","author":"Nicas","year":"2019"},{"key":"2023041909002515900_ocad022-B53","author":"Angwin","year":"2016"},{"key":"2023041909002515900_ocad022-B54","author":"Comfort","year":"2016"},{"key":"2023041909002515900_ocad022-B55","author":"Ornstein","year":"2018"},{"key":"2023041909002515900_ocad022-B56","author":"Ross","year":"2017"},{"issue":"8","key":"2023041909002515900_ocad022-B57","doi-asserted-by":"crossref","first-page":"1323","DOI":"10.1093\/jamia\/ocac065","article-title":"A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models","volume":"29","author":"Wang","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"2023041909002515900_ocad022-B58","doi-asserted-by":"crossref","first-page":"1334","DOI":"10.1093\/jamia\/ocac070","article-title":"An objective framework for evaluating unrecognized bias in medical AI models predicting COVID-19 outcomes","volume":"29","author":"Estri","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"20","key":"2023041909002515900_ocad022-B59","doi-asserted-by":"crossref","first-page":"1591","DOI":"10.1001\/jama.1995.03530200027032","article-title":"A controlled trial to improve care for seriously III hospitalized patients: the study to understand prognoses and preferences for outcomes and risks of treatments (SUPPORT)","volume":"274","author":"Connors","year":"1995","journal-title":"JAMA"},{"key":"2023041909002515900_ocad022-B60","doi-asserted-by":"crossref","first-page":"1148","DOI":"10.1136\/bmj.332.7550.1148","article-title":"Informed choice for screening: implications for evaluation","volume":"332","author":"Irwig","year":"2006","journal-title":"BMJ"},{"key":"2023041909002515900_ocad022-B61","author":"Jacobs","year":"2022"},{"key":"2023041909002515900_ocad022-B62","volume-title":"The Social Control of Technology","author":"Collingridge","year":"1980"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/5\/819\/49872997\/ocad022.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/5\/819\/49872997\/ocad022.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,6]],"date-time":"2023-12-06T18:18:02Z","timestamp":1701886682000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/5\/819\/7056673"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,24]]},"references-count":62,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,2,24]]},"published-print":{"date-parts":[[2023,4,19]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad022","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,5,1]]},"published":{"date-parts":[[2023,2,24]]}}}