{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T08:56:23Z","timestamp":1793350583031,"version":"4.5.0"},"reference-count":37,"publisher":"American Medical Association (AMA)","issue":"23","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA"],"published-print":{"date-parts":[[2023,12,19]]},"abstract":"\n Importance<\/jats:title>\n Artificial intelligence (AI) could support clinicians when diagnosing hospitalized patients; however, systematic bias in AI models could worsen clinician diagnostic accuracy. Recent regulatory guidance has called for AI models to include explanations to mitigate errors made by models, but the effectiveness of this strategy has not been established.<\/jats:p>\n <\/jats:sec>\n \n Objectives<\/jats:title>\n To evaluate the impact of systematically biased AI on clinician diagnostic accuracy and to determine if image-based AI model explanations can mitigate model errors.<\/jats:p>\n <\/jats:sec>\n \n Design, Setting, and Participants<\/jats:title>\n Randomized clinical vignette survey study administered between April 2022 and January 2023 across 13 US states involving hospitalist physicians, nurse practitioners, and physician assistants.<\/jats:p>\n <\/jats:sec>\n \n Interventions<\/jats:title>\n Clinicians were shown 9 clinical vignettes of patients hospitalized with acute respiratory failure, including their presenting symptoms, physical examination, laboratory results, and chest radiographs. Clinicians were then asked to determine the likelihood of pneumonia, heart failure, or chronic obstructive pulmonary disease as the underlying cause(s) of each patient\u2019s acute respiratory failure. To establish baseline diagnostic accuracy, clinicians were shown 2 vignettes without AI model input. Clinicians were then randomized to see 6 vignettes with AI model input with or without AI model explanations. Among these 6 vignettes, 3 vignettes included standard-model predictions, and 3 vignettes included systematically biased model predictions.<\/jats:p>\n <\/jats:sec>\n \n Main Outcomes and Measures<\/jats:title>\n Clinician diagnostic accuracy for pneumonia, heart failure, and chronic obstructive pulmonary disease.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Median participant age was 34 years (IQR, 31-39) and 241 (57.7%) were female. Four hundred fifty-seven clinicians were randomized and completed at least 1 vignette, with 231 randomized to AI model predictions without explanations, and 226 randomized to AI model predictions with explanations. Clinicians\u2019 baseline diagnostic accuracy was 73.0% (95% CI, 68.3% to 77.8%) for the 3 diagnoses. When shown a standard AI model without explanations, clinician accuracy increased over baseline by 2.9 percentage points (95% CI, 0.5 to 5.2) and by 4.4 percentage points (95% CI, 2.0 to 6.9) when clinicians were also shown AI model explanations. Systematically biased AI model predictions decreased clinician accuracy by 11.3 percentage points (95% CI, 7.2 to 15.5) compared with baseline and providing biased AI model predictions with explanations decreased clinician accuracy by 9.1 percentage points (95% CI, 4.9 to 13.2) compared with baseline, representing a nonsignificant improvement of 2.3 percentage points (95% CI, \u22122.7 to 7.2) compared with the systematically biased AI model.<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n Although standard AI models improve diagnostic accuracy, systematically biased AI models reduced diagnostic accuracy, and commonly used image-based AI model explanations did not mitigate this harmful effect.<\/jats:p>\n <\/jats:sec>\n \n Trial Registration<\/jats:title>\n \n ClinicalTrials.gov Identifier:\n NCT06098950<\/jats:ext-link>\n <\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jama.2023.22295","type":"journal-article","created":{"date-parts":[[2023,12,19]],"date-time":"2023-12-19T11:01:21Z","timestamp":1702983681000},"page":"2275","source":"Crossref","is-referenced-by-count":248,"title":["Measuring the Impact of AI in the Diagnosis of Hospitalized Patients"],"prefix":"10.1001","volume":"330","author":[{"given":"Sarah","family":"Jabbour","sequence":"first","affiliation":[{"name":"Computer Science and Engineering, University of Michigan, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Fouhey","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of Michigan, Ann Arbor"},{"name":"Now with Computer Science Courant Institute, New York University, New York"},{"name":"Now with Electrical and Computer Engineering Tandon School of Engineering, New York University, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie","family":"Shepard","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of Michigan, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas S.","family":"Valley","sequence":"additional","affiliation":[{"name":"Pulmonary and Critical Care Medicine, Department of Internal Medicine, University of Michigan Medical School, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ella A.","family":"Kazerooni","sequence":"additional","affiliation":[{"name":"Department of Radiology, University of Michigan Medical School, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikola","family":"Banovic","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of Michigan, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jenna","family":"Wiens","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of Michigan, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael W.","family":"Sjoding","sequence":"additional","affiliation":[{"name":"Pulmonary and Critical Care Medicine, Department of Internal Medicine, University of Michigan Medical School, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","reference":[{"issue":"8","key":"joi230134r1","doi-asserted-by":"publisher","first-page":"1229","DOI":"10.1038\/s41591-020-0942-0","article-title":"Human-computer collaboration for skin cancer recognition.","volume":"26","author":"Tschandl","year":"2020","journal-title":"Nat Med"},{"issue":"22","key":"joi230134r2","doi-asserted-by":"publisher","first-page":"2402","DOI":"10.1001\/jama.2016.17216","article-title":"Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.","volume":"316","author":"Gulshan","year":"2016","journal-title":"JAMA"},{"issue":"5","key":"joi230134r3","doi-asserted-by":"publisher","first-page":"775","DOI":"10.1038\/s41591-021-01343-4","article-title":"Deep learning in histopathology: the path to the clinic.","volume":"27","author":"van der Laak","year":"2021","journal-title":"Nat Med"},{"issue":"1","key":"joi230134r4","doi-asserted-by":"publisher","first-page":"27988","DOI":"10.1038\/srep27988","article-title":"Multi-class texture analysis in colorectal cancer histology.","volume":"6","author":"Kather","year":"2016","journal-title":"Sci Rep"},{"key":"joi230134r5","first-page":"750","article-title":"Deep learning applied to chest x-rays: exploiting and preventing shortcuts.","volume":"126","author":"Jabbour","year":"2020","journal-title":"Proc Mach Learn Res"},{"issue":"6","key":"joi230134r6","doi-asserted-by":"publisher","first-page":"e406","DOI":"10.1016\/S2589-7500(22)00063-2","article-title":"AI recognition of patient race in medical imaging: a modelling study.","volume":"4","author":"Gichoya","year":"2022","journal-title":"Lancet Digit Health"},{"issue":"6464","key":"joi230134r7","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations.","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"issue":"6","key":"joi230134r8","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/S0147-9563(95)80020-4","article-title":"Gender bias in the diagnosis and treatment of coronary artery disease.","volume":"24","author":"Beery","year":"1995","journal-title":"Heart Lung"},{"issue":"1","key":"joi230134r9","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1038\/s41746-021-00385-9","article-title":"Do as AI say: susceptibility in deployment of clinical decision-aids.","volume":"4","author":"Gaube","year":"2021","journal-title":"NPJ Digit Med"},{"issue":"6","key":"joi230134r15","doi-asserted-by":"publisher","DOI":"10.1097\/CCE.0000000000000128","article-title":"The epidemiology of respiratory failure in the United States 2002-2017: a serial cross-sectional study.","volume":"2","author":"Kempker","year":"2020","journal-title":"Crit Care Explor"},{"issue":"2","key":"joi230134r16","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1097\/ACM.0b013e31823f71e6","article-title":"Relating faults in diagnostic reasoning with diagnostic errors and patient harm.","volume":"87","author":"Zwaan","year":"2012","journal-title":"Acad Med"},{"issue":"6","key":"joi230134r19","doi-asserted-by":"publisher","first-page":"1060","DOI":"10.1093\/jamia\/ocac030","article-title":"Combining chest x-rays and electronic health record (EHR) data using machine learning to diagnose acute respiratory failure.","volume":"29","author":"Jabbour","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"joi230134r20","doi-asserted-by":"publisher","first-page":"R82","DOI":"10.1186\/cc4926","article-title":"Acute respiratory failure in the elderly: etiology, emergency diagnosis and prognosis.","volume":"10","author":"Ray","year":"2006","journal-title":"Crit Care"},{"issue":"1","key":"joi230134r22","doi-asserted-by":"publisher","first-page":"27","DOI":"10.2307\/2684406","article-title":"A note on the delta method.","volume":"46","author":"Oehlert","year":"1992","journal-title":"Am Stat"},{"issue":"7","key":"joi230134r23","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1038\/s42256-021-00338-7","article-title":"AI for radiographic COVID-19 detection selects shortcuts over signal.","volume":"3","author":"DeGrave","year":"2021","journal-title":"Nat Mach Intell"},{"issue":"3","key":"joi230134r24","doi-asserted-by":"publisher","first-page":"R82","DOI":"10.1186\/cc4926","article-title":"Acute respiratory failure in the elderly: etiology, emergency diagnosis and prognosis.","volume":"10","author":"Ray","year":"2006","journal-title":"Crit Care"},{"issue":"CSCW1","key":"joi230134r26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3579460","article-title":"Being trustworthy is not enough: how untrustworthy artificial intelligence (AI) can deceive the end-users and gain their trust.","volume":"7","author":"Banovic","year":"2023","journal-title":"Proc ACM Hum Comput Interact"},{"key":"joi230134r27","first-page":"1","article-title":"What is AI literacy? competencies and design considerations.","author":"Long","year":"2020","journal-title":"Proc Conf Hum Factors Comput Syst"},{"key":"joi230134r28","first-page":"4768","article-title":"A unified approach to interpreting model predictions.","author":"Lundberg","year":"2017","journal-title":"Proc 31st Int Conf Neural Info Process Systems"},{"issue":"11","key":"joi230134r30","doi-asserted-by":"publisher","first-page":"12280","DOI":"10.1609\/aaai.v36i11.21491","article-title":"Expert-informed, user-centric explanations for machine learning.","volume":"36","author":"Pazzani","year":"2022","journal-title":"Proc AAAI Conf Art Intel"},{"issue":"4","key":"joi230134r31","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1001\/jama.2022.23867","article-title":"Prevention of bias and discrimination in clinical practice algorithms.","volume":"329","author":"Shachar","year":"2023","journal-title":"JAMA"},{"key":"joi230134r32","first-page":"47824","article-title":"Nondiscrimination in health programs and activities: final rule.","volume":"87","author":"Office for Civil Rights, Office of the Secretary of Health and Human Services","year":"2022","journal-title":"Fed Regist"},{"issue":"12","key":"joi230134r34","doi-asserted-by":"publisher","DOI":"10.1016\/j.xcrm.2022.100824","article-title":"Teaching artificial intelligence as a fundamental toolset of medicine.","volume":"3","author":"\u00d6tles","year":"2022","journal-title":"Cell Rep Med"},{"issue":"1","key":"joi230134r35","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1038\/s41746-020-0253-3","article-title":"Presenting machine learning model information to clinical end users with model facts labels.","volume":"3","author":"Sendak","year":"2020","journal-title":"NPJ Digit Med"},{"issue":"11","key":"joi230134r36","doi-asserted-by":"publisher","first-page":"e213524","DOI":"10.1001\/jamahealthforum.2021.3524","article-title":"Practice trends and characteristics of US hospitalists from 2012 to 2018.","volume":"2","author":"Ryskina","year":"2021","journal-title":"JAMA Health Forum"},{"key":"joi230134r21","first-page":"275","volume-title":"Markov Chain Monte Carlo in Practice","author":"Clayton","year":"1996"},{"key":"joi230134r10","doi-asserted-by":"crossref","unstructured":"Bu\u00e7inca? Z, Malaya? MB, Gajos? KZ. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. ArXiv. Preprint posted February 19, 2021. doi:10.48550\/arXiv.2102.09692","DOI":"10.1145\/3449287"},{"key":"joi230134r11","doi-asserted-by":"crossref","unstructured":"Vasconcelos? H, J\u00f6rke? M, Grunde-McLaughlin? M, Gerstenberg? T, Bernstein? MS, Krishna? R. Explanations can reduce overreliance on AI systems during decision-making. ArXiv. Preprint posted December 13, 2022. doi:10.48550\/arXiv.2212.06823","DOI":"10.1145\/3579605"},{"key":"joi230134r12","unstructured":"Clinical decision support software: guidance for industry and Food and Drug Administration staff. US Food and Drug Administration. September 28, 2022. Accessed March 1, 2023. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/clinical-decision-support-software"},{"key":"joi230134r13","doi-asserted-by":"crossref","unstructured":"Bhatt? U, Xiang? A, Sharma? S, . Explainable machine learning in deployment. Paper presented at: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency; January 27-30, 2020; Barcelona, Spain:648-657. doi:10.1145\/3351095.3375624","DOI":"10.1145\/3351095.3375624"},{"key":"joi230134r14","doi-asserted-by":"crossref","unstructured":"Selvaraju? RR, Cogswell? M, Das? A, Vedantam? R, Parikh? D, Batra? D. Grad-CAM: visual explanations from deep networks via gradient-based localization.? ArXiv. Preprint posted October 7, 2016. doi:10.48550\/arXiv.1610.02391","DOI":"10.1109\/ICCV.2017.74"},{"key":"joi230134r17","doi-asserted-by":"crossref","unstructured":"Vasconcelos? H, J\u00f6rke? M, Grunde-McLaughlin? M, Gerstenberg? T, Bernstein? M, Krishna? R. Explanations can reduce overreliance on AI systems during decision-making.? arXiv. Preprint posted December 13, 2022. doi:10.48550\/arXiv.2212.06823","DOI":"10.1145\/3579605"},{"key":"joi230134r18","unstructured":"Society of Hospital Medicine. Accessed August 28, 2023. https:\/\/www.hospitalmedicine.org"},{"key":"joi230134r25","doi-asserted-by":"crossref","unstructured":"Bai? B, Liang? J, Zhang? G, Li? H, Bai? K, Wang? F. Why attentions may not be interpretable?? arXiv. Preprint posted June 10, 2020. doi:10.48550\/arXiv.2006.05656","DOI":"10.1145\/3447548.3467307"},{"key":"joi230134r29","unstructured":"Ribeiro? MT, Singh? S, Guestrin? C. Model-agnostic interpretability of machine learning.? arXiv. Preprint posted June 16, 2016. doi:10.48550\/arXiv.1606.05386"},{"key":"joi230134r33","unstructured":"Blueprint for an AI Bill of Rights: making automated systems work for the American people. White House. Posted 2023. Accessed March 1, 2023. https:\/\/www.whitehouse.gov\/ostp\/ai-bill-of-rights"},{"key":"joi230134r37","unstructured":"Bubeck? S, Chandrasekaran? V, Eldan? R, . Sparks of artificial general intelligence: early experiments with GPT-4.? arXiv. Preprint March 22, 2023. doi:10.48550\/arXiv.2303.12712"}],"container-title":["JAMA"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jama\/articlepdf\/2812908\/jama_jabbour_2023_oi_230134_1702929668.72609.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,19]],"date-time":"2023-12-19T11:01:30Z","timestamp":1702983690000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jama\/fullarticle\/2812908"}},"subtitle":["A Randomized Clinical Vignette Survey Study"],"short-title":[],"issued":{"date-parts":[[2023,12,19]]},"references-count":37,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2023,12,19]]}},"URL":"https:\/\/doi.org\/10.1001\/jama.2023.22295","relation":{},"ISSN":["0098-7484"],"issn-type":[{"value":"0098-7484","type":"print"}],"subject":[],"published":{"date-parts":[[2023,12,19]]}}}