{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,28]],"date-time":"2026-07-28T08:21:40Z","timestamp":1785226900854,"version":"3.55.0"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2023,4,22]],"date-time":"2023-04-22T00:00:00Z","timestamp":1682121600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["K99LM014097-01"],"award-info":[{"award-number":["K99LM014097-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01AG062499-01"],"award-info":[{"award-number":["R01AG062499-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM013995-01"],"award-info":[{"award-number":["R01LM013995-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,6,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad072","type":"journal-article","created":{"date-parts":[[2023,4,23]],"date-time":"2023-04-23T00:28:25Z","timestamp":1682209705000},"page":"1237-1245","source":"Crossref","is-referenced-by-count":382,"title":["Using AI-generated suggestions from ChatGPT to optimize clinical decision support"],"prefix":"10.1093","volume":"30","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aileen P","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Barron L","family":"Patterson","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan P","family":"Wanderer","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"},{"name":"Department of Anesthesiology, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1387-640X","authenticated-orcid":false,"given":"Robert W","family":"Turer","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, University of Texas Southwestern Medical Center , Dallas, Texas, USA"},{"name":"Clinical Informatics Center, University of Texas Southwestern Medical Center , Dallas, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1941-1817","authenticated-orcid":false,"given":"Scott D","family":"Nelson","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2292-9147","authenticated-orcid":false,"given":"Allison B","family":"McCoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5811-8915","authenticated-orcid":false,"given":"Dean F","family":"Sittig","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, University of Texas Health Science Center , Houston, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-145X","authenticated-orcid":false,"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,4,22]]},"reference":[{"issue":"7494","key":"2023062008370178600_ocad072-B1","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","article-title":"Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success","volume":"330","author":"Kawamoto","year":"2005","journal-title":"BMJ"},{"key":"2023062008370178600_ocad072-B2","doi-asserted-by":"crossref","first-page":"104037","DOI":"10.1016\/j.ijmedinf.2019.104037","article-title":"Quantifying the competitiveness of the electronic health record market and its implications for interoperability","volume":"136","author":"Sorace","year":"2020","journal-title":"Int J Med Inform"},{"key":"2023062008370178600_ocad072-B3","author":"Clinical Decision Support (CDS) | HealthIT.gov"},{"issue":"1","key":"2023062008370178600_ocad072-B4","doi-asserted-by":"crossref","first-page":"29","DOI":"10.7326\/0003-4819-157-1-201207030-00450","article-title":"Effect of clinical decision-support systems","volume":"157","author":"Bright","year":"2012","journal-title":"Ann Intern Med"},{"issue":"2","key":"2023062008370178600_ocad072-B5","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1111\/jrh.12043","article-title":"Association between clinical decision support system use and rural quality disparities in the treatment of pneumonia","volume":"30","author":"Mitchell","year":"2014","journal-title":"J Rural Health"},{"issue":"1","key":"2023062008370178600_ocad072-B6","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1097\/MLR.0000000000000251","article-title":"Eliminating health care disparities with mandatory clinical decision support","volume":"53","author":"Lau","year":"2015","journal-title":"Med Care"},{"issue":"3","key":"2023062008370178600_ocad072-B7","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.cvdhj.2020.11.001","article-title":"Rapid review: identification of digital health interventions in atherosclerotic-related cardiovascular disease populations to address racial, ethnic, and socioeconomic health disparities","volume":"1","author":"Thomas Craig","year":"2020","journal-title":"Cardiovasc Digit Health J"},{"key":"2023062008370178600_ocad072-B8","first-page":"38","article-title":"Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives","volume":"23","author":"Sirajuddin","year":"2009","journal-title":"J Healthc Inf Manag"},{"issue":"4","key":"2023062008370178600_ocad072-B9","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1016\/j.ijmedinf.2013.12.006","article-title":"What, if all alerts were specific \u2013 estimating the potential impact on drug interaction alert burden","volume":"83","author":"Seidling","year":"2014","journal-title":"Int J Med Inform"},{"issue":"2","key":"2023062008370178600_ocad072-B10","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1197\/jamia.M1809","article-title":"Overriding of drug safety alerts in computerized physician order entry","volume":"13","author":"van der Sijs","year":"2006","journal-title":"J Am Med Inform Assoc"},{"issue":"21","key":"2023062008370178600_ocad072-B11","doi-asserted-by":"crossref","first-page":"2625","DOI":"10.1001\/archinte.163.21.2625","article-title":"Physicians\u2019 decisions to override computerized drug alerts in primary care","volume":"163","author":"Weingart","year":"2003","journal-title":"Arch Intern Med"},{"issue":"11","key":"2023062008370178600_ocad072-B12","doi-asserted-by":"crossref","first-page":"1556","DOI":"10.1093\/jamia\/ocy112","article-title":"Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review","volume":"25","author":"Powers","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"2023062008370178600_ocad072-B13","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1177\/1060028019863419","article-title":"The effect of patient-specific drug-drug interaction alerting on the frequency of alerts: a pilot study","volume":"53","author":"Horn","year":"2019","journal-title":"Ann Pharmacother"},{"key":"2023062008370178600_ocad072-B14","doi-asserted-by":"crossref","first-page":"103453","DOI":"10.1016\/j.jbi.2020.103453","article-title":"Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow","volume":"106","author":"Olakotan","year":"2020","journal-title":"J Biomed Inform"},{"issue":"1","key":"2023062008370178600_ocad072-B15","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1197\/jamia.M2808","article-title":"Tiering drug-drug interaction alerts by severity increases compliance rates","volume":"16","author":"Paterno","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2023062008370178600_ocad072-B16","doi-asserted-by":"crossref","first-page":"e20174111","DOI":"10.1542\/peds.2017-4111","article-title":"Optimizing drug-drug interaction alerts using a multidimensional approach","volume":"143","author":"Daniels","year":"2019","journal-title":"Pediatrics"},{"issue":"2","key":"2023062008370178600_ocad072-B17","doi-asserted-by":"crossref","first-page":"144","DOI":"10.2146\/ajhp140095","article-title":"Reduction of clinical support warnings through recategorization of severity levels","volume":"72","author":"Parke","year":"2015","journal-title":"Am J Health Syst Pharm"},{"issue":"9","key":"2023062008370178600_ocad072-B18","doi-asserted-by":"crossref","first-page":"1481","DOI":"10.1097\/CCM.0000000000002580","article-title":"Technologic distractions (part 1): summary of approaches to manage alert quantity with intent to reduce alert fatigue and suggestions for alert fatigue metrics","volume":"45","author":"Kane-Gill","year":"2017","journal-title":"Crit Care Med"},{"issue":"6","key":"2023062008370178600_ocad072-B19","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1093\/jamia\/ocac027","article-title":"Clinician collaboration to improve clinical decision support: the Clickbusters initiative","volume":"29","author":"McCoy","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"26","key":"2023062008370178600_ocad072-B20","doi-asserted-by":"crossref","first-page":"2445","DOI":"10.1056\/NEJMp1303712","article-title":"From mindless to mindful practice \u2013 cognitive bias and clinical decision making","volume":"368","author":"Croskerry","year":"2013","journal-title":"N Engl J Med"},{"issue":"2","key":"2023062008370178600_ocad072-B21","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1136\/bmjqs-2016-005267","article-title":"Premature closure? Not so fast","volume":"26","author":"Dhaliwal","year":"2017","journal-title":"BMJ Qual Saf"},{"issue":"5","key":"2023062008370178600_ocad072-B22","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1093\/jamia\/ocab292","article-title":"The potential for leveraging machine learning to filter medication alerts","volume":"29","author":"Liu","year":"2022","journal-title":"J Am Med Informatics Assoc"},{"issue":"13","key":"2023062008370178600_ocad072-B23","doi-asserted-by":"crossref","first-page":"1086","DOI":"10.1093\/ajhp\/zxac045","article-title":"Improving the specificity of drug-drug interaction alerts: can it be done?","volume":"79","author":"Reese","year":"2022","journal-title":"Am J Health Syst Pharm"},{"key":"2023062008370178600_ocad072-B24","author":"ChatGPT: Optimizing Language Models for Dialogue","year":"2022"},{"key":"2023062008370178600_ocad072-B25","doi-asserted-by":"publisher","DOI":"10.1101\/2022.12.19.22283643","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2023062008370178600_ocad072-B26","author":"Jeblick","year":"2022"},{"key":"2023062008370178600_ocad072-B27","author":"Brown","year":"2020"},{"issue":"2","key":"2023062008370178600_ocad072-B28","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1197\/jamia.M3092","article-title":"A \u2018fundamental theorem\u2019 of biomedical informatics","volume":"16","author":"Friedman","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2023062008370178600_ocad072-B29","first-page":"273","article-title":"A crowdsourcing framework for medical data sets","volume":"2017","author":"Ye","year":"2018","journal-title":"Proc AMIA Jt Summits Transl Sci"},{"key":"2023062008370178600_ocad072-B30","doi-asserted-by":"publisher","first-page":"1906","DOI":"10.18653\/v1\/2020.acl-main.173","author":"Maynez","year":"2020"},{"key":"2023062008370178600_ocad072-B31","first-page":"126","article-title":"Another look at Likert scales","volume":"31","author":"Willits","year":"2016","journal-title":"J Rural Soc Sci"},{"key":"2023062008370178600_ocad072-B32","first-page":"11","article-title":"Five-point Likert items: t test versus Mann-Whitney-Wilcoxon","volume":"15","author":"de Winter","year":"2010","journal-title":"Pract Assess Res Eval"},{"key":"2023062008370178600_ocad072-B33","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/B978-0-12-818697-8.00110-2","volume-title":"Comprehensive Clinical Psychology","author":"Shou","year":"2022"},{"issue":"2","key":"2023062008370178600_ocad072-B34","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"issue":"2","key":"2023062008370178600_ocad072-B35","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1177\/1098214005283748","article-title":"A general inductive approach for analyzing qualitative evaluation data","volume":"27","author":"Thomas","year":"2006","journal-title":"Am J Eval"},{"issue":"1","key":"2023062008370178600_ocad072-B36","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"6","key":"2023062008370178600_ocad072-B37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/bib\/bbac409","article-title":"BioGPT: generative pre-trained transformer for biomedical text generation and mining","volume":"23","author":"Luo","year":"2022","journal-title":"Brief Bioinform"},{"key":"2023062008370178600_ocad072-B38","author":"Ouyang","year":"2022"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/7\/1237\/50634330\/ocad072.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/7\/1237\/50634330\/ocad072.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T08:39:07Z","timestamp":1687250347000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/7\/1237\/7136722"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,22]]},"references-count":38,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,4,22]]},"published-print":{"date-parts":[[2023,6,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad072","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,7,1]]},"published":{"date-parts":[[2023,4,22]]}}}