{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T16:18:43Z","timestamp":1791476323643,"version":"4.3.4"},"reference-count":21,"publisher":"American Diabetes Association","issue":"2","license":[{"start":{"date-parts":[[2024,9,6]],"date-time":"2024-09-06T00:00:00Z","timestamp":1725580800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.diabetesjournals.org\/journals\/pages\/license"}],"funder":[{"name":"National Institutes of Health\/National Cancer Institute"},{"DOI":"10.13039\/100006093","name":"Patient Centered Outcomes Research Institute","doi-asserted-by":"crossref","award":["CER-2017C3-9230"],"award-info":[{"award-number":["CER-2017C3-9230"]}],"id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100006093","name":"Patient Centered Outcomes Research Institute","doi-asserted-by":"crossref","award":["ME-2022C1-26378"],"award-info":[{"award-number":["ME-2022C1-26378"]}],"id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Cancer Center","award":["P30 CA008748"],"award-info":[{"award-number":["P30 CA008748"]}]}],"content-domain":{"domain":["diabetesjournals.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,2,1]]},"abstract":"<jats:sec>\n                  <jats:title>OBJECTIVE<\/jats:title>\n                  <jats:p>To explore how the commercially available large language model (LLM) GPT-4 compares to endocrinologists when addressing medical questions when there is uncertainty regarding the best answer.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>RESEARCH DESIGN AND METHODS<\/jats:title>\n                  <jats:p>This study compared responses from GPT-4 to responses from 31 endocrinologists using hypothetical clinical vignettes focused on diabetes, specifically examining the prescription of metformin versus alternative treatments. The primary outcome was the choice between metformin and other treatments.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>RESULTS<\/jats:title>\n                  <jats:p>With a simple prompt, GPT-4 chose metformin in 12% (95% CI 7.9\u201317%) of responses, compared with 31% (95% CI 23\u201339%) of endocrinologist responses. After modifying the prompt to encourage metformin use, the selection of metformin by GPT-4 increased to 25% (95% CI 22\u201328%). GPT-4 rarely selected metformin in patients with impaired kidney function, or a history of gastrointestinal distress (2.9% of responses, 95% CI 1.4\u20135.5%). In contrast, endocrinologists often prescribed metformin even in patients with a history of gastrointestinal distress (21% of responses, 95% CI 12\u201336%). GPT-4 responses showed low variability on repeated runs except at intermediate levels of kidney function.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>CONCLUSIONS<\/jats:title>\n                  <jats:p>In clinical scenarios with no single right answer, GPT-4\u2019s responses were reasonable, but differed from endocrinologists\u2019 responses in clinically important ways. Value judgments are needed to determine when these differences should be addressed by adjusting the model. We recommend against reliance on LLM output until it is shown to align not just with clinical guidelines but also with patient and clinician preferences, or it demonstrates improvement in clinical outcomes over standard of care.<\/jats:p>\n               <\/jats:sec>","DOI":"10.2337\/dc24-1067","type":"journal-article","created":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T15:30:02Z","timestamp":1725895802000},"page":"185-192","update-policy":"https:\/\/doi.org\/10.2337\/ada-journal-policies","source":"Crossref","is-referenced-by-count":15,"title":["Large Language Model GPT-4 Compared to Endocrinologist Responses on Initial Choice of Glucose-Lowering Medication Under Conditions of Clinical Uncertainty"],"prefix":"10.2337","volume":"48","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0259-969X","authenticated-orcid":false,"given":"James H.","family":"Flory","sequence":"first","affiliation":[{"name":"1Memorial Sloan Kettering Cancer Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jessica S.","family":"Ancker","sequence":"additional","affiliation":[{"name":"2Vanderbilt University Medical Center, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scott Y.H.","family":"Kim","sequence":"additional","affiliation":[{"name":"3National Institutes of Health, Bethesda, MD"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gilad","family":"Kuperman","sequence":"additional","affiliation":[{"name":"1Memorial Sloan Kettering Cancer Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aleksandr","family":"Petrov","sequence":"additional","affiliation":[{"name":"1Memorial Sloan Kettering Cancer Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Vickers","sequence":"additional","affiliation":[{"name":"1Memorial Sloan Kettering Cancer Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1167","published-online":{"date-parts":[[2024,9,6]]},"reference":[{"key":"2025012220115307700_B1","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1055\/a-2281-7092","article-title":"A survey of clinicians' views of the utility of large language models","volume":"15","author":"Spotnitz","year":"2024","journal-title":"Appl Clin Inform"},{"key":"2025012220115307700_B2","doi-asserted-by":"crossref","first-page":"e2336483","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of Chatbot responses to physician questions","volume":"6","author":"Goodman","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2025012220115307700_B3","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2025012220115307700_B4","article-title":"Capabilities of GPT-4 on medical challenge problems","author":"Nori","year":"2023"},{"key":"2025012220115307700_B5","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025012220115307700_B6","doi-asserted-by":"crossref","first-page":"e230424","DOI":"10.1148\/radiol.230424","article-title":"Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT","volume":"307","author":"Haver","year":"2023","journal-title":"Radiology"},{"key":"2025012220115307700_B7","doi-asserted-by":"crossref","first-page":"e49324","DOI":"10.2196\/49324","article-title":"Large language models for therapy recommendations across 3 clinical specialties: Comparative study","volume":"25","author":"Wilhelm","year":"2023","journal-title":"J Med Internet Res"},{"key":"2025012220115307700_B8","doi-asserted-by":"crossref","first-page":"4","DOI":"10.7812\/TPP\/13-036","article-title":"Comparative effectiveness topics from a large, integrated delivery system","volume":"17","author":"Danforth","year":"2013","journal-title":"Perm J"},{"key":"2025012220115307700_B9","doi-asserted-by":"crossref","first-page":"S158","DOI":"10.2337\/dc24-S009","article-title":"9. Pharmacologic approaches to glycemic treatment: Standards of Care in Diabetes-2024","volume":"47","author":"American Diabetes Association Professional Practice Committee","year":"2024","journal-title":"Diabetes Care"},{"key":"2025012220115307700_B10","doi-asserted-by":"crossref","first-page":"1476","DOI":"10.1007\/s11606-022-07838-1","article-title":"Prescriber uncertainty as opportunity to improve care of type 2 diabetes with chronic kidney disease: Mixed methods study","volume":"38","author":"Flory","year":"2023","journal-title":"J Gen Intern Med"},{"key":"2025012220115307700_B11","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1016\/j.eprac.2023.02.001","article-title":"American Association of Clinical Endocrinology consensus statement: Comprehensive type 2 diabetes management algorithm - 2023 update","volume":"29","author":"Samson","year":"2023","journal-title":"Endocr Pract"},{"key":"2025012220115307700_B12","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1056\/NEJM198707163170304","article-title":"Equipoise and the ethics of clinical research","volume":"317","author":"Freedman","year":"1987","journal-title":"N Engl J Med"},{"key":"2025012220115307700_B13","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1186\/1745-6215-12-258","article-title":"A statistical framework for quantifying clinical equipoise for individual cases during randomized controlled surgical trials","volume":"12","author":"Parsons","year":"2011","journal-title":"Trials"},{"key":"2025012220115307700_B14","doi-asserted-by":"crossref","first-page":"605","DOI":"10.1177\/1740774516656945","article-title":"A framework for analysis of research risks and benefits to participants in standard of care pragmatic clinical trials","volume":"13","author":"Chen","year":"2016","journal-title":"Clin Trials"},{"key":"2025012220115307700_B15","doi-asserted-by":"crossref","first-page":"1405","DOI":"10.1161\/CIRCULATIONAHA.116.023041","article-title":"Citizen petition to the US Food and Drug Administration to change prescribing guidelines: The metformin experience","volume":"134","author":"Lipska","year":"2016","journal-title":"Circulation"},{"key":"2025012220115307700_B16","article-title":"A framework for few-shot language model evaluation","author":"Gao","year":"2023"},{"key":"2025012220115307700_B17","doi-asserted-by":"crossref","first-page":"148","DOI":"10.3390\/medicina60010148","article-title":"Chain of thought utilization in large language models and application in nephrology","volume":"60","author":"Miao","year":"2024","journal-title":"Medicina (Kaunas)"},{"key":"2025012220115307700_B18","volume-title":"Categorical Data Analysis.","author":"Agresti","year":"2013","edition":"3rd ed"},{"key":"2025012220115307700_B19","doi-asserted-by":"crossref","first-page":"1433","DOI":"10.2147\/PPA.S211614","article-title":"Overcoming barriers to the use of metformin: patient and provider perspectives","volume":"13","author":"Flory","year":"2019","journal-title":"Patient Prefer Adherence"},{"key":"2025012220115307700_B20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3597307","article-title":"Biases in large language models: Origins, inventory, and discussion","volume":"15","author":"Navigli","year":"2023","journal-title":"J Data and Information Quality"},{"key":"2025012220115307700_B21","article-title":"Self-consistency improves chain of thought reasoning in language models","author":"Wang","year":"2022","journal-title":"arXiv"}],"container-title":["Diabetes Care"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/diabetesjournals.org\/care\/article-pdf\/48\/2\/185\/781036\/dc241067.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/diabetesjournals.org\/care\/article-pdf\/48\/2\/185\/781036\/dc241067.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,22]],"date-time":"2025-01-22T20:12:48Z","timestamp":1737576768000},"score":1,"resource":{"primary":{"URL":"https:\/\/diabetesjournals.org\/care\/article\/48\/2\/185\/157187\/Large-Language-Model-GPT-4-Compared-to"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,6]]},"references-count":21,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,9,6]]},"published-print":{"date-parts":[[2025,2,1]]}},"URL":"https:\/\/doi.org\/10.2337\/dc24-1067","relation":{},"ISSN":["0149-5992","1935-5548"],"issn-type":[{"value":"0149-5992","type":"print"},{"value":"1935-5548","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,6]]}}}