{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T01:32:47Z","timestamp":1793410367104,"version":"4.5.0"},"reference-count":19,"publisher":"American Medical Association (AMA)","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"\n Importance<\/jats:title>\n The introduction of large language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4; OpenAI), has generated significant interest in health care, yet studies evaluating their performance in a clinical setting are lacking. Determination of clinical acuity, a measure of a patient\u2019s illness severity and level of required medical attention, is one of the foundational elements of medical reasoning in emergency medicine.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To determine whether an LLM can accurately assess clinical acuity in the emergency department (ED).<\/jats:p>\n <\/jats:sec>\n \n Design, Setting, and Participants<\/jats:title>\n This cross-sectional study identified all adult ED visits from January 1, 2012, to January 17, 2023, at the University of California, San Francisco, with a documented Emergency Severity Index (ESI) acuity level (immediate, emergent, urgent, less urgent, or nonurgent) and with a corresponding ED physician note. A sample of 10\u202f000 pairs of ED visits with nonequivalent ESI scores, balanced for each of the 10 possible pairs of 5 ESI scores, was selected at random.<\/jats:p>\n <\/jats:sec>\n \n Exposure<\/jats:title>\n The potential of the LLM to classify acuity levels of patients in the ED based on the ESI across 10\u202f000 patient pairs. Using deidentified clinical text, the LLM was queried to identify the patient with a higher-acuity presentation within each pair based on the patients\u2019 clinical history. An earlier LLM was queried to allow comparison with this model.<\/jats:p>\n <\/jats:sec>\n \n Main Outcomes and Measures<\/jats:title>\n Accuracy score was calculated to evaluate the performance of both LLMs across the 10\u202f000-pair sample. A 500-pair subsample was manually classified by a physician reviewer to compare performance between the LLMs and human classification.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n From a total of 251\u202f401 adult ED visits, a balanced sample of 10\u202f000 patient pairs was created wherein each pair comprised patients with disparate ESI acuity scores. Across this sample, the LLM correctly inferred the patient with higher acuity for 8940 of 10\u202f000 pairs (accuracy,\u20090.89 [95% CI, 0.89-0.90]). Performance of the comparator LLM (accuracy,\u20090.84 [95% CI, 0.83-0.84]) was below that of its successor. Among the 500-pair subsample that was also manually classified, LLM performance (accuracy,\u20090.88 [95% CI, 0.86-0.91]) was comparable with that of the physician reviewer (accuracy,\u20090.86 [95% CI, 0.83-0.89]).<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n In this cross-sectional study of 10\u202f000 pairs of ED visits, the LLM accurately identified the patient with higher acuity when given pairs of presenting histories extracted from patients\u2019 first ED documentation. These findings suggest that the integration of an LLM into ED workflows could enhance triage processes while maintaining triage quality and warrants further investigation.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2024.8895","type":"journal-article","created":{"date-parts":[[2024,5,7]],"date-time":"2024-05-07T11:08:07Z","timestamp":1715080087000},"page":"e248895","source":"Crossref","is-referenced-by-count":161,"title":["Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department"],"prefix":"10.1001","volume":"7","author":[{"given":"Christopher Y. K.","family":"Williams","sequence":"first","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Travis","family":"Zack","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brenda Y.","family":"Miao","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Madhumita","family":"Sushil","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michelle","family":"Wang","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron E.","family":"Kornblith","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco"},{"name":"Department of Emergency Medicine, University of California, San Francisco"},{"name":"Department of Pediatrics, University of California, San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Atul J.","family":"Butte","sequence":"additional","affiliation":[{"name":"Bakar Computational Health Sciences Institute, University of California, San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2024,5,7]]},"reference":[{"issue":"2","key":"zoi240332r3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models.","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"issue":"1","key":"zoi240332r4","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a generative artificial intelligence model in a complex diagnostic challenge.","volume":"330","author":"Kanjee","year":"2023","journal-title":"JAMA"},{"issue":"12","key":"zoi240332r5","doi-asserted-by":"publisher","first-page":"1454","DOI":"10.1111\/acem.2012.19.issue-12","article-title":"Assessing diagnostic reasoning: a consensus statement summarizing theory, practice, and future needs.","volume":"19","author":"Ilgen","year":"2012","journal-title":"Acad Emerg Med"},{"issue":"3","key":"zoi240332r6","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooad045","article-title":"A certified de-identification system for all clinical text documents for information extraction at scale.","volume":"6","author":"Radhakrishnan","year":"2023","journal-title":"JAMIA Open"},{"issue":"1","key":"zoi240332r8","doi-asserted-by":"publisher","first-page":"16492","DOI":"10.1038\/s41598-023-43436-9","article-title":"Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments.","volume":"13","author":"Brin","year":"2023","journal-title":"Sci Rep"},{"issue":"3","key":"zoi240332r9","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231362","article-title":"Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer.","volume":"308","author":"Fink","year":"2023","journal-title":"Radiology"},{"issue":"1","key":"zoi240332r11","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1038\/s41746-023-00957-x","article-title":"Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models.","volume":"6","author":"Alsentzer","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"3","key":"zoi240332r17","doi-asserted-by":"publisher","DOI":"10.1056\/CAT.23.0404","article-title":"Ambient artificial intelligence scribes to alleviate the burden of clinical documentation.","volume":"5","author":"Tierney","year":"2024","journal-title":"NEJM Catal"},{"issue":"11","key":"zoi240332r19","doi-asserted-by":"publisher","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","article-title":"Potential biases in machine learning algorithms using electronic health record data.","volume":"178","author":"Gianfrancesco","year":"2018","journal-title":"JAMA Intern Med"},{"key":"zoi240332r1","unstructured":"OpenAI. Introducing ChatGPT. Accessed March 18, 2023. https:\/\/openai.com\/blog\/chatgpt"},{"key":"zoi240332r2","unstructured":"OpenAI, Achiam? J, Adler? S, . GPT-4 technical report.? arXiv. Preprint posted online March 27, 2023. doi:10.48550\/arXiv.2303.08774"},{"key":"zoi240332r7","unstructured":"Emergency Nurses Association. Emergency Severity Index Handbook. 5th ed. Emergency Nurses Association; 2023. Accessed January 30, 2024. https:\/\/californiaena.org\/wp-content\/uploads\/2023\/05\/ESI-Handbook-5th-Edition-3-2023.pdf"},{"key":"zoi240332r10","unstructured":"Singhal? K, Tu? T, Gottweis? J, . Towards expert-level medical question answering with large language models.? arXiv. Preprint posted online May 16, 2023."},{"key":"zoi240332r12","unstructured":"Sushil? M, Kennedy? VE, Miao? BY, Mandair? D, Zack? T, Butte? AJ. Extracting detailed oncologic history and treatment plan from medical oncology notes with large language models.? arXiv. Preprint posted online August 7, 2023."},{"key":"zoi240332r13","unstructured":"Ouyang? L, Wu? J, Jiang? X, . Training language models to follow instructions with human feedback.? arXiv. Preprint posted online March 4, 2022."},{"key":"zoi240332r14","unstructured":"Miao? BY, Williams? CY, Chinedu-Eneh? E, . Identifying reasons for contraceptive switching from real-world data using large language models.? arXiv. Preprint posted online February 5, 2024."},{"key":"zoi240332r15","unstructured":"Lehman? E, Hernandez? E, Mahajan? D, . Do we still need clinical language models?? arXiv. Preprint posted online February 16, 2023. http:\/\/arxiv.org\/abs\/2302.08091"},{"key":"zoi240332r16","unstructured":"Nori? H, Lee? YT, Zhang? S, . Can generalist foundation models outcompete special-purpose tuning? case study in medicine.? arXiv. Preprint posted online November 27, 2023."},{"key":"zoi240332r18","unstructured":"Liu? P, Yuan? W, Fu? J, Jiang? Z, Hayashi? H, Neubig? G. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing.? arXiv. Preprint posted online July 28, 2021. doi:10.48550\/arXiv.2107.13586"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2818387\/williams_2024_oi_240332_1714165663.37592.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,7]],"date-time":"2024-05-07T11:08:15Z","timestamp":1715080095000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2818387"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,7]]},"references-count":19,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,5,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2024.8895","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,7]]}}}