{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T22:46:19Z","timestamp":1776379579727,"version":"3.51.2"},"reference-count":26,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T00:00:00Z","timestamp":1770336000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Korea University Ansan Hospital","award":["O2411941"],"award-info":[{"award-number":["O2411941"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Accurate triage in emergency departments (ED) is critical for appropriate resource allocation. While artificial intelligence (AI) has been explored for triage, prior models relied on summarized clinical scenarios. We aimed to develop and evaluate large language models (LLMs) trained on real-world clinical conversations to classify patient urgency.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We used a nationally curated dataset of anonymized triage-level conversations from 3 tertiary Korean hospitals. Two BERT-based models were developed to classify urgency per the Korean Triage and Acuity Scale (KTAS) into urgent (KTAS 3) or non-urgent (KTAS 4-5). One model tokenized the entire conversation, while the other applied a hierarchical structure with sentence-level tokenization and speaker-role embeddings. Performance metrics included accuracy, precision, recall, and F1-score. We compared our models against ChatGPT GPT-4o and ClinicalBERT, and assessed explainability using SHapley Additive exPlanations (SHAP).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>A total of 5244 clinical conversations, 1057 triage-level dialogues were used, with 950 for training and 107 for testing. Our model with hierarchical structure achieved accuracies of 75.94%, significantly outperforming ChatGPT (56.68%) or fine-tuned ClinicalBERT (69.42%). For urgent cases, the best model achieved a recall of 0.9610, outperforming ChatGPT (0.5352). SHapley Additive exPlanations analysis confirmed that our model focused on clinically relevant cues aligned with KTAS criteria.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>BERT-based LLMs trained on real-world ED conversations significantly outperform general-purpose models like ChatGPT in triage accuracy. This approach demonstrates the potential for enhancing clinical decision support with interpretable and efficient AI.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag007","type":"journal-article","created":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T12:43:07Z","timestamp":1768394587000},"page":"847-854","source":"Crossref","is-referenced-by-count":0,"title":["Development of BERT-based large language models for emergency department triage using real-world conversations"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7105-1951","authenticated-orcid":false,"given":"Sukyo","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Emergency Medicine, Korea University Ansan Hospital , Ansan-si, Gyeonggi-do 15355,","place":["Republic of Korea"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sumin","family":"Jung","sequence":"additional","affiliation":[{"name":"Core Research & Development Center, Korea University Ansan Hospital , Ansan-si, Gyeonggi-do 15355,","place":["Republic of Korea"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jong-Hak","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Korea University Ansan Hospital , Ansan-si, Gyeonggi-do 15355,","place":["Republic of Korea"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hanjin","family":"Cho","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Korea University Ansan Hospital , Ansan-si, Gyeonggi-do 15355,","place":["Republic of Korea"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sungwoo","family":"Moon","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Korea University Ansan Hospital , Ansan-si, Gyeonggi-do 15355,","place":["Republic of Korea"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6003-2552","authenticated-orcid":false,"given":"Sejoong","family":"Ahn","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Korea University Ansan Hospital , Ansan-si, Gyeonggi-do 15355,","place":["Republic of Korea"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2026,2,6]]},"reference":[{"key":"2026041617412525200_ocag007-B1","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.annemergmed.2018.09.022","article-title":"Triage performance in emergency medicine: a systematic review","volume":"74","author":"Hinson","year":"2019","journal-title":"Ann Emerg Med"},{"key":"2026041617412525200_ocag007-B2","doi-asserted-by":"crossref","first-page":"e026471","DOI":"10.1136\/bmjopen-2018-026471","article-title":"Performance of triage systems in emergency care: a systematic review and meta-analysis","volume":"9","author":"Zachariasse","year":"2019","journal-title":"BMJ Open"},{"key":"2026041617412525200_ocag007-B3","doi-asserted-by":"crossref","first-page":"S18","DOI":"10.1017\/cem.2017.365","article-title":"Revisions to the Canadian emergency department triage and acuity scale (CTAS) guidelines 2016","volume":"19","author":"Bullard","year":"2017","journal-title":"Can J Emerg Med"},{"key":"2026041617412525200_ocag007-B4","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1197\/j.aem.2003.06.013","article-title":"Reliability and validity of scores on the Emergency Severity Index version 3","volume":"11","author":"Tanabe","year":"2004","journal-title":"Acad Emerg Med"},{"key":"2026041617412525200_ocag007-B5","first-page":"547","article-title":"Korean triage and acuity scale (KTAS)","volume":"28","author":"Park","year":"2017","journal-title":"J Korean Soc Emerg Med"},{"key":"2026041617412525200_ocag007-B6","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/j.ajem.2024.12.024","article-title":"Evaluating LLM-based generative AI tools in emergency triage: a comparative study of ChatGPT Plus, Copilot Pro, and triage nurses","volume":"89","author":"Arslan","year":"2025","journal-title":"Am J Emerg Med"},{"key":"2026041617412525200_ocag007-B7","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1007\/s43678-023-00616-w","article-title":"Repeatability, reproducibility, and diagnostic accuracy of a commercial large language model (ChatGPT) to perform emergency department triage using the Canadian triage and acuity scale","volume":"26","author":"Franc","year":"2024","journal-title":"Can J Emerg Med"},{"key":"2026041617412525200_ocag007-B8","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1016\/j.ajem.2023.10.034","article-title":"Performance of Google bard and ChatGPT in mass casualty incidents triage","volume":"75","author":"Gan","year":"2024","journal-title":"Am J Emerg Med"},{"key":"2026041617412525200_ocag007-B9","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1016\/j.ajem.2024.05.001","article-title":"Comparative analysis of ChatGPT, Gemini and emergency medicine specialist in ESI triage assessment","volume":"81","author":"Meral","year":"2024","journal-title":"Am J Emerg Med"},{"key":"2026041617412525200_ocag007-B10","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1016\/j.ajem.2024.02.008","article-title":"Human intelligence versus Chat-GPT: who performs better in correctly classifying patients in triage?","volume":"79","author":"Zaboli","year":"2024","journal-title":"Am J Emerg Med"},{"key":"2026041617412525200_ocag007-B11","author":"National Information Society Agency","year":"2023"},{"key":"2026041617412525200_ocag007-B12","first-page":"36963","article-title":"A. Language model tokenizers introduce unfairness between languages","volume":"36","author":"Petrov","year":"2023","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026041617412525200_ocag007-B13","author":"Devlin","year":"2019"},{"key":"2026041617412525200_ocag007-B14","author":"Clark","year":"2020"},{"key":"2026041617412525200_ocag007-B15","doi-asserted-by":"publisher","first-page":"233","author":"Lu","DOI":"10.1007\/978-3-030-88942-5_18"},{"key":"2026041617412525200_ocag007-B16","author":"Alsentzer","year":"2019"},{"key":"2026041617412525200_ocag007-B17","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat Mach Intell"},{"key":"2026041617412525200_ocag007-B18","first-page":"4768","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Lundberg","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026041617412525200_ocag007-B19","author":"Kokhlikyan","year":"2020"},{"key":"2026041617412525200_ocag007-B20","doi-asserted-by":"publisher","first-page":"e175","DOI":"10.3346\/jkms.2021.36.e175","article-title":"Automatic classification of the Korean Triage Acuity Scale in simulated emergency rooms using speech recognition and natural language processing: a proof of concept study","volume":"36","author":"Kim","year":"2021","journal-title":"J Korean Med Sci"},{"key":"2026041617412525200_ocag007-B21","doi-asserted-by":"publisher","author":"Abacha","DOI":"10.18653\/v1\/2025.findings-acl.1159"},{"key":"2026041617412525200_ocag007-B22","doi-asserted-by":"publisher","first-page":"1921","DOI":"10.1093\/jamia\/ocae103","article-title":"Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room","volume":"31","author":"Glicksberg","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026041617412525200_ocag007-B23","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1186\/s12873-025-01337-2","article-title":"Performance of ChatGPT, Gemini and DeepSeek for non-critical triage support using real-world conversations in emergency department","volume":"25","author":"Lee","year":"2025","journal-title":"BMC Emerg Med"},{"key":"2026041617412525200_ocag007-B24","first-page":"28492","author":"Radford","year":"2023"},{"key":"2026041617412525200_ocag007-B25","doi-asserted-by":"publisher","author":"Baek","year":"2024","DOI":"10.1109\/ICUFN61752.2024.10625181"},{"key":"2026041617412525200_ocag007-B26","doi-asserted-by":"crossref","first-page":"e0216972","DOI":"10.1371\/journal.pone.0216972","article-title":"Triage accuracy and causes of mistriage using the Korean Triage and Acuity Scale","volume":"14","author":"Moon","year":"2019","journal-title":"PLoS One"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag007\/66799015\/ocag007.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/4\/847\/66799015\/ocag007.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/4\/847\/66799015\/ocag007.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T21:43:05Z","timestamp":1776375785000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/4\/847\/8466363"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,6]]},"references-count":26,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,2,6]]},"published-print":{"date-parts":[[2026,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag007","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,4]]},"published":{"date-parts":[[2026,2,6]]}}}