{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T13:56:54Z","timestamp":1775483814689,"version":"3.50.1"},"reference-count":44,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T00:00:00Z","timestamp":1744416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R00LM014097-02"],"award-info":[{"award-number":["R00LM014097-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM013995-01"],"award-info":[{"award-number":["R01LM013995-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>This study aims to develop and evaluate an approach using large language models (LLMs) and a knowledge graph to triage patient messages that need emergency care. The goal is to notify patients when their messages indicate an emergency, guiding them to seek immediate help rather than using the patient portal, to improve patient safety.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We selected 1020 messages sent to Vanderbilt University Medical Center providers between January 1, 2022 and March 7, 2023. We developed four models to triage these messages for emergencies: (1) Prompt-Only: the patient message was input with a prompt directly into the LLM; (2) Na\u00efve Retrieval Augmented Generation (RAG): provided retrieved information as context to the LLM; (3) RAG from Knowledge Graph with Local Search: a knowledge graph was used to retrieve locally relevant information based on semantic similarities; (4) RAG from Knowledge Graph with Global Search: a knowledge graph was used to retrieve globally relevant information through hierarchical community detection. The knowledge base was a triage book covering 225 protocols.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The RAG from Knowledge Graph model with global search outperformed other models, achieving an accuracy of 0.99, a sensitivity of 0.98, and a specificity of 0.99. It demonstrated significant improvements in triaging emergency messages compared to LLM without RAG and na\u00efve RAG.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>The traditional LLM without any retrieval mechanism underperformed compared to models with RAG, which aligns with the expected benefits of augmenting LLMs with domain-specific knowledge sources. Our results suggest that providing external knowledge, especially in a structured manner and in community summaries, can improve LLM performance in triaging patient portal messages.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>LLMs can effectively assist in triaging emergency patient messages after integrating with a knowledge graph about a nurse triage book. Future research should focus on expanding the knowledge graph and deploying the system to evaluate its impact on patient outcomes.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf059","type":"journal-article","created":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T21:17:43Z","timestamp":1744492663000},"page":"1032-1039","source":"Crossref","is-referenced-by-count":18,"title":["Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation"],"prefix":"10.1093","volume":"32","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212,","place":["United States"]},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37240,","place":["United States"]}]},{"given":"Aileen P","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2292-9147","authenticated-orcid":false,"given":"Allison B","family":"McCoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212,","place":["United States"]}]},{"given":"Sean S","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]}]},{"given":"Bryan","family":"Steitz","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-145X","authenticated-orcid":false,"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2025,4,12]]},"reference":[{"key":"2025052712464019800_ocaf059-B1","doi-asserted-by":"publisher","first-page":"e17273","DOI":"10.2196\/17273","article-title":"Characterizing patient-clinician communication in secure medical messages: retrospective study","volume":"24","author":"Huang","year":"2022","journal-title":"J Med Internet Res"},{"key":"2025052712464019800_ocaf059-B2","doi-asserted-by":"publisher","first-page":"e16521","DOI":"10.2196\/16521","article-title":"A retrospective analysis of provider-to-patient secure messages: how much are they increasing, who is doing the work, and is the work happening after hours?","volume":"8","author":"North","year":"2020","journal-title":"JMIR Med Inform"},{"key":"2025052712464019800_ocaf059-B3","doi-asserted-by":"publisher","first-page":"4002","DOI":"10.1007\/s11606-022-07766-0","article-title":"The electronic health record inbox: recommendations for relief","volume":"37","author":"Sinsky","year":"2022","journal-title":"J Gen Intern Med"},{"key":"2025052712464019800_ocaf059-B4","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1093\/jamia\/ocab268","article-title":"Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use","volume":"29","author":"Holmgren","year":"2022","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B5","doi-asserted-by":"publisher","first-page":"942","DOI":"10.1093\/jamia\/ocx021","article-title":"An analysis of patient-provider secure messaging at two Veterans Health Administration medical centers: message content and resolution through secure messaging","volume":"24","author":"Shimada","year":"2017","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B6","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025052712464019800_ocaf059-B7","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"2025052712464019800_ocaf059-B8","doi-asserted-by":"publisher","first-page":"583","DOI":"10.5664\/jcsm.10948","article-title":"Evaluating insomnia queries from an artificial intelligence chatbot for patient education","volume":"20","author":"Alapati","year":"2024","journal-title":"J Clin Sleep Med"},{"key":"2025052712464019800_ocaf059-B9","doi-asserted-by":"publisher","first-page":"1237","DOI":"10.1093\/jamia\/ocad072","article-title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","volume":"30","author":"Liu","year":"2023","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B10","doi-asserted-by":"publisher","first-page":"1388","DOI":"10.1093\/jamia\/ocae041","article-title":"Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B11","doi-asserted-by":"publisher","first-page":"1665","DOI":"10.1093\/JAMIA\/OCAE142","article-title":"Using large language model to guide patients to create efficient and comprehensive clinical care message","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B12","doi-asserted-by":"publisher","first-page":"1367","DOI":"10.1093\/jamia\/ocae052","article-title":"Leveraging large language models for generating responses to patient messages\u2014a subjective analysis","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B13","doi-asserted-by":"publisher","first-page":"e555-61","DOI":"10.1016\/S2589-7500(24)00097-9","article-title":"The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study","volume":"6","author":"Levine","year":"2024","journal-title":"Lancet Digit Health"},{"key":"2025052712464019800_ocaf059-B14","article-title":"Hallucination is inevitable: an innate limitation of large language models","author":"Xu","year":"2024"},{"key":"2025052712464019800_ocaf059-B15","author":"Zhang","year":"2023"},{"key":"2025052712464019800_ocaf059-B16","author":"Huang","year":"2023"},{"key":"2025052712464019800_ocaf059-B17","doi-asserted-by":"publisher","first-page":"5994","DOI":"10.1038\/s41598-017-05778-z","article-title":"Learning a health knowledge graph from electronic medical records","volume":"7","author":"Rotmensch","year":"2017","journal-title":"Sci Rep"},{"key":"2025052712464019800_ocaf059-B18","author":"Lo","year":"2023"},{"key":"2025052712464019800_ocaf059-B19","first-page":"131","author":"Tang","year":"2024"},{"key":"2025052712464019800_ocaf059-B20","author":"Li","year":"2024"},{"key":"2025052712464019800_ocaf059-B21","author":"Peng","year":"2023"},{"key":"2025052712464019800_ocaf059-B22","doi-asserted-by":"crossref","article-title":"Improving retrieval-augmented generation in medicine with iterative follow-up questions","author":"Xiong","DOI":"10.1142\/9789819807024_0015"},{"key":"2025052712464019800_ocaf059-B23","doi-asserted-by":"publisher","author":"Huang","DOI":"10.18653\/V1\/2024.FINDINGS-ACL.94"},{"key":"2025052712464019800_ocaf059-B24","doi-asserted-by":"publisher","DOI":"10.1093\/JAMIA\/OCAF008","article-title":"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines","author":"Liu","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"2025052712464019800_ocaf059-B25","author":"Briggs","year":"; 2021."},{"key":"2025052712464019800_ocaf059-B26","author":"Gao","year":"2023"},{"key":"2025052712464019800_ocaf059-B27","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: how language models use long contexts","volume":"12","author":"Liu","year":"2024","journal-title":"Trans Assoc Comput Linguist"},{"key":"2025052712464019800_ocaf059-B28","author":"Xu","year":"2024"},{"key":"2025052712464019800_ocaf059-B29","doi-asserted-by":"publisher","first-page":"3580","DOI":"10.1109\/TKDE.2024.3352100","article-title":"Unifying large language models and knowledge graphs: a roadmap","volume":"36","author":"Pan","year":"2024","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2025052712464019800_ocaf059-B30","article-title":"GraphEval: a knowledge-graph based LLM hallucination evaluation framework","author":"Sansford","year":"2024"},{"key":"2025052712464019800_ocaf059-B31","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1093\/jamia\/ocac219","article-title":"Automated deidentification of radiology reports combining transformer and \u201chide in plain sight\u201d rule-based methods","volume":"30","author":"Chambon","year":"2023","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B32","doi-asserted-by":"publisher","first-page":"5233","DOI":"10.1038\/s41598-019-41695-z","article-title":"From Louvain to Leiden: guaranteeing well-connected communities","volume":"9","author":"Traag","year":"2019","journal-title":"Sci Rep"},{"key":"2025052712464019800_ocaf059-B33","author":"Edge","year":"2024"},{"key":"2025052712464019800_ocaf059-B34","author":"Chroma"},{"key":"2025052712464019800_ocaf059-B35","doi-asserted-by":"publisher","first-page":"6086","DOI":"10.1038\/s41598-024-56706-x","article-title":"Evaluation metrics and statistical tests for machine learning","volume":"14","author":"Rainio","year":"2024","journal-title":"Sci Rep"},{"key":"2025052712464019800_ocaf059-B36","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","author":"Demsar","year":"2006","journal-title":"J Mach Learn Res"},{"key":"2025052712464019800_ocaf059-B37","author":"Nemenyi","year":"1963"},{"key":"2025052712464019800_ocaf059-B38","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1007\/s43678-023-00616-w","article-title":"Repeatability, reproducibility, and diagnostic accuracy of a commercial large language model (ChatGPT) to perform emergency department triage using the Canadian triage and acuity scale","volume":"26","author":"Franc","year":"2024","journal-title":"Can J Emerg Med"},{"key":"2025052712464019800_ocaf059-B39","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1080\/10903127.2024.2374400","article-title":"Emergency patient triage improvement through a retrieval-augmented generation enhanced large-scale language model","volume":"29","author":"Yazaki","year":"2025","journal-title":"Prehospital Emerg Care"},{"key":"2025052712464019800_ocaf059-B40","doi-asserted-by":"publisher","first-page":"e53297","DOI":"10.2196\/53297","article-title":"Triage performance across large language models, ChatGPT, and untrained doctors in emergency medicine: comparative study","volume":"26","author":"Masanneck","year":"2024","journal-title":"J Med Internet Res."},{"key":"2025052712464019800_ocaf059-B41","doi-asserted-by":"publisher","first-page":"e48568","DOI":"10.2196\/48568","article-title":"Utility of ChatGPT in clinical practice","volume":"25","author":"Liu","year":"2023","journal-title":"J Med Internet Res."},{"key":"2025052712464019800_ocaf059-B42","doi-asserted-by":"publisher","first-page":"923","DOI":"10.1093\/jamia\/ocaa229","article-title":"Physicians\u2019 electronic inbox work patterns and factors associated with high inbox work duration","volume":"28","author":"Akbar","year":"2021","journal-title":"J Am Med Inf Assoc"},{"key":"2025052712464019800_ocaf059-B43","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1370\/afm.2121","article-title":"Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations","volume":"15","author":"Arndt","year":"2017","journal-title":"Ann Fam Med"},{"key":"2025052712464019800_ocaf059-B44","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1038\/s41746-024-01185-7","article-title":"Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine","volume":"7","author":"Jin","year":"2024","journal-title":"NPJ Digit Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/6\/1032\/62922452\/ocaf059.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/6\/1032\/62922452\/ocaf059.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,27]],"date-time":"2025-05-27T16:46:52Z","timestamp":1748364412000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/6\/1032\/8112816"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,12]]},"references-count":44,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,4,12]]},"published-print":{"date-parts":[[2025,6,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf059","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,6]]},"published":{"date-parts":[[2025,4,12]]}}}