{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,28]],"date-time":"2026-06-28T07:10:19Z","timestamp":1782630619529,"version":"3.54.5"},"reference-count":40,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T00:00:00Z","timestamp":1776211200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000142","name":"Arthritis Society","doi-asserted-by":"publisher","award":["SOG-24-0377)"],"award-info":[{"award-number":["SOG-24-0377)"]}],"id":[{"id":"10.13039\/501100000142","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Health Serv."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Natural Language Processing (NLP) models show promise in enhancing interpretation and triage of outpatient referrals across diverse specialties.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To conduct a systematic literature review and narrative synthesis of recent studies that utilized NLP-based models for triage-related tasks such as urgency prioritization, referral classification, and justification review.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Medline, Embase, Web of Science, and CINAHL databases were searched for articles published up to February 17 2024, limiting searches to the last 5 years prior to the search. All citations were imported into Covidence for duplicate removal and screening. We included studies that utilized NLP techniques to triage outpatient referrals to a specialist (medical or surgical), and included comparison to human triage. Abstracts and full texts were each screened independently by two reviewers. Data from each study were extracted independently by two reviewers using a standardized extraction form, including fields such as study design, dataset size, specialty, models tested, and outcomes reported. Results were synthesized narratively, organized by key themes focused on data, model and clinical applicability. Quality and risk of bias assessment was performed using the PROBAST-AI and Technology Readiness scales.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>A total of 4,225 titles and abstracts were reviewed resulting in 26 full-text reviews. A total of 10 studies were used for data extraction and synthesis. These studies spanned a wide range of medical specialties including surgery, medical specialties, and radiology. Tasks included predicting condition and priority level. Most domains were assessed as low or uncertain risk of bias. Outcome measures varied across studies, but overall, 7 studies reported high levels of accuracy compared to manual workflows. We summarized key differences in dataset preprocessing and augmentation, triage model, and feasibility and clinical applicability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>NLP shows promise in augmenting human triage of outpatient referrals to specialty care. To realize the full potential of NLP for triage, future work should prioritize standardized reporting and prospective validation to support safe and effective integration into healthcare systems.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frhs.2026.1797583","type":"journal-article","created":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T05:41:01Z","timestamp":1776231661000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Systematic literature review and narrative synthesis of the use of natural language processing to triage outpatient referrals"],"prefix":"10.3389","volume":"6","author":[{"given":"Angela Lim","family":"Fung","sequence":"first","affiliation":[{"name":"Institute of Medical Science, University of Toronto","place":["Toronto, ON, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamed","family":"Abdalla","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine, Department of Medicine, University of Alberta","place":["Edmonton, AB, Canada"]},{"name":"The Alberta Machine Intelligence Institute","place":["Edmonton, AB, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu Blanche","family":"Cheng","sequence":"additional","affiliation":[{"name":"Department of Laboratory Medicine and Pathobiology, Temerty Faculty of Medicine, University of Toronto","place":["Toronto, ON, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manar","family":"Elsayed","sequence":"additional","affiliation":[{"name":"Division of Rheumatology, Department of Medicine, University of Alberta","place":["Edmonton, AB, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dagmara","family":"Chojecki","sequence":"additional","affiliation":[{"name":"Institute of Health Economics","place":["Edmonton, AB, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph Ross","family":"Mitchell","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine, Department of Medicine, University of Alberta","place":["Edmonton, AB, Canada"]},{"name":"The Alberta Machine Intelligence Institute","place":["Edmonton, AB, Canada"]},{"name":"Department of Computing Sciences, University of Alberta","place":["Edmonton, AB, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carrie","family":"Ye","sequence":"additional","affiliation":[{"name":"Division of Rheumatology, Department of Medicine, University of Alberta","place":["Edmonton, AB, Canada"]},{"name":"Arthritis Research Canada","place":["Vancouver, BC, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,4,15]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1186\/s13643-025-02841-z","article-title":"Interventions to improve referrals from primary care to outpatient specialist services for chronic conditions: a systematic review and framework synthesis update","volume":"14","author":"Bolton Saghdaoui","year":"2025","journal-title":"Syst Rev"},{"key":"B2","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1080\/03009742.2022.2112833","article-title":"Impact of triage by a rheumatologist on appropriateness of referrals from primary to secondary care: a cluster randomized trial","volume":"52","author":"van Delft","year":"2023","journal-title":"Scand J Rheumatol"},{"key":"B3","doi-asserted-by":"publisher","first-page":"2006","DOI":"10.1016\/S0140-6736(20)32340-0","article-title":"Global estimates of the need for rehabilitation based on the global burden of disease study 2019: a systematic analysis for the global burden of disease study 2019","volume":"396","author":"Cieza","year":"2021","journal-title":"Lancet"},{"key":"B4","volume-title":"Waiting Times Health Services","year":"2020"},{"key":"B5","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1016\/j.physio.2015.10.011","article-title":"Specific timely appointments for triage reduced waiting lists in an outpatient physiotherapy service","volume":"102","author":"Harding","year":"2016","journal-title":"Physiotherapy"},{"key":"B6","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1093\/pch\/pxab086","article-title":"Patient triage to specialist outpatient clinics-time to standardize terminology","volume":"27","author":"Belostotsky","year":"2022","journal-title":"Paediatr Child Health"},{"key":"B7","first-page":"1378","article-title":"Triage of referrals to an outpatient rheumatology clinic: analysis of referral information and triage","volume":"35","author":"Graydon","year":"2008","journal-title":"J Rheumatol"},{"key":"B8","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1016\/j.tacc.2021.02.007","article-title":"Natural language processing in medicine: a review","volume":"38","author":"Locke","year":"2021","journal-title":"Tren Anaesth Crit Care"},{"key":"B9","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1038\/s41746-022-00730-6","article-title":"A survey on clinical natural language processing in the United Kingdom from 2007 to 2022","volume":"5","author":"Wu","year":"2022","journal-title":"NPJ Digit Med"},{"key":"B10","doi-asserted-by":"publisher","first-page":"106649","DOI":"10.1016\/j.compbiomed.2023.106649","article-title":"Natural language processing in electronic health records in relation to healthcare decision-making: a systematic review","volume":"155","author":"Hossain","year":"2023","journal-title":"Comput Biol Med"},{"key":"B11","doi-asserted-by":"crossref","first-page":"5357","DOI":"10.18653\/v1\/2021.emnlp-main.435","article-title":"Incorporating medical knowledge in BERT for clinical relation extraction","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Roy","year":"2021"},{"key":"B12","doi-asserted-by":"publisher","first-page":"101139","DOI":"10.1016\/j.imu.2022.101139","article-title":"Comparison of BERT implementations for natural language processing of narrative medical documents","volume":"36","author":"Turchin","year":"2023","journal-title":"Inform Med Unlocked"},{"key":"B13","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1186\/s13643-021-01626-4","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"10","author":"Page","year":"2021","journal-title":"Syst Rev"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e082505","DOI":"10.1136\/bmj-2024-082505","article-title":"PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods","volume":"388","author":"Moons","year":"2025","journal-title":"Br Med J"},{"key":"B15","doi-asserted-by":"publisher","first-page":"1486","DOI":"10.1007\/s00134-020-06045-y","article-title":"Machine learning in intensive care medicine: ready for take-off?","volume":"46","author":"Fleuren","year":"2020","journal-title":"Intensive Care Med"},{"key":"B16","doi-asserted-by":"publisher","first-page":"674","DOI":"10.1016\/j.radi.2022.05.005","article-title":"Development of lumbar spine MRI referrals vetting models using machine learning and deep learning algorithms: comparison models vs healthcare professionals","volume":"28","author":"Alanazi","year":"2022","journal-title":"Radiography (Lond)"},{"key":"B17","doi-asserted-by":"publisher","first-page":"100961","DOI":"10.1016\/j.imu.2022.100961","article-title":"Machine learning and deep learning-based natural language processing for auto-vetting the appropriateness of lumbar spine magnetic resonance imaging referrals","volume":"30","author":"Alanazi","year":"2022","journal-title":"Inform Med Unlocked"},{"key":"B18","doi-asserted-by":"publisher","first-page":"113899","DOI":"10.1016\/j.dss.2022.113899","article-title":"Improving triaging from primary care into secondary care using heterogeneous data-driven hybrid machine learning","volume":"166","author":"Wang","year":"2023","journal-title":"Decis Support Syst"},{"key":"B19","doi-asserted-by":"publisher","first-page":"e21252","DOI":"10.2196\/21252","article-title":"Patient triage by topic modeling of referral letters: feasibility study","volume":"8","author":"Spasic","year":"2020","journal-title":"JMIR Med Inform"},{"key":"B20","doi-asserted-by":"publisher","first-page":"e46857","DOI":"10.2196\/46857","article-title":"Natural language processing of referral letters for machine learning-based triaging of patients with low back pain to the most appropriate intervention: retrospective study","volume":"26","author":"Fudickar","year":"2024","journal-title":"J Med Internet Res"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1192975","DOI":"10.3389\/fdgth.2023.1192975","article-title":"Artificial intelligence in medical referrals triage based on clinical prioritization criteria","volume":"5","author":"Abdel-Hafez","year":"2023","journal-title":"Front Digit Health"},{"key":"B22","first-page":"85","article-title":"Applying natural language processing neural network architectures to augment appointment request review of self-referred patients to an academic medical center","volume":"2022","author":"Aakre","year":"2022","journal-title":"AMIA Summits Transl Sci Proc"},{"key":"B23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/EHB52898.2021.9657630","article-title":"Application of latent semantic analysis and supervised learning methods to automate triage of referral letters for spinal surgery","volume-title":"2021 International Conference on e-Health and Bioengineering (EHB)","author":"Guzman","year":"2021"},{"key":"B24","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1186\/s13244-022-01267-8","article-title":"Automated vetting of radiology referrals: exploring natural language processing and traditional machine learning approaches","volume":"13","author":"Poto\u010dnik","year":"2022","journal-title":"Insights Imaging"},{"key":"B25","first-page":"227","article-title":"Artificial intelligence in ophthalmology triaging","volume-title":"Artificial Intelligence in Ophthalmology","author":"Tan","year":"2021"},{"key":"B26","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1186\/s12911-021-01565-z","article-title":"Supporting the classification of patients in public hospitals in Chile by designing, deploying and validating a system based on natural language processing","volume":"21","author":"Villena","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"B27","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"B28","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat Med"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1884","DOI":"10.1093\/jamia\/ocae184","article-title":"The potential and pitfalls of using a large language model such as ChatGPT, GPT-4, or LLaMA as a clinical assistant","volume":"31","author":"Zhang","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"B30","doi-asserted-by":"publisher","first-page":"8236","DOI":"10.1038\/s41467-024-52415-1","article-title":"Evaluating the use of large language models to provide clinical recommendations in the emergency department","volume":"15","author":"Williams","year":"2024","journal-title":"Nat Commun"},{"key":"B31","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1038\/s41591-024-03416-6","article-title":"A generalist medical language model for disease diagnosis assistance","volume":"31","author":"Liu","year":"2025","journal-title":"Nat Med"},{"key":"B32","doi-asserted-by":"publisher","first-page":"e72062","DOI":"10.2196\/72062","article-title":"Large language models in medical diagnostics: scoping review with bibliometric analysis","volume":"27","author":"Su","year":"2025","journal-title":"J Med Internet Res"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1101\/2024.04.09.24305594","article-title":"Inherent bias in electronic health records: a scoping review of sources of bias","author":"Perets","year":"2024","journal-title":"medRxiv"},{"key":"B34","doi-asserted-by":"publisher","first-page":"e078378","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","year":"2024","journal-title":"Br Med J"},{"key":"B35","doi-asserted-by":"publisher","first-page":"e2335377","DOI":"10.1001\/jamanetworkopen.2023.35377","article-title":"APPRAISE-AI tool for quantitative evaluation of AI studies for clinical decision support","volume":"6","author":"Kwong","year":"2023","journal-title":"JAMA Netw Open"},{"key":"B36","doi-asserted-by":"crossref","first-page":"e001632","DOI":"10.1136\/bmjmed-2025-001632","article-title":"Reporting guideline for chatbot health advice studies: the chatbot assessment reporting tool (CHART) statement","volume":"4","year":"2025","journal-title":"BMJ Med"},{"key":"B37","doi-asserted-by":"publisher","first-page":"e52508","DOI":"10.2196\/52508","article-title":"Consolidated reporting guidelines for prognostic and diagnostic machine learning models (CREMLS)","volume":"26","author":"El Emam","year":"2024","journal-title":"J Med Internet Res"},{"key":"B38","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","year":"2025","journal-title":"Nat Med"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.2196\/87057","article-title":"The Alberta risk of bias assessment tool (AQAT:roB) for the evaluation of medical large language model question-answer studies: development and pilot validation","author":"Ye","year":"2026","journal-title":"J Med Internet Res"},{"key":"B40","doi-asserted-by":"publisher","first-page":"e048008","DOI":"10.1136\/bmjopen-2020-048008","article-title":"Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence","volume":"11","author":"Collins","year":"2021","journal-title":"BMJ Open"}],"container-title":["Frontiers in Health Services"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frhs.2026.1797583\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T05:41:03Z","timestamp":1776231663000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frhs.2026.1797583\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,15]]},"references-count":40,"alternative-id":["10.3389\/frhs.2026.1797583"],"URL":"https:\/\/doi.org\/10.3389\/frhs.2026.1797583","relation":{},"ISSN":["2813-0146"],"issn-type":[{"value":"2813-0146","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,15]]},"article-number":"1797583"}}