{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T03:34:08Z","timestamp":1792985648813,"version":"4.5.0"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T00:00:00Z","timestamp":1791504000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T00:00:00Z","timestamp":1791504000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Rheumatol Int"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Triage of rheumatology outpatient referrals is a high-volume administrative task that consumes senior specialist time without advancing patient care. The human triage system is only moderately accurate and reproducible. We assessed whether contemporary large language models (LLMs) are able to perform well enough to support automating this task in practice. In addition, the effects of different prompting techniques on triage accuracy and cost was assessed to help identify the optimal approach. Twenty referral scenarios spanning the urgency spectrum were created by a certified Australian rheumatologist. Four rheumatologists triaged all cases independently and blinded to produce a consensus reference standard; inter-rater reliability was quantified. Twenty-three LLMs each triaged every referral into one of five urgency categories, three times (1380 outputs per referral). The experiment was run with a simple prompt and repeated with an advanced prompt supplying explicit triage expectations and worked examples. In line with published literature, agreement among the four rheumatologists was moderate (Fleiss kappa 0.60; mean pairwise linear-weighted kappa 0.79), with all four assigning categories within one level of each other in 90% of cases. All 2760 model calls returned an interpretable result. Under the simple prompt, performance separated into distinct tiers, larger models were more accurate (Spearman rho\u2009=\u20090.42;\n                    <jats:italic>P<\/jats:italic>\n                    =.047) and accuracy tracked cost. Advanced prompting reduced between-model variance in accuracy 5.3-fold (0.014 to 0.003; Levene\n                    <jats:italic>P<\/jats:italic>\n                    =.01), abolished the size-accuracy association (rho=-0.05;\n                    <jats:italic>P<\/jats:italic>\n                    =.83) and removed the accuracy-cost relationship. The best models matched expert consensus on 18 of 20 cases, comparable to the reproducibility observed among the rheumatologists themselves. Under-triage errors persisted with some LLMs. Contemporary LLMs categorised rheumatology referral urgency with an accuracy and reproducibility comparable to that reported for human triage systems, although human triage was not tested in this study. Advanced prompting appeared to substitute for the reasoning capability of larger models, suggesting that adequate performance on this task may not require the most expensive models. These findings indicate that automation of this administrative task is technically feasible. Candidate models that warrant prospective evaluation are identified.\n                  <\/jats:p>","DOI":"10.1007\/s00296-026-06317-8","type":"journal-article","created":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T08:36:12Z","timestamp":1791534972000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Automating the triage of rheumatology outpatient referrals: a comparative evaluation of 23 large language models under simple and advanced prompting"],"prefix":"10.1007","volume":"46","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2960-1075","authenticated-orcid":false,"given":"Lynden","family":"Roberts","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,10,9]]},"reference":[{"key":"6317_CR1","doi-asserted-by":"publisher","first-page":"843","DOI":"10.31128\/ajgp-08-23-6921","volume":"52","author":"M Wright","year":"2023","unstructured":"Wright M, Brell R (2023) Balancing care and responsibility: the role of the general practitioner in specialist referrals. Aust J Gen Pract 52:843\u2013847. https:\/\/doi.org\/10.31128\/ajgp-08-23-6921","journal-title":"Aust J Gen Pract"},{"key":"6317_CR2","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1016\/j.healthpol.2022.07.003","volume":"126","author":"G Mariotti","year":"2022","unstructured":"Mariotti G, Siciliani L, Rebba V, Coretti S, Gentilini M (2022) Consensus among clinicians on referrals\u2019 priority and use of digital decision-making support systems. Health Policy 126:906\u2013914. https:\/\/doi.org\/10.1016\/j.healthpol.2022.07.003","journal-title":"Health Policy"},{"key":"6317_CR3","doi-asserted-by":"publisher","first-page":"555","DOI":"10.3899\/jrheum.220300","volume":"49","author":"EM Miloslavsky","year":"2022","unstructured":"Miloslavsky EM, Marston B (2022) The challenge of addressing the rheumatology workforce shortage. J Rheumatol 49:555\u2013557. https:\/\/doi.org\/10.3899\/jrheum.220300","journal-title":"J Rheumatol"},{"key":"6317_CR4","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1038\/s41746-025-01495-4","volume":"8","author":"TD Maarseveen","year":"2025","unstructured":"Maarseveen TD, Glas HK, van den Veris-van Dieren J, Knevel R (2025) Improving musculoskeletal care with AI enhanced triage through data driven screening of referral letters. npj Digit Med 8:98. https:\/\/doi.org\/10.1038\/s41746-025-01495-4","journal-title":"npj Digit Med"},{"key":"6317_CR5","doi-asserted-by":"publisher","first-page":"3537","DOI":"10.1002\/art.27692","volume":"62","author":"MP n der Linden","year":"2010","unstructured":"n der Linden MP, le Cessie S, van der Raza K, Knevel R, Huizinga TW et al (2010) Long-term impact of delay in assessment of patients with early arthritis. Arthritis Rheum 62:3537\u20133546. https:\/\/doi.org\/10.1002\/art.27692","journal-title":"Arthritis Rheum"},{"key":"6317_CR6","first-page":"1378","volume":"35","author":"SL Graydon","year":"2008","unstructured":"Graydon SL, Thompson AE (2008) Triage of referrals to an outpatient rheumatology clinic: analysis of referral information and triage. J Rheumatol 35:1378\u20131383","journal-title":"J Rheumatol"},{"key":"6317_CR7","doi-asserted-by":"publisher","first-page":"1980","DOI":"10.3899\/jrheum.140167","volume":"41","author":"AE Thompson","year":"2014","unstructured":"Thompson AE, Haig SL, LeRiche NG, Rohekar G, Rohekar S, Pope JE (2014) Comprehensive arthritis referral study - phase 2: analysis of the comprehensive arthritis referral tool. J Rheumatol 41:1980\u20131989. https:\/\/doi.org\/10.3899\/jrheum.140167","journal-title":"J Rheumatol"},{"key":"6317_CR8","doi-asserted-by":"publisher","first-page":"929","DOI":"10.1177\/0269215510370527","volume":"24","author":"KE Harding","year":"2010","unstructured":"Harding KE, Taylor NF, Leggat SG, Wise VL (2010) Prioritizing patients for community rehabilitation services: do clinicians agree on triage decisions? Clin Rehabil 24:929\u2013937. https:\/\/doi.org\/10.1177\/0269215510370527","journal-title":"Clin Rehabil"},{"key":"6317_CR9","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1177\/0269215510389344","volume":"25","author":"KE Harding","year":"2011","unstructured":"Harding KE, Taylor NF, Leggat SG, Wise VL (2011) A training programme did not increase agreement between allied health clinicians prioritizing patients for community rehabilitation. Clin Rehabil 25:599\u2013606. https:\/\/doi.org\/10.1177\/0269215510389344","journal-title":"Clin Rehabil"},{"key":"6317_CR10","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1071\/AH11109","volume":"36","author":"L Horsfall","year":"2012","unstructured":"Horsfall L, Skoien R, Moss C, Scott IA (2012) Triage of referrals to outpatient hepatology services: an ineffective tool to prioritise patients? Aust Health Rev 36:443\u2013448. https:\/\/doi.org\/10.1071\/AH11109","journal-title":"Aust Health Rev"},{"key":"6317_CR11","doi-asserted-by":"publisher","first-page":"e46857","DOI":"10.2196\/46857","volume":"26","author":"S Fudickar","year":"2024","unstructured":"Fudickar S, Kalke A, Schemke S, Zderic C, Boll S, Hein A et al (2024) Natural language processing of referral letters for machine learning-based triaging of patients with low back pain to the most appropriate intervention: retrospective study. J Med Internet Res 26:e46857. https:\/\/doi.org\/10.2196\/46857","journal-title":"J Med Internet Res"},{"key":"6317_CR12","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW et al (2023) Large language models encode clinical knowledge. Nature 620:172\u2013180. https:\/\/doi.org\/10.1038\/s41586-023-06291-2","journal-title":"Nature"},{"key":"6317_CR13","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","volume":"2","author":"TH Kung","year":"2023","unstructured":"Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C et al (2023) Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health 2:e0000198. https:\/\/doi.org\/10.1371\/journal.pdig.0000198","journal-title":"PLoS Digit Health"},{"key":"6317_CR14","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1038\/s41746-025-01684-1","volume":"8","author":"F Gaber","year":"2025","unstructured":"Gaber F, Shaik M, Allega F, Bilecz AJ, Busch F, Goon K et al (2025) Evaluating large language model workflows in clinical decision support for triage, referral, and diagnosis. npj Digit Med 8:263. https:\/\/doi.org\/10.1038\/s41746-025-01684-1","journal-title":"npj Digit Med"},{"key":"6317_CR15","doi-asserted-by":"publisher","DOI":"10.1093\/fampra\/cmaf087","author":"S Hack","year":"2025","unstructured":"Hack S, Attal R, Yogev D, Alsleibi S, Gvili B, Shahal D et al (2025) Evaluating large language models for specialist referral triage in primary care: a quantitative study using otolaryngology scenarios. Fam Pract. https:\/\/doi.org\/10.1093\/fampra\/cmaf087","journal-title":"Fam Pract"},{"key":"6317_CR16","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1007\/s00296-023-05464-6","volume":"44","author":"M Krusche","year":"2024","unstructured":"Krusche M, Callhoff J, Knitza J, Ruffer N (2024) Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatol Int 44:303\u2013306. https:\/\/doi.org\/10.1007\/s00296-023-05464-6","journal-title":"Rheumatol Int"},{"key":"6317_CR17","doi-asserted-by":"publisher","first-page":"24824","DOI":"10.52202\/068431-1800","volume":"35","author":"J Wei","year":"2022","unstructured":"Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F et al (2022) Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst 35:24824\u201324837","journal-title":"Adv Neural Inf Process Syst"},{"key":"6317_CR18","doi-asserted-by":"publisher","DOI":"10.1093\/ehjdh\/ztaf068","author":"D Garin","year":"2025","unstructured":"Garin D, Cook S, Ferry C, Bennar W, Togni M, Meier P et al (2025) Improving large language models accuracy for aortic stenosis treatment via Heart Team simulation: a prompt design analysis. Eur Heart J Digit Health. https:\/\/doi.org\/10.1093\/ehjdh\/ztaf068","journal-title":"Eur Heart J Digit Health"},{"key":"6317_CR19","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/s41746-024-01010-1","volume":"7","author":"T Savage","year":"2024","unstructured":"Savage T, Nayak A, Gallo R, Rangan E, Chen JH (2024) Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. npj Digit Med 7:20. https:\/\/doi.org\/10.1038\/s41746-024-01010-1","journal-title":"npj Digit Med"},{"key":"6317_CR20","first-page":"1877","volume":"33","author":"TB Brown","year":"2020","unstructured":"Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877\u20131901","journal-title":"Adv Neural Inf Process Syst"},{"key":"6317_CR21","doi-asserted-by":"publisher","first-page":"e55318","DOI":"10.2196\/55318","volume":"12","author":"S Sivarajkumar","year":"2024","unstructured":"Sivarajkumar S, Kelley M, Samolyk-Mazzanti A, Visweswaran S, Wang Y (2024) An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study. JMIR Med Inf 12:e55318. https:\/\/doi.org\/10.2196\/55318","journal-title":"JMIR Med Inf"},{"key":"6317_CR22","doi-asserted-by":"publisher","first-page":"e92315","DOI":"10.2196\/92315","volume":"28","author":"Z Ma","year":"2026","unstructured":"Ma Z, Chen R, Wang A, Xi Y, Liang M, Yuan S et al (2026) Performance of DeepSeek V3.2 and ChatGPT 5.1 in musculoskeletal triage and differential diagnosis of outpatients with low back pain: multidimensional comparative study. J Med Internet Res 28:e92315. https:\/\/doi.org\/10.2196\/92315","journal-title":"J Med Internet Res"},{"key":"6317_CR23","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","volume":"30","author":"P Hager","year":"2024","unstructured":"Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M et al (2024) Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 30:2613\u20132622. https:\/\/doi.org\/10.1038\/s41591-024-03097-1","journal-title":"Nat Med"},{"issue":"6","key":"6317_CR24","doi-asserted-by":"publisher","first-page":"2233","DOI":"10.1007\/s10067-025-07417-9","volume":"44","author":"N\u00c7 \u00c7elik","year":"2025","unstructured":"\u00c7elik N\u00c7, Altunel K\u0131l\u0131n\u00e7 E (2025) Assessment of ChatGPT\u2019s adherence to EULAR diagnostic criteria and therapeutic protocols for rheumatoid arthritis at two distinct time points, 14 days apart. Clin Rheumatol 44(6):2233\u20132239. https:\/\/doi.org\/10.1007\/s10067-025-07417-9","journal-title":"Clin Rheumatol"},{"key":"6317_CR25","doi-asserted-by":"publisher","first-page":"763","DOI":"10.1093\/rheumatology\/keg213","volume":"42","author":"GJ Gormley","year":"2003","unstructured":"Gormley GJ, Steele WK, Gilliland A, Leggett P, Wright GD, Bell AL et al (2003) Can diagnostic triage by general practitioners or rheumatology nurses improve the positive predictive value of referrals to early arthritis clinics? Rheumatology (Oxford) 42:763\u2013768. https:\/\/doi.org\/10.1093\/rheumatology\/keg213","journal-title":"Rheumatology (Oxford)"},{"key":"6317_CR26","doi-asserted-by":"publisher","first-page":"e12","DOI":"10.1016\/S2589-7500(23)00225-X","volume":"6","author":"T Zack","year":"2024","unstructured":"Zack T, Lehman E, Suzgun M, Rodriguez JA, Celi LA, Gichoya J et al (2024) Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health 6:e12\u2013e22. https:\/\/doi.org\/10.1016\/S2589-7500(23)00225-X","journal-title":"Lancet Digit Health"},{"key":"6317_CR27","doi-asserted-by":"publisher","unstructured":"Wang X, Wei J, Schuurmans D, Le Q, Chi E, Narang S et al (2022) Self-consistency improves chain of thought reasoning in language models.https:\/\/doi.org\/10.48550\/arXiv.2203.11171","DOI":"10.48550\/arXiv.2203.11171"}],"container-title":["Rheumatology International"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00296-026-06317-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00296-026-06317-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00296-026-06317-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T03:03:04Z","timestamp":1792983784000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00296-026-06317-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,9]]},"references-count":27,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["6317"],"URL":"https:\/\/doi.org\/10.1007\/s00296-026-06317-8","relation":{},"ISSN":["1437-160X"],"issn-type":[{"value":"1437-160X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,9]]},"assertion":[{"value":"2 August 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 September 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 October 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The author declares that they have no competing interests.","order":1,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The study was conducted under an ethics waiver governed by the institutional Human Research Ethics Committee and followed the principles of the Declaration of Helsinki. No patient-identifiable information was used in the study.","order":2,"name":"Ethics","label":"Ethical approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":3,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"283"}}