{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T20:09:13Z","timestamp":1790194153805,"version":"4.1.0"},"reference-count":23,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"1","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":["J Am Acad Orthop Surg","JAAOS"],"published-print":{"date-parts":[[2026,1,1]]},"abstract":"<jats:sec>\n                    <jats:title>Introduction:<\/jats:title>\n                    <jats:p>Large language models (LLMs) are promising tools for clinical decision support but require thorough validation to ensure safety and reliability. This study assessed a knowledge and intelligence messaging interface (KIMI; RevelAi Health), an LLM enhanced with retrieval-augmented generation configured with American Academy of Orthopaedic Surgeons guidelines for distal radius fracture management and a persistent system-prompt layer. The goal was to evaluate KIMI's efficacy in acuity triaging and generating appropriate patient-facing responses for distal radius fracture management.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>We analyzed KIMI-generated responses to 100 simulated patient queries. Four clinical experts independently assessed responses for guideline concordance, safety, clarity, and acuity. Probabilities for adequate scoring in all domains were modeled. Bayesian mixed-effects logistic regression and ordered logistic regression models were used for binary and ordinal scoring outcomes, respectively, to account for repeated measures and within-reviewer correlations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>Reviewer evaluations of KIMI responses demonstrated high performance across safety and quality domains. Posterior average probability of responses being rated as safe was 94.2% (95% credible interval [CI]: 91.2 to 96.9), as concordant was 88.7% (95% CI: 85.0 to 92.0), and as clear was 93.7% (95% CI: 90.5 to 96.5). Posterior average probability of exact agreement between reviewer-assigned and LLM-assigned acuity levels was 62.9% (95% CI: 58.0 to 67.7). Surgical queries were associated with slightly higher safety ratings (95.4% versus 91.3%) and acuity agreement (63.9% versus 60.6%) than nonsurgical queries. Query category markedly influenced acuity agreement. LLM-assigned acuity was markedly associated with reviewer-assigned acuity across all models even when adjusting for both query type and category (odds ratio = 2.66; 95% CI: 1.81 to 3.83).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion:<\/jats:title>\n                    <jats:p>KIMI generated responses that were generally safe, clinically concordant, and clearly communicated. These findings support the feasibility of deploying enhanced LLMs for asynchronous patient engagement in low-to-moderate risk care coordination settings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.5435\/jaaos-d-25-00456","type":"journal-article","created":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T22:00:13Z","timestamp":1756850413000},"page":"e106-e115","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":3,"title":["Artificial Intelligence in Triaging Patient Questions: An Evaluation of a Large Language Model for Distal Radius Fractures"],"prefix":"10.5435","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7777-4226","authenticated-orcid":false,"given":"Riley","family":"Kahan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christine","family":"Shen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricia","family":"Wellborn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Lauder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel","family":"Berchuck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hadi","family":"Javeed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian","family":"Pean","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2155-4558","authenticated-orcid":false,"given":"Andrew","family":"Federer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2025,8,27]]},"reference":[{"key":"R2-20251219","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med"},{"key":"R3-20251219","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"R4-20251219","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1126\/science.adh2586","article-title":"Experimental evidence on the productivity effects of generative artificial intelligence","volume":"381","author":"Noy","year":"2023","journal-title":"Science"},{"key":"R5-20251219","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1038\/s41746-023-00819-6","article-title":"Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers","volume":"6","author":"Gao","year":"2023","journal-title":"NPJ Digit Med"},{"key":"R6-20251219","doi-asserted-by":"crossref","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med"},{"key":"R7-20251219","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1186\/s13054-019-2351-7","article-title":"Emergency department triage prediction of clinical outcomes using machine learning models","volume":"23","author":"Raita","year":"2019","journal-title":"Crit Care"},{"key":"R8-20251219","doi-asserted-by":"crossref","first-page":"e0201016","DOI":"10.1371\/journal.pone.0201016","article-title":"Predicting hospital admission at emergency department triage using machine learning","volume":"13","author":"Hong","year":"2018","journal-title":"PLoS One"},{"key":"R9-20251219","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/j.ajem.2024.12.024","article-title":"Evaluating LLM-Based generative AI tools in emergency triage: A comparative study of ChatGPT Plus, Copilot Pro, and triage nurses","volume":"89","author":"Arslan","year":"2025","journal-title":"Am J Emerg Med"},{"key":"R10-20251219","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1111\/jnu.12055","article-title":"The relationship between emergency department crowding and patient outcomes: A systematic review","volume":"46","author":"Carter","year":"2014","journal-title":"J Nurs Scholarsh"},{"key":"R11-20251219","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1111\/acem.12682","article-title":"Crowding measures associated with the quality of emergency department care: A systematic review","volume":"22","author":"Stang","year":"2015","journal-title":"Acad Emerg Med"},{"key":"R13-20251219","doi-asserted-by":"crossref","first-page":"2120","DOI":"10.1007\/s10439-023-03248-4","article-title":"In reference to \u201crole of chat GPT in public health,\u201d to highlight the AI's Iicorrect Rrference Ggneration","volume":"51","author":"Frosolini","year":"2023","journal-title":"Ann Biomed Eng"},{"key":"R14-20251219","doi-asserted-by":"crossref","first-page":"e46924","DOI":"10.2196\/46924","article-title":"Artificial intelligence can generate fraudulent but authentic-looking scientific medical articles: Pandora's box has been opened","volume":"25","author":"M\u00e1jovsk\u00fd","year":"2023","journal-title":"J Med Internet Res"},{"key":"R15-20251219","doi-asserted-by":"crossref","first-page":"199","DOI":"10.2106\/JBJS.G.00347","article-title":"Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites","volume":"90","author":"Badarudeen","year":"2008","journal-title":"J Bone Joint Surg Am"},{"key":"R16-20251219","doi-asserted-by":"crossref","first-page":"1881","DOI":"10.1016\/j.jhsa.2012.05.021","article-title":"The effect of search term on the quality and accuracy of online information regarding distal radius fractures","volume":"37","author":"Dy","year":"2012","journal-title":"J Hand Surg Am"},{"key":"R17-20251219","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1308\/rcsann.2022.0024","article-title":"Evaluation of the quality of information online for arthroscopic Bankart repair","volume":"105","author":"Shazil Jamal","year":"2023","journal-title":"Ann R Coll Surg Engl"},{"key":"R18-20251219","doi-asserted-by":"crossref","first-page":"825","DOI":"10.1016\/j.jhsa.2016.05.006","article-title":"Readability of patient education materials in hand surgery and health literacy best practices for improvement","volume":"41","author":"Hadden","year":"2016","journal-title":"J Hand Surg Am"},{"key":"R19-20251219","doi-asserted-by":"crossref","first-page":"186.e1","DOI":"10.1016\/j.jhsa.2021.03.025","article-title":"Readability of online hand surgery patient educational materials: Evaluating the trend since 2008","volume":"47","author":"Zhang","year":"2022","journal-title":"J Hand Surg Am"},{"key":"R20-20251219","doi-asserted-by":"crossref","first-page":"986","DOI":"10.1016\/j.jhsa.2024.05.007","article-title":"Optimizing readability of patient-facing hand surgery education materials using chat generative pretrained transformer (ChatGPT) 3.5","volume":"49","author":"Vallurupalli","year":"2024","journal-title":"J Hand Surg Am"},{"key":"R21-20251219","doi-asserted-by":"crossref","first-page":"795","DOI":"10.1177\/15589447241238372","article-title":"Experimenting with the new frontier: Artificial intelligence-powered chat bots in hand surgery","volume":"20","author":"Al Rawi","year":"2025","journal-title":"Hand (N Y)"},{"key":"R22-20251219","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.jhsa.2023.10.019","article-title":"Appropriateness and reliability of an online artificial intelligence platform's responses to common questions regarding distal radius fractures","volume":"49","author":"Christy","year":"2024","journal-title":"J Hand Surg Am"},{"key":"R23-20251219","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1016\/j.jhsa.2023.08.003","article-title":"Evaluation of online artificial intelligence-generated information on common hand procedures","volume":"48","author":"Crook","year":"2023","journal-title":"J Hand Surg Am"},{"key":"R24-20251219","doi-asserted-by":"crossref","first-page":"709","DOI":"10.1001\/jama.2014.4","article-title":"Surgical site infections following ambulatory surgery procedures","volume":"311","author":"Owens","year":"2014","journal-title":"JAMA"},{"key":"R25-20251219","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1038\/s43856-024-00717-2","article-title":"Current applications and challenges in large language models for patient care: A systematic review","volume":"5","author":"Busch","year":"2025","journal-title":"Commun Med (Lond)"}],"container-title":["Journal of the American Academy of Orthopaedic Surgeons"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ovid.com\/10.5435\/JAAOS-D-25-00456","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T14:39:11Z","timestamp":1787668751000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ovid.com\/10.5435\/JAAOS-D-25-00456"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,27]]},"references-count":23,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.5435\/jaaos-d-25-00456","relation":{},"ISSN":["1067-151X","1940-5480"],"issn-type":[{"value":"1067-151X","type":"print"},{"value":"1940-5480","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,27]]},"assertion":[{"value":"2025-04-09","name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}}]}}