{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T03:04:02Z","timestamp":1787799842609,"version":"build-2784847793"},"reference-count":25,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"16","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,8,20]]},"abstract":"<jats:sec>\n                    <jats:title>Background:<\/jats:title>\n                    <jats:p>Privacy-preserving large language models (PP-LLMs) hold potential for assisting clinicians with documentation. We evaluated a PP-LLM to improve the clinical information on radiology request forms for musculoskeletal magnetic resonance imaging (MRI) and to automate protocoling, which ensures that the most appropriate imaging is performed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>The present retrospective study included musculoskeletal MRI radiology request forms that had been randomly collected from June to December 2023. Studies without electronic medical record (EMR) entries were excluded. An institutional PP-LLM (Claude Sonnet 3.5) augmented the original radiology request forms by mining EMRs, and, in combination with rule-based processing of the LLM outputs, suggested appropriate protocols using institutional guidelines. Clinical information on the original and PP-LLM radiology request forms were compared with use of the RI-RADS (Reason for exam Imaging Reporting and Data System) grading by 2 musculoskeletal (MSK) radiologists independently (MSK1, with 13 years of experience, and MSK2, with 11 years of experience). These radiologists established a consensus reference standard for protocoling, against which the PP-LLM and of 2 second-year board-certified radiologists (RAD1 and RAD2) were compared. Inter-rater reliability was assessed with use of the Gwet AC1, and the percentage agreement with the reference standard was calculated.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>Overall, 500 musculoskeletal MRI radiology request forms were analyzed for 407 patients (202 women and 205 men with a mean age [and standard deviation] of 50.3 \u00b1 19.5 years) across a range of anatomical regions, including the spine\/pelvis (143 MRI scans; 28.6%), upper extremity (169 scans; 33.8%) and lower extremity (188 scans; 37.6%). Two hundred and twenty-two (44.4%) of the 500 MRI scans required contrast. The clinical information provided in the PP-LLM-augmented radiology request forms was rated as superior to that in the original requests. Only 0.4% to 0.6% of PP-LLM radiology request forms were rated as limited\/deficient, compared with 12.4% to 22.6% of the original requests (p &lt; 0.001). Almost-perfect inter-rater reliability was observed for LLM-enhanced requests (AC1 = 0.99; 95% confidence interval [CI], 0.99 to 1.0), compared with substantial agreement for the original forms (AC1 = 0.62; 95% CI, 0.56 to 0.67). For protocoling, MSK1 and MSK2 showed almost-perfect agreement on the region\/coverage (AC1 = 0.96; 95% CI, 0.95 to 0.98) and contrast requirement (AC1 = 0.98; 95% CI, 0.97 to 0.99). Compared with the consensus reference standard, protocoling accuracy for the PP-LLM was 95.8% (95% CI, 94.0% to 97.6%), which was significantly higher than that for both RAD1 (88.6%; 95% CI, 85.8% to 91.4%) and RAD2 (88.2%; 95% CI, 85.4% to 91.0%) (p &lt; 0.001 for both).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions:<\/jats:title>\n                    <jats:p>Musculoskeletal MRI request form augmentation with an institutional LLM provided superior clinical information and improved protocoling accuracy compared with clinician requests and non-MSK-trained radiologists. Institutional adoption of such LLMs could enhance the appropriateness of MRI utilization and patient care.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Level of Evidence:<\/jats:title>\n                    <jats:p>\n                      Diagnostic\n                      <jats:underline>Level III<\/jats:underline>\n                      . See Instructions for Authors for a complete description of levels of evidence.\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.2106\/jbjs.24.01429","type":"journal-article","created":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T14:01:30Z","timestamp":1751983290000},"page":"1833-1840","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":4,"title":["An Institutional Large Language Model for Musculoskeletal MRI Improves Protocol Adherence and Accuracy"],"prefix":"10.2106","volume":"107","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0082-9033","authenticated-orcid":false,"given":"James Thomas Patrick Decourcy","family":"Hallinan","sequence":"first","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"},{"name":"Department of Diagnostic Radiology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6190-9478","authenticated-orcid":false,"given":"Naomi Wenxin","family":"Leow","sequence":"additional","affiliation":[{"name":"AIO Innovation Office, National University Health System, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi Xian","family":"Low","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8789-2029","authenticated-orcid":false,"given":"Aric","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0262-3802","authenticated-orcid":false,"given":"Wilson","family":"Ong","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7014-9376","authenticated-orcid":false,"given":"Matthew Ding Zhou","family":"Chan","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ganakirthana Kalpenya","family":"Devi","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie Shengjie","family":"He","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4125-1733","authenticated-orcid":false,"given":"Daniel De-Liang","family":"Loh","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Desmond Shi Wei","family":"Lim","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1240-5908","authenticated-orcid":false,"given":"Xi Zhen","family":"Low","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7053-2083","authenticated-orcid":false,"given":"Ee Chin","family":"Teo","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shaheryar Mohammad","family":"Furqan","sequence":"additional","affiliation":[{"name":"Division of Biomedical Informatics, Department of Surgery, Yong Loo Lin School of Medicine, National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wilson Wei Yang","family":"Tham","sequence":"additional","affiliation":[{"name":"National University Spine Institute, Department of Orthopaedic Surgery, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiong Hao","family":"Tan","sequence":"additional","affiliation":[{"name":"National University Spine Institute, Department of Orthopaedic Surgery, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naresh","family":"Kumar","sequence":"additional","affiliation":[{"name":"National University Spine Institute, Department of Orthopaedic Surgery, National University Hospital, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9631-4121","authenticated-orcid":false,"given":"Andrew","family":"Makmur","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"},{"name":"Department of Diagnostic Radiology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8682-9914","authenticated-orcid":false,"given":"Ting","family":"Yonghan","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Imaging, National University Hospital, Singapore"},{"name":"Department of Diagnostic Radiology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2025,7,8]]},"reference":[{"issue":"3","key":"bib1-20250820","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1111\/1754-9485.12577","article-title":"Quality of referral: What information should be included in a request for diagnostic imaging when a patient is referred to a clinical radiologist?","volume":"61","author":"G","year":"2017","journal-title":"J Med Imaging Radiat Oncol."},{"issue":"2","key":"bib2-20250820","first-page":"e13335","article-title":"Evaluation of Radiology Request Forms in a Tertiary Care Hospital: An Audit With a Focus on the Impact of Technological Intervention","volume":"13","author":"Barakzai","year":"2021","journal-title":"Cureus."},{"issue":"12","key":"bib3-20250820","doi-asserted-by":"crossref","first-page":"844","DOI":"10.1016\/j.jacr.2009.08.003","article-title":"Strategies for managing imaging utilization","volume":"6","author":"Bernardy","year":"2009","journal-title":"J Am Coll Radiol."},{"issue":"3","key":"bib4-20250820","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1016\/j.jacr.2009.11.010","article-title":"Analysis of appropriateness of outpatient CT and MRI referred from primary care clinics at an academic medical center: how critical is the need for improved decision support?","volume":"7","author":"Lehnert","year":"2010","journal-title":"J Am Coll Radiol."},{"issue":"2","key":"bib5-20250820","doi-asserted-by":"crossref","first-page":"e210114","DOI":"10.1148\/ryai.210114","article-title":"Overview of Noninterpretive Artificial Intelligence Models for Safety, Quality, Workflow, and Education Applications in Radiology Practice","volume":"4","author":"Tadavarthi","year":"2022","journal-title":"Radiol Artif Intell."},{"issue":"2","key":"bib6-20250820","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1007\/s11547-024-01762-6","article-title":"Structured request form in musculoskeletal radiology examinations (CONCERTO): results of an expert Delphi consensus-structured radiology request form for correct classification of patients to undergo radiological examinations of the Italian Society of Medical and Interventional Radiology (SIRM), the Italian Society of Rheumatology (SIR) and the Italian Society of Orthopedics and Traumatology (SIOT)","volume":"129","author":"Salaffi","year":"2024","journal-title":"Radiol Med."},{"issue":"8","key":"bib7-20250820","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1016\/j.amjmed.2012.11.034","article-title":"Impact of provider-led, technology-enabled radiology management program on imaging","volume":"126","author":"Ip","year":"2013","journal-title":"Am J Med."},{"issue":"2","key":"bib8-20250820","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/s10278-017-0021-3","article-title":"Automatic Determination of the Need for Intravenous Contrast in Musculoskeletal MRI Examinations Using IBM Watson\u2019s Natural Language Processing Algorithm","volume":"31","author":"Trivedi","year":"2018","journal-title":"J Digit Imaging."},{"issue":"1","key":"bib9-20250820","doi-asserted-by":"crossref","first-page":"e232756","DOI":"10.1148\/radiol.232756","article-title":"Chatbots and Large Language Models in Radiology: A Practical Primer for Clinical and Research Applications","volume":"310","author":"Bhayana","year":"2024","journal-title":"Radiology."},{"issue":"5","key":"bib10-20250820","doi-asserted-by":"crossref","first-page":"e230877","DOI":"10.1148\/radiol.230877","article-title":"GPT-4 for Automated Determination of Radiological Study and Protocol based on Radiology Request Forms: A Feasibility Study","volume":"307","author":"Gertz","year":"2023","journal-title":"Radiology."},{"issue":"10","key":"bib11-20250820","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1016\/j.jacr.2023.06.009","article-title":"ChatGPT-4 Assistance in Optimizing Emergency Department Radiology Referrals and Imaging Selection","volume":"20","author":"Barash","year":"2023","journal-title":"J Am Coll Radiol."},{"issue":"5","key":"bib12-20250820","doi-asserted-by":"crossref","first-page":"e231259","DOI":"10.1148\/radiol.231259","article-title":"Evaluating GPT4 on Impressions Generation in Radiology Reports","volume":"307","author":"Sun","year":"2023","journal-title":"Radiology."},{"issue":"12","key":"bib13-20250820","first-page":"e50881","article-title":"Assessing ChatGPT\u2019s Proficiency in Simplifying Radiological Reports for Healthcare Professionals and Patients","volume":"15","author":"Sarangi","year":"2023","journal-title":"Cureus."},{"issue":"2","key":"bib14-20250820","doi-asserted-by":"crossref","first-page":"e232561","DOI":"10.1148\/radiol.232561","article-title":"Accuracy of ChatGPT, Google Bard, and Microsoft Bing for Simplifying Radiology Reports","volume":"309","author":"Amin","year":"2023","journal-title":"Radiology."},{"issue":"1","key":"bib15-20250820","doi-asserted-by":"crossref","first-page":"e230970","DOI":"10.1148\/radiol.230970","article-title":"A Context-based Chatbot Surpasses Trained Radiologists and Generic ChatGPT in Following the ACR Appropriateness Guidelines","volume":"308","author":"Rau","year":"2023","journal-title":"Radiology."},{"issue":"2","key":"bib16-20250820","doi-asserted-by":"crossref","first-page":"126","DOI":"10.3348\/kjr.2023.0997","article-title":"Large Language Models: A Guide for Radiologists","volume":"25","author":"Kim","year":"2024","journal-title":"Korean J Radiol."},{"issue":"1","key":"bib17-20250820","doi-asserted-by":"crossref","first-page":"e231147","DOI":"10.1148\/radiol.231147","article-title":"Feasibility of Using the Privacy-preserving Large Language Model Vicuna for Labeling Radiology Reports","volume":"309","author":"Mukherjee","year":"2023","journal-title":"Radiology."},{"issue":"1","key":"bib18-20250820","doi-asserted-by":"crossref","first-page":"e232335","DOI":"10.1148\/radiol.232335","article-title":"Feasibility and Prospect of Privacy-preserving Large Language Models in Radiology","volume":"309","author":"Cai","year":"2023","journal-title":"Radiology."},{"key":"bib19-20250820","doi-asserted-by":"crossref","first-page":"108661","DOI":"10.1016\/j.ejrad.2019.108661","article-title":"Reason for exam Imaging Reporting and Data System (RI-RADS): A grading system to standardize radiology requisitions","volume":"120","author":"Abedi","year":"2019","journal-title":"Eur J Radiol."},{"issue":"Pt 1","key":"bib20-20250820","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","year":"2008","journal-title":"Br J Math Stat Psychol."},{"issue":"2","key":"bib21-20250820","doi-asserted-by":"crossref","first-page":"80","DOI":"10.4274\/dir.2023.232417","article-title":"Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions","volume":"30","author":"Akinci D\u2019Antonoli","year":"2024","journal-title":"Diagn Interv Radiol."},{"key":"bib22-20250820","doi-asserted-by":"crossref","first-page":"109993","DOI":"10.1016\/j.clinimag.2023.109993","article-title":"Improving radiology workflow using ChatGPT and artificial intelligence","volume":"103","author":"Mese","year":"2023","journal-title":"Clin Imaging."},{"issue":"4","key":"bib23-20250820","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med."},{"issue":"1","key":"bib24-20250820","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1002\/jmrs.424","article-title":"The effect of clinical information on radiology reporting: A systematic review","volume":"68","author":"Castillo","year":"2021","journal-title":"J Med Radiat Sci."},{"issue":"1","key":"bib25-20250820","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1038\/s41746-024-01029-4","article-title":"Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs","volume":"7","author":"Wang","year":"2024","journal-title":"NPJ Digit Med."}],"container-title":["Journal of Bone and Joint Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ovid.com\/10.2106\/JBJS.24.01429","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T13:52:58Z","timestamp":1787752378000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ovid.com\/10.2106\/JBJS.24.01429"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,8]]},"references-count":25,"journal-issue":{"issue":"16","published-print":{"date-parts":[[2025]]}},"URL":"https:\/\/doi.org\/10.2106\/jbjs.24.01429","relation":{},"ISSN":["0021-9355","1535-1386"],"issn-type":[{"value":"0021-9355","type":"print"},{"value":"1535-1386","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,8]]}}}