{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:07:59Z","timestamp":1782846479271,"version":"3.54.5"},"reference-count":31,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Incomplete clinical details on magnetic resonance imaging (MRI) examination requests (MERs) can lead to suboptimal protocol selection. An institutional secure large language model (sLLM) with access to manually retrieved salient data from the electronic medical record (EMR) may improve request completeness and protocol accuracy across multiple MRI subspecialties.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The objective of this study was to compare clinician MERs with sLLM-augmented MERs for information quality and to evaluate the protocoling accuracy of the sLLM versus board-certified radiologists across body, musculoskeletal, and neuroradiology MRI.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This retrospective study included 608 random outpatient MRI examinations performed between September 2023 and July 2024 (body 206, musculoskeletal 203, neuroradiology 199). The cohort comprised 528 patients (mean 51.2 years, SD 19.2; range 4\u201093; n=279, 52.8% women, n=249, 47.2% men). MERs without EMR access were excluded. A privately hosted Anthropic Claude 3.5 model (temperature 0) augmented each MER with manually retrieved salient EMR data and, via rule-based parsing, mapped the extracted elements onto predefined institutional criteria to recommend region or coverage and contrast use. Two experienced radiologists established a consensus reference standard. Two board-certified general radiologists (Rad 3 and Rad 4) and the sLLM were compared with this standard. Clinical information quality was graded using the Reason-for-Exam Imaging Reporting and Data System (RI-RADS). Interrater reliability was quantified with Gwet AC1. Paired accuracies were compared with the McNemar test to determine whether there was a statistically significant difference.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Interreader agreement for RI-RADS was almost perfect for sLLM-augmented MERs (AC1 0.97, 95% CI 0.94\u20100.99) and moderate for clinician MERs (AC1 0.43, 95% CI 0.34\u20100.52). Limited or deficient clinical information (RI-RADS C\/D) fell to 0% to 0.7% (0\/608 to 4\/608) with sLLM augmentation vs 4.1% to 20.4% (25\/608 to 124\/608) for clinician MERs. Overall protocol accuracy was 93.1% (566\/608; 95% CI 89.6\u201096.6) for the sLLM, 91.4% (556\/608; 95% CI 87.6\u201095.3) for Rad 3, and 92.1% (560\/608; 95% CI 88.4\u201095.8) for Rad 4 (sLLM vs Rad 3\n P<\/jats:italic>\n =.23 vs Rad 4\n P<\/jats:italic>\n =.40). Region or coverage accuracy was similar (sLLM: 579\/608, 95.2%; Rad 3: 585\/608, 96.2%; Rad 4: 573\/608, 94.2%;\n P<\/jats:italic>\n =.46 and\n P<\/jats:italic>\n =.36). Contrast decisions were more accurate using the sLLM at 94.4% (574\/608; 95% CI 91.3\u201097.5) vs Rad 3 at 92.1% (560\/608; 95% CI 88.4\u201095.8;\n P<\/jats:italic>\n =.027) and were not significantly different to Rad 4 at 92.9% (565\/608; 95% CI 89.4\u201096.4;\n P<\/jats:italic>\n =.16). Subspecialty analyses showed similar patterns, with the sLLM outperforming Rad 4 for musculoskeletal MRI contrast decisions (96.6% vs 91.1%;\n P<\/jats:italic>\n =.006) and matching readers elsewhere. Manual review indicated that sLLM improvements arose from EMR details not listed on the MER (infection\/inflammation, tumor history, prior surgery). No clinically significant hallucinations were identified in a manual review of discordant cases.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Across body, musculoskeletal, and neuroradiology MRI, sLLM-augmented examination requests improved clinical context and enhanced contrast selection while demonstrating accuracy comparable to general radiologists for region or coverage. Integrating sLLMs into routine vetting workflows may reduce manual workload in protocol selection for more efficient, standardized protocoling.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/82579","type":"journal-article","created":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T17:48:11Z","timestamp":1775584091000},"page":"e82579-e82579","source":"Crossref","is-referenced-by-count":0,"title":["Initial Insights Into an Institutional Secure Large Language Model for Magnetic Resonance Imaging Examination Requests: Retrospective Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0082-9033","authenticated-orcid":false,"given":"James Thomas Patrick Decourcy","family":"Hallinan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6190-9478","authenticated-orcid":false,"given":"Naomi Wenxin","family":"Leow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-5374-0390","authenticated-orcid":false,"given":"Yi Xian","family":"Low","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8789-2029","authenticated-orcid":false,"given":"Aric","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0262-3802","authenticated-orcid":false,"given":"Wilson","family":"Ong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7014-9376","authenticated-orcid":false,"given":"Matthew Ding Zhou","family":"Chan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8650-1708","authenticated-orcid":false,"given":"Ganakirthana Kalpenya","family":"Devi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1662-0622","authenticated-orcid":false,"given":"Stephanie Shengjie","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4125-1733","authenticated-orcid":false,"given":"Daniel De-Liang","family":"Loh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8011-9439","authenticated-orcid":false,"given":"Desmond Shi Wei","family":"Lim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1240-5908","authenticated-orcid":false,"given":"Xi Zhen","family":"Low","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2944-6431","authenticated-orcid":false,"given":"Mei Chin","family":"Lim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7060-3649","authenticated-orcid":false,"given":"Clement","family":"Yong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9586-3647","authenticated-orcid":false,"given":"Weizhong Jonathan","family":"Sng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7053-2083","authenticated-orcid":false,"given":"Ee Chin","family":"Teo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9237-2586","authenticated-orcid":false,"given":"Jiong Hao","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4538-5031","authenticated-orcid":false,"given":"Naresh","family":"Kumar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9631-4121","authenticated-orcid":false,"given":"Andrew","family":"Makmur","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8682-9914","authenticated-orcid":false,"given":"Yonghan","family":"Ting","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,4,7]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1016\/j.jacr.2009.11.010","article-title":"Analysis of appropriateness of outpatient CT and MRI referred from primary care clinics at an academic medical center: how critical is the need for improved decision support?","volume":"7","author":"Lehnert","journal-title":"J Am Coll Radiol"},{"issue":"12","key":"R2","doi-asserted-by":"publisher","first-page":"844","DOI":"10.1016\/j.jacr.2009.08.003","article-title":"Strategies for managing imaging utilization","volume":"6","author":"Bernardy","journal-title":"J Am Coll Radiol"},{"issue":"3","key":"R3","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1111\/1754-9485.12577","article-title":"Quality of referral: what information should be included in a request for diagnostic imaging when a patient is referred to a clinical radiologist?","volume":"61","author":"G Pitman","journal-title":"J Med Imaging Radiat Oncol"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.13335","article-title":"Evaluation of radiology request forms in a tertiary care hospital: an audit with a focus on the impact of technological intervention","volume":"13","author":"Barakzai","journal-title":"Cureus"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1002\/jmrs.424","article-title":"The effect of clinical information on radiology reporting: a systematic review","volume":"68","author":"Castillo","journal-title":"J Med Radiat Sci"},{"issue":"8","key":"R6","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1016\/j.amjmed.2012.11.034","article-title":"Impact of provider-led, technology-enabled radiology management program on imaging","volume":"126","author":"Ip","journal-title":"Am J Med"},{"issue":"2","key":"R7","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1007\/s10278-017-0021-3","article-title":"Automatic determination of the need for intravenous contrast in musculoskeletal MRI examinations using IBM Watson\u2019s natural language processing algorithm","volume":"31","author":"Trivedi","journal-title":"J Digit Imaging"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232756","article-title":"Chatbots and large language models in radiology: a practical primer for clinical and research applications","volume":"310","author":"Bhayana","journal-title":"Radiology"},{"issue":"5","key":"R9","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230877","article-title":"GPT-4 for automated determination of radiological study and protocol based on radiology request forms: a feasibility study","volume":"307","author":"Gertz","journal-title":"Radiology"},{"issue":"10","key":"R10","doi-asserted-by":"publisher","first-page":"998","DOI":"10.1016\/j.jacr.2023.06.009","article-title":"ChatGPT-4 assistance in optimizing emergency department radiology referrals and imaging selection","volume":"20","author":"Barash","journal-title":"J Am Coll Radiol"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","first-page":"126","DOI":"10.3348\/kjr.2023.0997","article-title":"Large language models: a guide for radiologists","volume":"25","author":"Kim","journal-title":"Korean J Radiol"},{"issue":"5","key":"R12","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231259","article-title":"Evaluating GPT4 on impressions generation in radiology reports","volume":"307","author":"Sun","journal-title":"Radiology"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinimag.2023.109993","article-title":"Improving radiology workflow using ChatGPT and artificial intelligence","volume":"103","author":"Mese","journal-title":"Clin Imaging"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231147","article-title":"Feasibility of using the privacy-preserving large language model vicuna for labeling radiology reports","volume":"309","author":"Mukherjee","journal-title":"Radiology"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230970","article-title":"A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines","volume":"308","author":"Rau","journal-title":"Radiology"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232335","article-title":"Feasibility and prospect of privacy-preserving large language models in radiology","volume":"309","author":"Cai","journal-title":"Radiology"},{"issue":"3","key":"R17","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1016\/j.spinee.2024.10.021","article-title":"MRI spine request form enhancement and auto protocoling using a secure institutional large language model","volume":"25","author":"Hallinan","journal-title":"Spine J"},{"issue":"16","key":"R18","doi-asserted-by":"publisher","first-page":"1833","DOI":"10.2106\/JBJS.24.01429","article-title":"An institutional large language model for musculoskeletal MRI improves protocol adherence and accuracy","volume":"107","author":"Hallinan","journal-title":"J Bone Joint Surg Am"},{"issue":"2","key":"R19","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.242134","article-title":"Leveraging large language models to generate clinical histories for oncologic imaging requisitions","volume":"314","author":"Bhayana","journal-title":"Radiology"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","first-page":"1541","DOI":"10.1007\/s00330-025-11888-4","article-title":"GPT-4 for automated sequence-level determination of MRI protocols based on radiology request forms from clinical routine","volume":"36","author":"Terzis","journal-title":"Eur Radiol"},{"issue":"2","key":"R21","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.243910","article-title":"AI-generated clinical histories for radiology reports: closing the information gap","volume":"314","author":"Tavakoli","journal-title":"Radiology"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2019.108661","article-title":"Reason for Exam Imaging Reporting and Data System (RI-RADS): a grading system to standardize radiology requisitions","volume":"120","author":"Abedi","journal-title":"Eur J Radiol"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"Asgari","journal-title":"NPJ Digit Med"},{"issue":"Pt 1","key":"R24","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","journal-title":"Br J Math Stat Psychol"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinimag.2025.110573","article-title":"Evaluating large language models for imaging modality selection: potential to reduce unnecessary contrast agent use and radiation exposure","volume":"125","author":"\u00c7amur","journal-title":"Clin Imaging"},{"issue":"6","key":"R26","doi-asserted-by":"publisher","first-page":"4336","DOI":"10.1007\/s10278-025-01433-6","article-title":"Efficacy of fine-tuned large language model in CT protocol assignment as clinical decision-supporting system","volume":"38","author":"Kanemaru","journal-title":"J Imaging Inform Med"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-88925-1","article-title":"ChatGPT performance in assessing musculoskeletal MRI scan appropriateness based on ACR appropriateness criteria","volume":"15","author":"Tan","journal-title":"Sci Rep"},{"issue":"7","key":"R28","doi-asserted-by":"publisher","first-page":"1201","DOI":"10.1016\/j.jvir.2025.03.012","article-title":"Proof-of-concept prompted large language model for radiology procedure request routing","volume":"36","author":"Triana","journal-title":"J Vasc Interv Radiol"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"80","DOI":"10.4274\/dir.2023.232417","article-title":"Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions","volume":"30","author":"Akinci D\u2019Antonoli","journal-title":"Diagn Interv Radiol"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrai.2025.100021","article-title":"Large language models in radiology: a technical and clinical perspective","volume":"2","author":"Kao","journal-title":"Eur J Radiol Artif Intell"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01021-3","article-title":"Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support","volume":"5","author":"Omar","journal-title":"Commun Med (Lond)"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e82579","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:16:34Z","timestamp":1782843394000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e82579"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,7]]},"references-count":31,"URL":"https:\/\/doi.org\/10.2196\/82579","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,7]]},"article-number":"v28i3e82579"}}