{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:57:43Z","timestamp":1782845863085,"version":"3.54.5"},"reference-count":59,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Neurotech"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Multimodal artificial intelligence systems combining text and image analysis represent a paradigm shift in clinical decision support. While GPT-4 with Vision (GPT-4V) has shown promise in medical imaging interpretation, existing studies report inconsistent performance (16%\u201080% accuracy) across radiological subspecialties. Critical knowledge gaps persist regarding GPT-4V\u2019s capability to integrate clinical history with imaging findings in complex neuroradiology scenarios, and fundamental questions remain about whether the model appropriately balances visual and textual information sources when formulating diagnoses. Furthermore, documented artificial intelligence hallucination rates of 35.5% to 63% in radiology applications raise urgent safety concerns, yet the relationship between modality utilization patterns and diagnostic accuracy remains unexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate GPT-4V\u2019s diagnostic accuracy on expert-validated neuroradiology board-style examination questions and to examine the model\u2019s self-reported reliance on imaging versus clinical text data when making diagnostic decisions. A secondary objective was to examine whether self-characterized modality utilization patterns differed systematically between correct and incorrect diagnoses, potentially identifying specific failure modes requiring targeted mitigation strategies.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This cross-sectional study evaluated GPT-4V using 29 neuroradiology cases from the RSNA (Radiological Society of North America) Case Collection, covering adult brain and central nervous system pathologies imaged via computed tomography or magnetic resonance imaging. The cases were authored by board-certified radiologists. GPT-4V was accessed via ChatGPT Plus (July 2024) with standardized prompts selecting 1 answer from 4 options, providing diagnostic rationale, and quantifying the percentage contributions of image versus text data. Binary scoring assessed diagnostic performance (correct=1, incorrect=0). Statistical analysis included Wilson score CIs, a binomial test comparing accuracy to chance, and a 2-tailed\n t<\/jats:italic>\n test comparing self-reported modality reliance between correct and incorrect diagnoses (\u03b1=.05, Cohen\n d<\/jats:italic>\n calculated).\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n GPT-4V correctly diagnosed 22 of 29 cases (76% accuracy, 95% CI 57.9%-87.8%), significantly exceeding the chance performance of 25% (\n z<\/jats:italic>\n =6.33;\n P<\/jats:italic>\n <\n .<\/jats:italic>\n 001). The model self-reported mean contributions of 66.1% from imaging (95% CI 63.5%\u201068.8%) and 33.9% from text (95% CI 31.2%\u201036.5%). Correct diagnoses (n=22) showed significantly lower self-reported image reliance (62.8%, 95% CI 61.3%\u201064.3%) compared to incorrect diagnoses (n=7; 76.7%, 95% CI 73.5%\u201080.0%), with a mean difference of 13.9 percentage points (95% CI 10.6\u201017.3;\n P<\/jats:italic>\n <.001; Cohen\n d<\/jats:italic>\n =4.08, 95% CI 2.73\u20105.43). All 7 incorrect diagnoses demonstrated image-dominant attribution \u226570% (Fisher exact test\n P<\/jats:italic>\n <\n .<\/jats:italic>\n 001), suggesting that excessive visual reliance may indicate diagnostic risk.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The 76% accuracy substantially exceeds prior GPT-4V radiology studies (43%), demonstrating that focused domain application with structured prompting enhances performance. Incorrect diagnoses are associated with higher self-reported visual reliance, suggesting a potential failure mode warranting experimental validation. This pattern identifies a potentially actionable signal for quality assurance systems. Clinical deployment should remain restricted to supervised educational applications with mandatory radiologist oversight until balanced context-aware integration is validated.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/69708","type":"journal-article","created":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T06:40:06Z","timestamp":1768977606000},"page":"e69708-e69708","source":"Crossref","is-referenced-by-count":0,"title":["Diagnostic Accuracy of GPT-4 With Vision in Neuroradiology Board-Style Exam Questions: Cross-Sectional Case-Based Study"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-0190-8343","authenticated-orcid":false,"given":"Tom T","family":"Sussan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9515-7696","authenticated-orcid":false,"given":"Rebekah R","family":"Brawley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1755-6596","authenticated-orcid":false,"given":"Joshua","family":"Eckroth","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-3512-293X","authenticated-orcid":false,"given":"James E","family":"Mossell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3152-075X","authenticated-orcid":false,"given":"Tao","family":"Weitao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,4,30]]},"reference":[{"issue":"6663","key":"R1","doi-asserted-by":"publisher","DOI":"10.1126\/science.adk6139","article-title":"As artificial intelligence goes multimodal, medical applications multiply","volume":"381","author":"Topol","journal-title":"Science"},{"issue":"9","key":"R2","doi-asserted-by":"publisher","first-page":"1773","DOI":"10.1038\/s41591-022-01981-2","article-title":"Multimodal biomedical AI","volume":"28","author":"Acosta","journal-title":"Nat Med"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01649-4","article-title":"Multimodal generative AI for interpreting 3D medical images and videos","volume":"8","author":"Lee","journal-title":"NPJ Digit Med"},{"issue":"4","key":"R4","doi-asserted-by":"publisher","first-page":"303","DOI":"10.4274\/dir.2024.242631","article-title":"The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review","volume":"31","author":"Simon","journal-title":"Diagn Interv Radiol"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Huang SC Jensen M Yeung-Levy S Lungren MP Poon H Chaudhari AS . A systematic review and implementation guidelines of multimodal foundation models in medical imaging. Res Sq. Preprint posted online on Apr 28, 2025. [doi: 10.21203\/rs.3.rs-5537908\/v1]","DOI":"10.21203\/rs.3.rs-5537908\/v1"},{"key":"R6","doi-asserted-by":"crossref","unstructured":"Yildirim N Richardson H Wetscherek MT et al. Multimodal healthcare AI: identifying and designing clinically relevant vision-language applications for radiology. Presented at: CHI \u201924: Proceedings of the 2024 CHI Conference on Human Factors in Computing System; May 11-16, 2024:1-22; Honolulu HI, USA. [doi: 10.1145\/3613904.3642013]","DOI":"10.1145\/3613904.3642013"},{"issue":"8","key":"R7","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"9","key":"R8","doi-asserted-by":"publisher","first-page":"3872","DOI":"10.1016\/j.acra.2024.06.046","article-title":"Large language models as tools to generate radiology board-style multiple-choice questions","volume":"31","author":"Mistry","journal-title":"Acad Radiol"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1055\/s-0043-1774743","article-title":"ChatGPT: chasing the storm in radiology training and education","volume":"33","author":"Sodhi","journal-title":"Indian J Radiol Imaging"},{"issue":"4","key":"R10","doi-asserted-by":"publisher","first-page":"1959","DOI":"10.1007\/s00330-024-11035-5","article-title":"Assessing GPT-4 multimodal performance in radiological image analysis","volume":"35","author":"Brin","journal-title":"Eur Radiol"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.101101","article-title":"A comparative evaluation of GPT-4 Turbo and Gemini-Pro in medical licensing exams: enhancing artificial intelligence\u2019s role in medical education","volume":"18","author":"Sussan","journal-title":"Cureus"},{"issue":"3","key":"R12","doi-asserted-by":"publisher","first-page":"1111","DOI":"10.1007\/s00330-024-11115-6","article-title":"Revolution or risk?\u2014Assessing the potential and challenges of GPT-4V in radiologic image interpretation","volume":"35","author":"Huppertz","journal-title":"Eur Radiol"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.233270","article-title":"Evaluating GPT-4V (GPT-4 with vision) on detection of radiologic findings on chest radiographs","volume":"311","author":"Zhou","journal-title":"Radiology"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240273","article-title":"Comparing diagnostic accuracy of radiologists versus GPT-4V and Gemini Pro Vision using image inputs from diagnosis please cases","volume":"312","author":"Suh","journal-title":"Radiology"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1007\/s00062-024-01426-y","article-title":"Comparing the diagnostic performance of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists in challenging neuroradiology cases","volume":"34","author":"Horiuchi","journal-title":"Clin Neuroradiol"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240153","article-title":"Performance of GPT-4 with vision on text-and image-based ACR diagnostic radiology in-training examination questions","volume":"312","author":"Hayden","journal-title":"Radiology"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-06458-z","article-title":"Evaluating diagnostic accuracy of large language models in neuroradiology cases using image inputs from JAMA neurology and JAMA clinical challenges","volume":"15","author":"Albaqshi","journal-title":"Sci Rep"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.3389\/fradi.2025.1670517","article-title":"A systematic review and meta-analysis of GPT-based differential diagnostic accuracy in radiological cases: 2023-2025","volume":"5","author":"Nguyen","journal-title":"Front Radiol"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01208-3","article-title":"Evaluating multimodal AI in medical diagnostics","volume":"7","author":"Kaczmarczyk","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240689","article-title":"Impact of multimodal prompt elements on diagnostic performance of GPT-4V in challenging brain MRI cases","volume":"314","author":"Schramm","journal-title":"Radiology"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/54948","article-title":"Integrating text and image analysis: exploring GPT-4V\u2019s capabilities in advanced radiological applications across subspecialties","volume":"26","author":"Busch","journal-title":"J Med Internet Res"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.2196\/55627","article-title":"Evaluating ChatGPT-4's diagnostic accuracy: impact of visual data integration","volume":"12","author":"Hirosawa","journal-title":"JMIR Med Inform"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","first-page":"506","DOI":"10.1007\/s00330-024-10902-5","article-title":"ChatGPT\u2019s diagnostic performance based on textual vs. visual information compared to radiologists\u2019 diagnostic performance in musculoskeletal radiology","volume":"35","author":"Horiuchi","journal-title":"Eur Radiol"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01185-7","article-title":"Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine","volume":"7","author":"Jin","journal-title":"NPJ Digit Med"},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"110101","DOI":"10.1016\/j.clinimag.2024.110101","article-title":"When vision meets reality: exploring the clinical applicability of GPT-4 with vision","volume":"108","author":"Deng","journal-title":"Clin Imaging"},{"issue":"12","key":"R26","doi-asserted-by":"publisher","first-page":"1849","DOI":"10.1007\/s11547-024-01915-7","article-title":"ChatGPT and radiology report: potential applications and limitations","volume":"129","author":"Parillo","journal-title":"Radiol Med"},{"issue":"10","key":"R27","doi-asserted-by":"publisher","first-page":"990","DOI":"10.1016\/j.jacr.2023.05.003","article-title":"Evaluating GPT as an adjunct for radiologic decision making: GPT-4 versus GPT-3.5 in a breast imaging pilot","volume":"20","author":"Rao","journal-title":"J Am Coll Radiol"},{"issue":"14","key":"R28","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics14141541","article-title":"Optimizing GPT-4 turbo diagnostic accuracy in neuroradiology through prompt engineering and confidence thresholds","volume":"14","author":"Wada","journal-title":"Diagnostics (Basel)"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240955","article-title":"Toward foundation models in radiology? Quantitative assessment of GPT-4V's multimodal and multianatomic region capabilities","volume":"313","author":"Strotzer","journal-title":"Radiology"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232756","article-title":"Chatbots and large language models in radiology: a practical primer for clinical and research applications","volume":"310","author":"Bhayana","journal-title":"Radiology"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1037\/amp0000191","article-title":"Journal article reporting standards for quantitative research in psychology: the APA Publications and Communications Board task force report","volume":"73","author":"Appelbaum","journal-title":"Am Psychol"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","DOI":"10.1186\/s13244-020-00924-0","article-title":"How to perform an excellent radiology board examination: a web-based checklist","volume":"12","author":"Dicle","journal-title":"Insights Imaging"},{"key":"R33","unstructured":"Definitions for purposes of this policy (45 CFR \u00a746.102). National Code of Federal Regulations. URL: https:\/\/www.ecfr.gov\/current\/title-45\/subtitle-A\/subchapter-A\/part-46\/subpart-A\/section-46.102 [Accessed 20-04-2026]"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240609","article-title":"Evaluation of GPT large language model performance on RSNA 2023 case of the day questions","volume":"313","author":"Mukherjee","journal-title":"Radiology"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00341-z","article-title":"Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines","volume":"3","author":"Huang","journal-title":"NPJ Digit Med"},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Alsentzer E Murphy J Boag W . Publicly available clinical BERT embeddings. Presented at: Proceedings of the 2nd Clinical Natural Language Processing Workshop; Jun 7, 2019:72-78; Minneapolis, Minnesota, USA. 2019.[doi: 10.18653\/v1\/W19-1909]","DOI":"10.18653\/v1\/W19-1909"},{"issue":"1","key":"R37","doi-asserted-by":"publisher","first-page":"59","DOI":"10.7326\/M19-2548","article-title":"Should health care demand interpretable artificial intelligence or accept \u201cblack box\u201d medicine?","volume":"172","author":"Wang","journal-title":"Ann Intern Med"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-020-01332-6","article-title":"Explainability for artificial intelligence in healthcare: a multidisciplinary perspective","volume":"20","author":"Amann","journal-title":"BMC Med Inform Decis Mak"},{"issue":"3","key":"R39","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1007\/s00234-005-0024-5","article-title":"Pattern of T2 hypointensity associated with ring-enhancing brain lesions can help to differentiate pathology","volume":"48","author":"Schwartz","journal-title":"Neuroradiology"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.230513","article-title":"Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR and RSNA","volume":"6","author":"Brady","journal-title":"Radiol Artif Intell"},{"issue":"3","key":"R41","doi-asserted-by":"publisher","first-page":"713","DOI":"10.1148\/radiol.2017171183","article-title":"When machines think: radiology's next frontier","volume":"285","author":"Dreyer","journal-title":"Radiology"},{"issue":"2","key":"R42","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2020200029","article-title":"Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers","volume":"2","author":"Mongan","journal-title":"Radiol Artif Intell"},{"issue":"1","key":"R43","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard","journal-title":"J Am Med Inform Assoc"},{"issue":"23","key":"R44","doi-asserted-by":"publisher","first-page":"2275","DOI":"10.1001\/jama.2023.22295","article-title":"Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study","volume":"330","author":"Jabbour","journal-title":"JAMA"},{"key":"R45","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0253-3","article-title":"Presenting machine learning model information to clinical end users with model facts labels","volume":"3","author":"Sendak","journal-title":"NPJ Digit Med"},{"issue":"4","key":"R46","doi-asserted-by":"publisher","first-page":"582","DOI":"10.1038\/s41591-021-01312-x","article-title":"How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals","volume":"27","author":"Wu","journal-title":"Nat Med"},{"issue":"9","key":"R47","doi-asserted-by":"publisher","first-page":"1364","DOI":"10.1038\/s41591-020-1034-x","article-title":"Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension","volume":"26","author":"Liu","journal-title":"Nat Med"},{"issue":"1","key":"R48","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1186\/s13244-023-01595-3","article-title":"A framework to integrate artificial intelligence training into radiology residency programs: preparing the future radiologist","volume":"15","author":"van Kooten","journal-title":"Insights Imaging"},{"issue":"6","key":"R49","doi-asserted-by":"publisher","first-page":"747","DOI":"10.1016\/j.acra.2018.03.007","article-title":"Toward augmented radiologists: changes in radiology education in the era of machine learning and artificial intelligence","volume":"25","author":"Tajmir","journal-title":"Acad Radiol"},{"issue":"1","key":"R50","doi-asserted-by":"publisher","DOI":"10.1186\/s41747-018-0061-6","article-title":"Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine","volume":"2","author":"Pesapane","journal-title":"Eur Radiol Exp"},{"issue":"6","key":"R51","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1001\/jama.2017.7797","article-title":"Unintended consequences of machine learning in medicine","volume":"318","author":"Cabitza","journal-title":"JAMA"},{"issue":"2","key":"R52","doi-asserted-by":"publisher","DOI":"10.2196\/16048","article-title":"Introducing artificial intelligence training in medical education","volume":"5","author":"Paranjape","journal-title":"JMIR Med Educ"},{"issue":"5","key":"R53","doi-asserted-by":"publisher","first-page":"614","DOI":"10.1067\/j.cpradiol.2020.06.011","article-title":"Medical student perspectives on the impact of artificial intelligence on the practice of medicine","volume":"50","author":"Park","journal-title":"Curr Probl Diagn Radiol"},{"issue":"6","key":"R54","doi-asserted-by":"publisher","first-page":"1668","DOI":"10.1148\/rg.2015150023","article-title":"Understanding and confronting our mistakes: the epidemiology of error in radiology and strategies for error reduction","volume":"35","author":"Bruno","journal-title":"Radiographics"},{"issue":"4","key":"R55","doi-asserted-by":"publisher","first-page":"739","DOI":"10.2214\/AJR.16.16963","article-title":"Interpretive error in radiology","volume":"208","author":"Waite","journal-title":"AJR Am J Roentgenol"},{"issue":"9","key":"R56","doi-asserted-by":"publisher","first-page":"698","DOI":"10.1016\/j.jacr.2010.03.004","article-title":"Long radiology workdays reduce detection and accommodation accuracy","volume":"7","author":"Krupinski","journal-title":"J Am Coll Radiol"},{"issue":"1","key":"R57","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1148\/rg.2018170107","article-title":"Bias in radiology: the how and why of misses and misinterpretations","volume":"38","author":"Busby","journal-title":"Radiographics"},{"issue":"5","key":"R58","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.2214\/ajr.175.5.1751233","article-title":"Interpretation of emergency department radiographs: a comparison of emergency medicine physicians with radiologists, residents with faculty, and film with digital display","volume":"175","author":"Eng","journal-title":"AJR Am J Roentgenol"},{"issue":"3","key":"R59","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1148\/radiol.11102222","article-title":"Rethinking peer review: what aviation can teach radiology about performance improvement","volume":"259","author":"Larson","journal-title":"Radiology"}],"container-title":["JMIR Neurotechnology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/neuro.jmir.org\/2026\/1\/e69708","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:57:47Z","timestamp":1782842267000},"score":1,"resource":{"primary":{"URL":"https:\/\/neuro.jmir.org\/2026\/1\/e69708"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,30]]},"references-count":59,"URL":"https:\/\/doi.org\/10.2196\/69708","relation":{},"ISSN":["2817-092X"],"issn-type":[{"value":"2817-092X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,30]]},"article-number":"v5i6663e69708"}}