{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T16:23:32Z","timestamp":1783527812748,"version":"3.55.0"},"reference-count":30,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:00:00Z","timestamp":1783382400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: To characterize the zero-shot diagnostic behavior of three commercial multimodal large language models (MLLMs) on cardiopulmonary chest radiograph findings and to assess sex-stratified performance differences. Methods: GPT-5.4, Claude Opus 4.5, and Gemini 2.5 Pro were evaluated in 4500 pathology-specific radiograph evaluations based on frontal chest radiographs from the publicly available CheXpert dataset. Three balanced cohorts of 1500 images each were constructed for cardiomegaly, pulmonary edema, and pleural effusion (375 per sex-by-label subgroup). All models received identical zero-shot prompts requesting binary classification. The primary outcome was area under the receiver operating characteristic curve (AUC-ROC) with 95% bootstrap confidence intervals. Secondary outcomes were sensitivity and specificity. Results: A total of 4500 pathology-specific radiograph evaluations were performed across the three cohorts (2250 male and 2250 female cohort entries; mean age 58.4 \u00b1 18.0 years). GPT-5.4 achieved the highest discrimination (AUC-ROC 0.836\u20130.883) but showed very low sensitivity (0.043\u20130.424) with near-perfect specificity (0.977\u20130.997). Claude Opus 4.5 showed moderate discrimination (AUC-ROC 0.698\u20130.761) with balanced sensitivity (0.396\u20130.876) and specificity (0.461\u20130.863). Gemini 2.5 Pro showed moderate discrimination (AUC-ROC 0.745\u20130.770) but favored sensitivity (0.673\u20130.973) at the expense of specificity (0.241\u20130.804). Sex-stratified analyses showed consistently higher AUC point estimates in male patients for cardiomegaly and pulmonary edema, but smaller and less directional differences for pleural effusion. Conclusions: Commercial MLLMs differ considerably in operating profiles, ranging from ultraconservative to aggressive detection, so that strong aggregate discrimination can mask sensitivity too low for reliable detection. None of the evaluated models are currently suitable for autonomous chest radiograph interpretation. Sex-stratified differences were modest but non-uniform, supporting subgroup-aware reporting rather than reliance on pooled metrics alone.<\/jats:p>","DOI":"10.3390\/diagnostics16132131","type":"journal-article","created":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T15:40:09Z","timestamp":1783525209000},"page":"2131","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Benchmarking Multimodal Large Language Models for Cardiopulmonary Findings on Chest Radiographs: Sex-Stratified Discrimination and Operating Characteristics"],"prefix":"10.3390","volume":"16","author":[{"given":"Matteo","family":"Haupt","sequence":"first","affiliation":[{"name":"Department of Diagnostic and Interventional Radiology, Carl von Ossietzky Universit\u00e4t Oldenburg, 26129 Oldenburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arne","family":"Bischoff","sequence":"additional","affiliation":[{"name":"Department of Diagnostic and Interventional Radiology, Carl von Ossietzky Universit\u00e4t Oldenburg, 26129 Oldenburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Myriam","family":"Atoubi","sequence":"additional","affiliation":[{"name":"Department of Diagnostic and Interventional Radiology, Carl von Ossietzky Universit\u00e4t Oldenburg, 26129 Oldenburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rohit Philip","family":"Thomas","sequence":"additional","affiliation":[{"name":"Department of Diagnostic and Interventional Radiology, Carl von Ossietzky Universit\u00e4t Oldenburg, 26129 Oldenburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin H.","family":"Maurer","sequence":"additional","affiliation":[{"name":"Department of Diagnostic and Interventional Radiology, Carl von Ossietzky Universit\u00e4t Oldenburg, 26129 Oldenburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"e232756","DOI":"10.1148\/radiol.232756","article-title":"Chatbots and Large Language Models in Radiology: A Practical Primer for Clinical and Research Applications","volume":"310","author":"Bhayana","year":"2024","journal-title":"Radiology"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e78041","DOI":"10.2196\/78041","article-title":"Trends and Trajectories in the Rise of Large Language Models in Radiology: Scoping Review","volume":"13","author":"Zaabi","year":"2025","journal-title":"JMIR Med. Inform."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"102888","DOI":"10.1016\/j.inffus.2024.102888","article-title":"A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine","volume":"117","author":"Xiao","year":"2025","journal-title":"Inf. Fusion"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3108","DOI":"10.1038\/s41467-025-58344-x","article-title":"A Clinically Accessible Small Multimodal Radiology Model and Evaluation Metric for Chest X-Ray Findings","volume":"16","author":"Huang","year":"2025","journal-title":"Nat. Commun."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e233270","DOI":"10.1148\/radiol.233270","article-title":"Evaluating GPT-V4 (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs","volume":"311","author":"Zhou","year":"2024","journal-title":"Radiology"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/s41747-025-00671-1","article-title":"Reliability of Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4 in Generating Standardized CMR Protocols","volume":"10","author":"Licu","year":"2026","journal-title":"Eur. Radiol. Exp."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wang, C.-H., Lin, P.-C., Shih, S.-L., Tsai, P.-S., and Huang, W.-H. (2026). Diagnostic Performance of ChatGPT-5 for Detecting Pediatric Pneumothorax on Chest Radiographs: A Multi-Prompt Evaluation. Diagnostics, 16.","DOI":"10.3390\/diagnostics16020232"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"18873","DOI":"10.1038\/s41598-026-58763-w","article-title":"Zero-Shot Multimodal Large Language Models Underperform a Domain-Trained CNN Baseline in Pediatric Wrist Fracture Detection","volume":"16","author":"Haupt","year":"2026","journal-title":"Sci. Rep."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Irvin, J., Ball, R.L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., and Langlotz, C.P. (2018). Deep Learning for Chest Radiograph Diagnosis: A Retrospective Comparison of the CheXNeXt Algorithm to Practicing Radiologists. PLoS Med., 15.","DOI":"10.1371\/journal.pmed.1002686"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e190228","DOI":"10.1148\/ryai.2021190228","article-title":"Deep Learning to Quantify Pulmonary Edema in Chest Radiographs","volume":"3","author":"Horng","year":"2021","journal-title":"Radiol. Artif. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Huang, Y.-Y., Lin, Y.-C., Tsai, S.-H., Chi, T.-K., Chen, T.-Y., Chung, S.-W., Li, K.-C., Tu, W.-C., R. Abu, P.A., and Chen, C.-C. (2025). Automated Computer-Assisted Diagnosis of Pleural Effusion in Chest X-Rays via Deep Learning. Diagnostics, 15.","DOI":"10.3390\/diagnostics15182322"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Yanar, E., Hardala\u00e7, F., and Ayturan, K. (2025). CELM: An Ensemble Deep Learning Model for Early Cardiomegaly Diagnosis in Chest Radiography. Diagnostics, 15.","DOI":"10.3390\/diagnostics15131602"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1038\/s42256-024-00965-w","article-title":"Exploring Scalable Medical Image Encoders beyond Text Supervision","volume":"7","author":"Sharma","year":"2025","journal-title":"Nat. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4374","DOI":"10.1007\/s00330-024-11339-6","article-title":"CXR-LLaVA: A Multimodal Large Language Model for Interpreting Chest X-Ray Images","volume":"35","author":"Lee","year":"2025","journal-title":"Eur. Radiol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"12592","DOI":"10.1073\/pnas.1919012117","article-title":"Gender Imbalance in Medical Imaging Datasets Produces Biased Classifiers for Computer-Aided Diagnosis","volume":"117","author":"Larrazabal","year":"2020","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2176","DOI":"10.1038\/s41591-021-01595-0","article-title":"Underdiagnosis Bias of Artificial Intelligence Algorithms Applied to Chest Radiographs in Under-Served Patient Populations","volume":"27","author":"Zhang","year":"2021","journal-title":"Nat. Med."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1007\/s10140-022-02019-3","article-title":"Deep Learning Prediction of Sex on Chest Radiographs: A Potential Contributor to Biased Algorithms","volume":"29","author":"Li","year":"2022","journal-title":"Emerg. Radiol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e230060","DOI":"10.1148\/ryai.230060","article-title":"Risk of Bias in Chest Radiography Deep Learning Foundation Models","volume":"5","author":"Glocker","year":"2023","journal-title":"Radiol. Artif. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e406","DOI":"10.1016\/S2589-7500(22)00063-2","article-title":"AI Recognition of Patient Race in Medical Imaging: A Modelling Study","volume":"4","author":"Gichoya","year":"2022","journal-title":"Lancet Digit. Health"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., and Shpanskaya, K. (2019). CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, PKP Publishing Services.","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Jain, S., Smit, A., Truong, S.Q., Nguyen, C.D., Huynh, M.-T., Jain, M., Young, V.A., Ng, A.Y., Lungren, M.P., and Rajpurkar, P. (2021, January 8). VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels. Proceedings of the Conference on Health, Inference, and Learning, Online.","DOI":"10.1145\/3450439.3451862"},{"key":"ref_22","unstructured":"Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., and Vedadi, E. (2024). Capabilities of Gemini Models in Medicine. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1959","DOI":"10.1007\/s00330-024-11035-5","article-title":"Assessing GPT-4 Multimodal Performance in Radiological Image Analysis","volume":"35","author":"Brin","year":"2025","journal-title":"Eur. Radiol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1111","DOI":"10.1007\/s00330-024-11115-6","article-title":"Revolution or Risk?\u2014Assessing the Potential and Challenges of GPT-4V in Radiologic Image Interpretation","volume":"35","author":"Huppertz","year":"2025","journal-title":"Eur. Radiol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1007\/s11604-024-01619-y","article-title":"Diagnostic Performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in \u201cDiagnosis Please\u201d Cases","volume":"42","author":"Sonoda","year":"2024","journal-title":"Jpn. J. Radiol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e240273","DOI":"10.1148\/radiol.240273","article-title":"Comparing Diagnostic Accuracy of Radiologists versus GPT-4V and Gemini Pro Vision Using Image Inputs from Diagnosis Please Cases","volume":"312","author":"Suh","year":"2024","journal-title":"Radiology"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chang, J., Lin, B.-R., Wang, T.-H., and Chen, C.-M. (2024). Deep Learning Model for Pleural Effusion Detection via Active Learning and Pseudo-Labeling: A Multisite Study. BMC Med. Imaging, 24.","DOI":"10.1186\/s12880-024-01260-1"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"214","DOI":"10.3348\/kjr.2025.1045","article-title":"Evaluating the Accuracy and Diagnostic Reasoning of Multimodal Large Language Models in Interpreting Neuroradiology Cases from RadioGraphics","volume":"27","author":"Suh","year":"2026","journal-title":"Korean J. Radiol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1093\/jamia\/ocae254","article-title":"Large Language Model Uncertainty Proxies: Discrimination and Calibration for Medical Diagnosis and Treatment","volume":"32","author":"Savage","year":"2025","journal-title":"J. Am. Med. Inf. Assoc."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e230342","DOI":"10.1148\/ryai.230342","article-title":"Improving Fairness of Automated Chest Radiograph Diagnosis by Contrastive Learning","volume":"6","author":"Lin","year":"2024","journal-title":"Radiol. Artif. Intell."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/13\/2131\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T15:55:45Z","timestamp":1783526145000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/13\/2131"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,7]]},"references-count":30,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["diagnostics16132131"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics16132131","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,7]]}}}