{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T08:46:04Z","timestamp":1786178764297,"version":"build-2736575974"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T00:00:00Z","timestamp":1786147200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T00:00:00Z","timestamp":1786147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002957","name":"Technische Universit\u00e4t Dresden","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002957","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    To externally validate the RSNA 2020 challenge 2\n                    <jats:sup>nd<\/jats:sup>\n                    -place deep-learning (DL) algorithm for detecting pulmonary embolism (PE) on computed tomography pulmonary angiography (CTPA) scans with additional focus on subsegmental-only PEs (SSPE). Between 2015 and 2022 1,038 CTPAs were retrospectively enrolled. An experienced radiologist (&gt;\u2009five years) labelled CTPAs for the presence, location (central, lobar and segmental, subsegmental-only) and side (right-sided, left-sided, bilateral) of PE. The DL model was tested for its ability to predict these labels by analyzing accuracy, sensitivity, specificity and area under the receiver operating characteristic curve (AUROC) using different model outcome probabilities for each analysis. Overall, 136 central, 375 peripheral PEs (303 lobar and segmental PEs, 72 SSPEs), and 527 patients without any PE were analyzed. The model correctly predicted the presence of any PE in 921\/1,038 patients (88.7%), yielding a sensitivity of 0.80, specificity of 0.97 and AUROC of 0.94. No central PE was missed, whereas 100\/375 (26.7%) peripheral PEs remained undetected. Using the corresponding model output probability, the central PE status was correctly identified in 999\/1,038 (96.2%) patients, while peripheral PE was correctly identified in 785\/1,038 (75.6%) patients. This corresponded to sensitivities of 0.94 and 0.77, specificities of 0.97 and 0.75, and AUROC values of 0.99 and 0.74, respectively. The model performed better in detecting right-sided compared to left-sided PEs (AUROC: right- vs. left-sided: 0.95 and 0.92,\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u20090.05). SSPE status was correctly predicted in 796\/1,038 (76.7%) patients, yielding a sensitivity of 0.94, a specificity of 0.22, and an AUROC of 0.48, respectively. The model demonstrated high performance in detecting any PE, performing best on central and slightly lower on lobar and segmental PE. Predicting the presence of SSPEs was difficult, likely due to the model not being explicitly trained for this subtype.\n                  <\/jats:p>","DOI":"10.1038\/s41598-026-65551-z","type":"journal-article","created":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T08:23:05Z","timestamp":1786177385000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["External validation of a top-ranked model from the RSNA pulmonary embolism detection challenge: assessment of generalizability"],"prefix":"10.1038","volume":"16","author":[{"given":"Patricia","family":"Hahlbohm","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sebastian","family":"Starke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Sch\u00f6n","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leo","family":"Misera","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marius","family":"Erdmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ralf-Thorsten","family":"Hoffmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Zwanenburg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jens-Peter","family":"K\u00fchn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steffen","family":"L\u00f6ck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,8,8]]},"reference":[{"key":"65551_CR1","doi-asserted-by":"publisher","first-page":"107602961985303","DOI":"10.1177\/1076029619853037","volume":"25","author":"B Rivera-Lebron","year":"2019","unstructured":"Rivera-Lebron, B. et al. Diagnosis, treatment and follow up of acute pulmonary embolism: Consensus practice from the PERT Consortium. Clin. Appl. Thromb. 25, 1076029619853037. https:\/\/doi.org\/10.1177\/1076029619853037 (2019).","journal-title":"Clin. Appl. Thromb."},{"key":"65551_CR2","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1093\/eurheartj\/ehz405","volume":"41","author":"SV Konstantinides","year":"2020","unstructured":"Konstantinides, S. V. et al. 2019 ESC Guidelines for the Diagnosis and Management of Acute Pulmonary Embolism Developed in Collaboration with the European Respiratory Society (ERS). Eur. Heart J. 41, 543\u2013603. https:\/\/doi.org\/10.1093\/eurheartj\/ehz405 (2020).","journal-title":"Eur. Heart J."},{"key":"65551_CR3","doi-asserted-by":"publisher","first-page":"111361","DOI":"10.1016\/j.ejrad.2024.111361","volume":"173","author":"E Langius-Wiffen","year":"2024","unstructured":"Langius-Wiffen, E. et al. External Validation of the RSNA 2020 Pulmonary Embolism Detection Challenge Winning Deep Learning Algorithm. Eur. J. Radiol. 173, 111361. https:\/\/doi.org\/10.1016\/j.ejrad.2024.111361 (2024).","journal-title":"Eur. J. Radiol."},{"key":"65551_CR4","doi-asserted-by":"publisher","first-page":"15814","DOI":"10.1038\/s41598-021-95249-3","volume":"11","author":"S Soffer","year":"2021","unstructured":"Soffer, S. et al. Deep Learning for Pulmonary Embolism Detection on Computed Tomography Pulmonary Angiogram: A Systematic Review and Meta-Analysis. Sci. Rep. 11, 15814. https:\/\/doi.org\/10.1038\/s41598-021-95249-3 (2021).","journal-title":"Sci. Rep."},{"key":"65551_CR5","doi-asserted-by":"publisher","first-page":"1443","DOI":"10.3390\/tomography9040115","volume":"9","author":"FA Eltawil","year":"2023","unstructured":"Eltawil, F. A., Atalla, M., Boulos, E., Amirabadi, A. & Tyrrell, P. N. Analyzing Barriers and Enablers for the Acceptance of Artificial Intelligence Innovations into Radiology Practice: A Scoping Review. Tomography 9, 1443\u20131455. https:\/\/doi.org\/10.3390\/tomography9040115 (2023).","journal-title":"Tomography"},{"key":"65551_CR6","unstructured":"RSNA Pulmonary Embolism Detection Challenge (2020). https:\/\/www.rsna.org\/rsnai\/ai-image-challenge\/rsna-pe-detection-challenge-2020 (accessed on 5 January 2025)."},{"key":"65551_CR7","doi-asserted-by":"publisher","first-page":"e200254","DOI":"10.1148\/ryai.2021200254","volume":"3","author":"E Colak","year":"2021","unstructured":"Colak, E. et al. The RSNA Pulmonary Embolism CT Dataset. Radiol. Artif. Intell. 3, e200254. https:\/\/doi.org\/10.1148\/ryai.2021200254 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"65551_CR8","doi-asserted-by":"publisher","first-page":"e210064","DOI":"10.1148\/ryai.210064","volume":"4","author":"AC Yu","year":"2022","unstructured":"Yu, A. C., Mohajer, B. & Eng, J. External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review. Radiol. Artif. Intell. 4, e210064. https:\/\/doi.org\/10.1148\/ryai.210064 (2022).","journal-title":"Radiol. Artif. Intell."},{"key":"65551_CR9","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","volume":"3","author":"M Roberts","year":"2021","unstructured":"Roberts, M. et al. Common Pitfalls and Recommendations for Using Machine Learning to Detect and Prognosticate for COVID-19 Using Chest Radiographs and CT Scans. Nat. Mach. Intell. 3, 199\u2013217. https:\/\/doi.org\/10.1038\/s42256-021-00307-0 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"65551_CR10","doi-asserted-by":"publisher","first-page":"e232873","DOI":"10.1148\/radiol.232873","volume":"310","author":"HP Forman","year":"2024","unstructured":"Forman, H. P. & Bhalla, S. Subsegmental Pulmonary Emboli and Chronic Pulmonary Emboli Should Not Be Ignored. Radiology 310, e232873. https:\/\/doi.org\/10.1148\/radiol.232873 (2024).","journal-title":"Radiology"},{"key":"65551_CR11","doi-asserted-by":"publisher","first-page":"e210068","DOI":"10.1148\/ryai.2021210068","volume":"3","author":"I Pan","year":"2021","unstructured":"Pan, I. Deep Learning for Pulmonary Embolism Detection: Tackling the RSNA 2020 AI Challenge. Radiol. Artif. Intell. 3, e210068. https:\/\/doi.org\/10.1148\/ryai.2021210068 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"65551_CR12","doi-asserted-by":"publisher","first-page":"461","DOI":"10.1177\/00033197221111862","volume":"74","author":"J Leonhardi","year":"2023","unstructured":"Leonhardi, J. et al. Computed Tomography Embolus Texture Analysis as a Prognostic Marker of Acute Pulmonary Embolism. Angiology 74, 461\u2013471. https:\/\/doi.org\/10.1177\/00033197221111862 (2023).","journal-title":"Angiology"},{"key":"65551_CR13","unstructured":"Pan, I. I-Pan\/Kaggle-Rsna-Pe 2025. https:\/\/github.com\/i-pan\/kaggle-rsna-pe, commit. hash 81ca9ee ( (2025). accessed on 14 July 2024)."},{"key":"65551_CR14","unstructured":"Efron, B. & Hastie, T. Bootstrap confidence intervals. In Computer Age Statistical Inference: Algorithms, Evidence, and Data Science 181\u2013207 (Cambridge University Press, 2016). ISBN 978-1-107-14989-2."},{"key":"65551_CR15","doi-asserted-by":"publisher","unstructured":"Huang, S. C. et al. PENet\u2014a Scalable Deep-Learning Model for Automated Diagnosis of Pulmonary Embolism Using Volumetric CT Imaging. Npj Digit. Med. 3 https:\/\/doi.org\/10.1038\/s41746-020-0266-y (2020).","DOI":"10.1038\/s41746-020-0266-y"},{"key":"65551_CR16","doi-asserted-by":"publisher","first-page":"6545","DOI":"10.1007\/s00330-020-06998-0","volume":"30","author":"T Weikert","year":"2020","unstructured":"Weikert, T. et al. Automated Detection of Pulmonary Embolism in CT Pulmonary Angiograms Using an AI-Powered Algorithm. Eur. Radiol. 30, 6545\u20136553. https:\/\/doi.org\/10.1007\/s00330-020-06998-0 (2020).","journal-title":"Eur. Radiol."},{"key":"65551_CR17","doi-asserted-by":"publisher","first-page":"109816","DOI":"10.1016\/j.ejrad.2021.109816","volume":"141","author":"L Schmuelling","year":"2021","unstructured":"Schmuelling, L. et al. Deep Learning-Based Automated Detection of Pulmonary Embolism on CT Pulmonary Angiograms: No Significant Effects on Report Communication Times and Patient Turnaround in the Emergency Department Nine Months after Technical Implementation. Eur. J. Radiol. 141, 109816. https:\/\/doi.org\/10.1016\/j.ejrad.2021.109816 (2021).","journal-title":"Eur. J. Radiol."},{"key":"65551_CR18","doi-asserted-by":"publisher","first-page":"e230702","DOI":"10.1148\/radiol.230702","volume":"309","author":"SA Rothenberg","year":"2023","unstructured":"Rothenberg, S. A. et al. Prospective Evaluation of AI Triage of Pulmonary Emboli on CT Pulmonary Angiograms. Radiology 309, e230702. https:\/\/doi.org\/10.1148\/radiol.230702 (2023).","journal-title":"Radiology"},{"key":"65551_CR19","doi-asserted-by":"publisher","first-page":"5831","DOI":"10.1007\/s00330-022-08645-2","volume":"32","author":"AB Cheikh","year":"2022","unstructured":"Cheikh, A. B. et al. How Artificial Intelligence Improves Radiological Interpretation in Suspected Pulmonary Embolism. Eur. Radiol. 32, 5831\u20135842. https:\/\/doi.org\/10.1007\/s00330-022-08645-2 (2022).","journal-title":"Eur. Radiol."},{"key":"65551_CR20","doi-asserted-by":"publisher","first-page":"110245","DOI":"10.1016\/j.clinimag.2024.110245","volume":"113","author":"A Ayobi","year":"2024","unstructured":"Ayobi, A. et al. Performance and Clinical Utility of an Artificial Intelligence-Enabled Tool for Pulmonary Embolism Detection. Clin. Imaging. 113, 110245. https:\/\/doi.org\/10.1016\/j.clinimag.2024.110245 (2024).","journal-title":"Clin. Imaging"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-65551-z","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-65551-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-65551-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T08:42:08Z","timestamp":1786178528000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-65551-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,8]]},"references-count":20,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["65551"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-65551-z","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,8]]},"assertion":[{"value":"15 February 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 August 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Ethics"}}],"article-number":"24545"}}