{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T22:37:57Z","timestamp":1787956677167,"version":"build-2784847793"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,9,26]],"date-time":"2023-09-26T00:00:00Z","timestamp":1695686400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,26]],"date-time":"2023-09-26T00:00:00Z","timestamp":1695686400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["517316550"],"award-info":[{"award-number":["517316550"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We determined if a convolutional neural network (CNN) deep learning model can accurately segment acute ischemic changes on non-contrast CT compared to neuroradiologists. Non-contrast CT (NCCT) examinations from 232 acute ischemic stroke patients who were enrolled in the DEFUSE 3 trial were included in this study. Three experienced neuroradiologists independently segmented hypodensity that reflected the ischemic core on each scan. The neuroradiologist with the most experience (expert A) served as the ground truth for deep learning model training. Two additional neuroradiologists\u2019 (experts B and C) segmentations were used for data testing. The 232 studies were randomly split into training and test sets. The training set was further randomly divided into 5 folds with training and validation sets. A 3-dimensional CNN architecture was trained and optimized to predict the segmentations of expert A from NCCT. The performance of the model was assessed using a set of volume, overlap, and distance metrics using non-inferiority thresholds of 20%, 3 ml, and 3 mm, respectively. The optimized model trained on expert A was compared to test experts B and C. We used a one-sided Wilcoxon signed-rank test to test for the non-inferiority of the model-expert compared to the inter-expert agreement. The final model performance for the ischemic core segmentation task reached a performance of 0.46 \u00b1 0.09 Surface Dice at Tolerance 5mm and 0.47 \u00b1 0.13 Dice when trained on expert A. Compared to the two test neuroradiologists the model-expert agreement was non-inferior to the inter-expert agreement, <jats:inline-formula><jats:alternatives><jats:tex-math>$$p &lt; 0.05$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mi>p<\/mml:mi>\n                    <mml:mo>&lt;<\/mml:mo>\n                    <mml:mn>0.05<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>. The before, CNN accurately delineates the hypodense ischemic core on NCCT in acute ischemic stroke patients with an accuracy comparable to neuroradiologists.<\/jats:p>","DOI":"10.1038\/s41598-023-42961-x","type":"journal-article","created":{"date-parts":[[2023,9,26]],"date-time":"2023-09-26T22:01:58Z","timestamp":1695765718000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Non-inferiority of deep learning ischemic stroke segmentation on non-contrast CT within 16-hours compared to expert neuroradiologists"],"prefix":"10.1038","volume":"13","author":[{"given":"Sophie","family":"Ostmeier","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian","family":"Axelrod","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin F. J.","family":"Verhaaren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Soren","family":"Christensen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abdelkader","family":"Mahammedi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yongkai","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Pulli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Li-Jia","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Greg","family":"Zaharchuk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeremy J.","family":"Heit","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,9,26]]},"reference":[{"key":"42961_CR1","doi-asserted-by":"publisher","first-page":"795","DOI":"10.1016\/s1474-4422(21)00252-0","volume":"20","author":"VL Feigin","year":"2021","unstructured":"Feigin, V. L. et al. Global regional and national burden of stroke and its risk factors, 1990\u20132019: A systematic analysis for the global burden of disease study 2019. Lancet Neurol. 20, 795\u2013820. https:\/\/doi.org\/10.1016\/s1474-4422(21)00252-0 (2021).","journal-title":"Lancet Neurol."},{"key":"42961_CR2","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1136\/neurintsurg-2018-014239","volume":"11","author":"N Lakomkin","year":"2019","unstructured":"Lakomkin, N. et al. Prevalence of large vessel occlusion in patients presenting with acute ischemic stroke: A 10-year systematic review of the literature. J. NeuroInterv. Surg. 11, 241\u2013245. https:\/\/doi.org\/10.1136\/neurintsurg-2018-014239 (2019).","journal-title":"J. NeuroInterv. Surg."},{"key":"42961_CR3","doi-asserted-by":"publisher","first-page":"657","DOI":"10.3389\/fneur.2017.00657","volume":"8","author":"BC Campbell","year":"2017","unstructured":"Campbell, B. C. et al. Endovascular thrombectomy for ischemic stroke increases disability-free survival, quality of life, and life expectancy and reduces cost. Front. Neurol. 8, 657 (2017).","journal-title":"Front. Neurol."},{"key":"42961_CR4","doi-asserted-by":"publisher","first-page":"e46","DOI":"10.1161\/STR.0000000000000158","volume":"49","author":"WJ Powers","year":"2018","unstructured":"Powers, W. J. et al. 2018 guidelines for the early management of patients with acute ischemic stroke: A guideline for healthcare professionals from the American Heart Association and American Stroke Association. Stroke 49, e46\u2013e99 (2018).","journal-title":"Stroke"},{"key":"42961_CR5","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1056\/NEJMoa1713973","volume":"378","author":"GW Albers","year":"2018","unstructured":"Albers, G. W. et al. Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. N. Engl. J. Med. 378, 708\u2013718. https:\/\/doi.org\/10.1056\/NEJMoa1713973 (2018).","journal-title":"N. Engl. J. Med."},{"key":"42961_CR6","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1056\/NEJMoa1706442","volume":"378","author":"RG Nogueira","year":"2017","unstructured":"Nogueira, R. G. et al. Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct. N. Engl. J. Med. 378, 11\u201321. https:\/\/doi.org\/10.1056\/NEJMoa1706442 (2017).","journal-title":"N. Engl. J. Med."},{"key":"42961_CR7","doi-asserted-by":"publisher","first-page":"e006989","DOI":"10.1161\/CIRCOUTCOMES.120.006989","volume":"14","author":"Y Kim","year":"2021","unstructured":"Kim, Y. et al. Utilization and availability of advanced imaging in patients with acute ischemic stroke. Circ. Cardiovasc. Qual. Outcomes 14, e006989. https:\/\/doi.org\/10.1161\/CIRCOUTCOMES.120.006989 (2021).","journal-title":"Circ. Cardiovasc. Qual. Outcomes"},{"key":"42961_CR8","doi-asserted-by":"publisher","DOI":"10.1177\/08465371211028823","author":"R McDonough","year":"2021","unstructured":"McDonough, R., Ospel, J. & Goyal, M. State of the art stroke imaging: A current perspective. Can. Assoc. Radiol. J.https:\/\/doi.org\/10.1177\/08465371211028823 (2021).","journal-title":"Can. Assoc. Radiol. J."},{"key":"42961_CR9","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1001\/jamaneurol.2020.0113","volume":"77","author":"AW Alexandrov","year":"2020","unstructured":"Alexandrov, A. W. & Fassbender, K. Triage based on preclinical scores-low-cost strategy for accelerating time to thrombectomy. JAMA Neurol. 77, 681\u2013682. https:\/\/doi.org\/10.1001\/jamaneurol.2020.0113 (2020).","journal-title":"JAMA Neurol."},{"key":"42961_CR10","doi-asserted-by":"publisher","first-page":"1017","DOI":"10.1001\/jama.1995.03530130023023","volume":"274","author":"W Hacke","year":"1995","unstructured":"Hacke, W. et al. Intravenous thrombolysis with recombinant tissue plasminogen activator for acute hemispheric stroke: The European Cooperative Acute Stroke Study (ECASS). JAMA 274, 1017\u20131025 (1995).","journal-title":"JAMA"},{"key":"42961_CR11","doi-asserted-by":"publisher","first-page":"2296","DOI":"10.1056\/NEJMoa1503780","volume":"372","author":"TG Jovin","year":"2015","unstructured":"Jovin, T. G. et al. Thrombectomy within 8 hours after symptom onset in ischemic stroke. N. Engl. J. Med. 372, 2296\u20132306 (2015).","journal-title":"N. Engl. J. Med."},{"key":"42961_CR12","doi-asserted-by":"publisher","first-page":"1723","DOI":"10.1016\/S0140-6736(16)00163-X","volume":"387","author":"M Goyal","year":"2016","unstructured":"Goyal, M. et al. Endovascular thrombectomy after large-vessel ischaemic stroke: A meta-analysis of individual patient data from five randomised trials. Lancet 387, 1723\u20131731 (2016).","journal-title":"Lancet"},{"key":"42961_CR13","doi-asserted-by":"publisher","first-page":"1272","DOI":"10.1056\/NEJMoa2213379","volume":"388","author":"X Huo","year":"2023","unstructured":"Huo, X. et al. Trial of endovascular therapy for acute ischemic stroke with large infarct. N. Engl. J. Med. 388, 1272\u20131283 (2023).","journal-title":"N. Engl. J. Med."},{"key":"42961_CR14","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa2214403","author":"A Sarraj","year":"2023","unstructured":"Sarraj, A. et al. Trial of endovascular thrombectomy for large ischemic strokes. N. Engl. J. Med.https:\/\/doi.org\/10.1056\/NEJMoa2214403 (2023).","journal-title":"N. Engl. J. Med."},{"key":"42961_CR15","doi-asserted-by":"publisher","first-page":"1303","DOI":"10.1056\/NEJMoa2118191","volume":"386","author":"S Yoshimura","year":"2022","unstructured":"Yoshimura, S. et al. Endovascular therapy for acute stroke with a large ischemic region. N. Engl. J. Med. 386, 1303\u20131313 (2022).","journal-title":"N. Engl. J. Med."},{"key":"42961_CR16","doi-asserted-by":"publisher","first-page":"1670","DOI":"10.1016\/S0140-6736(00)02237-6","volume":"355","author":"PA Barber","year":"2000","unstructured":"Barber, P. A., Demchuk, A. M., Zhang, J. & Buchan, A. M. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. Lancet 355, 1670\u20131674. https:\/\/doi.org\/10.1016\/S0140-6736(00)02237-6 (2000).","journal-title":"Lancet"},{"key":"42961_CR17","doi-asserted-by":"publisher","DOI":"10.3389\/fneur.2016.00245","author":"J Schroeder","year":"2017","unstructured":"Schroeder, J. & Thomalla, G. A critical review of Alberta stroke program early CT score for evaluation of acute stroke imaging. Front. Neurol.https:\/\/doi.org\/10.3389\/fneur.2016.00245 (2017).","journal-title":"Front. Neurol."},{"key":"42961_CR18","doi-asserted-by":"publisher","first-page":"2328","DOI":"10.1161\/STROKEAHA.120.030696","volume":"52","author":"A Hakim","year":"2021","unstructured":"Hakim, A. et al. Predicting infarct core from computed tomography perfusion in acute ischemia with machine learning: Lessons from the ISLES challenge. Stroke 52, 2328\u20132337 (2021).","journal-title":"Stroke"},{"key":"42961_CR19","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","volume":"18","author":"F Isensee","year":"2021","unstructured":"Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J. & Maier-Hein, K. H. nnu-net: A self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203\u2013211. https:\/\/doi.org\/10.1038\/s41592-020-01008-z (2021).","journal-title":"Nat. Methods"},{"key":"42961_CR20","doi-asserted-by":"publisher","first-page":"105033","DOI":"10.1016\/j.compbiomed.2021.105033","volume":"141","author":"H El-Hariri","year":"2022","unstructured":"El-Hariri, H. et al. Evaluating nnU-Net for early ischemic change segmentation on non-contrast computed tomography in patients with acute ischemic stroke. Comput. Biol. Med. 141, 105033 (2022).","journal-title":"Comput. Biol. Med."},{"key":"42961_CR21","doi-asserted-by":"publisher","first-page":"3147","DOI":"10.1161\/STROKEAHA.120.030620","volume":"51","author":"M Goyal","year":"2020","unstructured":"Goyal, M. et al. Challenging the ischemic core concept in acute ischemic stroke imaging. Stroke 51, 3147\u20133155 (2020).","journal-title":"Stroke"},{"key":"42961_CR22","doi-asserted-by":"publisher","first-page":"2557","DOI":"10.1109\/TIP.2016.2544703","volume":"25","author":"TA Lampert","year":"2016","unstructured":"Lampert, T. A., Stumpf, A. & Gan\u00e7arski, P. An empirical study into annotator agreement, ground truth estimation, and algorithm evaluation. IEEE Trans. Image Process. 25, 2557\u20132572. https:\/\/doi.org\/10.1109\/TIP.2016.2544703 (2016).","journal-title":"IEEE Trans. Image Process."},{"key":"42961_CR23","doi-asserted-by":"publisher","first-page":"2361","DOI":"10.1098\/rsta.2008.0040","volume":"366","author":"SK Warfield","year":"2008","unstructured":"Warfield, S. K., Zou, K. H. & Wells, W. M. Validation of image segmentation by estimating rater bias and variance. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 366, 2361\u20132375 (2008).","journal-title":"Philos. Trans. R. Soc. A Math. Phys. Eng. Sci."},{"key":"42961_CR24","doi-asserted-by":"publisher","first-page":"e200231","DOI":"10.1148\/ryai.200231","volume":"4","author":"L Hirsch","year":"2021","unstructured":"Hirsch, L. et al. Radiologist-level performance by using deep learning for segmentation of breast cancers on MRI scans. Radiol. Artif. Intell. 4, e200231 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"42961_CR25","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1038\/s43856-021-00062-8","volume":"1","author":"C-F Liu","year":"2021","unstructured":"Liu, C.-F. et al. Deep learning-based detection and segmentation of diffusion abnormalities in acute ischemic stroke. Commun. Med. 1, 61. https:\/\/doi.org\/10.1038\/s43856-021-00062-8 (2021).","journal-title":"Commun. Med."},{"key":"42961_CR26","doi-asserted-by":"publisher","first-page":"101984","DOI":"10.1016\/j.media.2021.101984","volume":"70","author":"H Kuang","year":"2021","unstructured":"Kuang, H., Menon, B. K., Sohn, S. I. L. & Qiu, W. EIS-Net: Segmenting early infarct and scoring ASPECTS simultaneously on non-contrast CT of patients with acute ischemic stroke. Med. Image Anal. 70, 101984. https:\/\/doi.org\/10.1016\/j.media.2021.101984 (2021).","journal-title":"Med. Image Anal."},{"key":"42961_CR27","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.media.2016.05.004","volume":"35","author":"M Havaei","year":"2017","unstructured":"Havaei, M. et al. Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18\u201331 (2017).","journal-title":"Med. Image Anal."},{"key":"42961_CR28","doi-asserted-by":"publisher","first-page":"1780","DOI":"10.1177\/0271678X15610586","volume":"36","author":"CW Cereda","year":"2016","unstructured":"Cereda, C. W. et al. A benchmarking tool to evaluate computer tomography perfusion infarct core predictions against DWI standard. J. Cereb. Blood Flow Metab. 36, 1780\u20131789 (2016).","journal-title":"J. Cereb. Blood Flow Metab."},{"key":"42961_CR29","doi-asserted-by":"publisher","first-page":"1557","DOI":"10.1212\/WNL.54.8.1557","volume":"54","author":"MG Lansberg","year":"2000","unstructured":"Lansberg, M. G., Albers, G. W., Beaulieu, C. & Marks, M. P. Comparison of diffusion-weighted MRI and CT in acute stroke. Neurology 54, 1557\u20131561 (2000).","journal-title":"Neurology"},{"key":"42961_CR30","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2021200127","author":"S Christensen","year":"2021","unstructured":"Christensen, S. et al. Optimizing deep learning algorithms for segmentation of acute infarcts on non-contrast material-enhanced CT scans of the brain using simulated lesions. Radiol. Artif. Intell.https:\/\/doi.org\/10.1148\/ryai.2021200127 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"42961_CR31","doi-asserted-by":"publisher","first-page":"638","DOI":"10.1148\/radiol.2020191193","volume":"294","author":"W Qiu","year":"2020","unstructured":"Qiu, W. et al. Machine learning for detecting early infarction in acute stroke with non-contrast-enhanced CT. Radiology 294, 638\u2013644 (2020).","journal-title":"Radiology"},{"key":"42961_CR32","doi-asserted-by":"publisher","first-page":"S47","DOI":"10.1212\/WNL.51.3_Suppl_3.S47","volume":"51","author":"GW Albers","year":"1998","unstructured":"Albers, G. W. Diffusion-weighted MRI for evaluation of acute stroke. Neurology 51, S47\u2013S49 (1998).","journal-title":"Neurology"},{"key":"42961_CR33","doi-asserted-by":"crossref","unstructured":"Almandoz, J. E.D., Pomerantz, S.R., Gonz\u00e1lez, R.G. & Lev, M.H. Imaging of acute ischemic stroke: Unenhanced computed tomography. Acute Ischemic Stroke, 43\u201356 (2011).","DOI":"10.1007\/978-3-642-12751-9_3"},{"key":"42961_CR34","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Doll\u00e1r, P. Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, 2980\u20132988 (2017).","DOI":"10.1109\/ICCV.2017.324"},{"key":"42961_CR35","unstructured":"Nikolov, S. et\u00a0al. Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy. Preprint at http:\/\/arxiv.org\/abs\/1809.04430 (2018)."},{"key":"42961_CR36","unstructured":"Maier-Hein, L. et\u00a0al. Metrics reloaded: Pitfalls and recommendations for image analysis validation. Preprint at http:\/\/arxiv.org\/abs\/2206.01653 (2022)."},{"key":"42961_CR37","doi-asserted-by":"publisher","first-page":"102927","DOI":"10.1016\/j.media.2023.102927","volume":"90","author":"S Ostmeier","year":"2023","unstructured":"Ostmeier, S. et al. USE-evaluator: Performance metrics for medical image segmentation models supervised by uncertain, small or empty reference annotations in neuroimaging. Med. Image Anal. 90, 102927 (2023).","journal-title":"Med. Image Anal."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-42961-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-42961-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-42961-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,19]],"date-time":"2023-11-19T19:04:11Z","timestamp":1700420651000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-42961-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,26]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["42961"],"URL":"https:\/\/doi.org\/10.1038\/s41598-023-42961-x","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,26]]},"assertion":[{"value":"18 April 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 September 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Sophie Ostmeier: none Brian Axelrod: none Benjamin F.J. Verhaaren: none Soren Christensen: none Abdelkader Mahammedi: none Yongkai Liu: none Benjamin Pulli: none Li-Jia Li: none Greg Zaharchuk: co-founder, equity of Subtle Medical, funding support GE Healthcare, consultant Biogen Jeremy J. Heit: Consultant for Medtronic and MicroVention, Member of the medical and scientific advisory board for iSchemaView.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"16153"}}