{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T15:41:47Z","timestamp":1777736507951,"version":"3.51.4"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2018,4,4]],"date-time":"2018-04-04T00:00:00Z","timestamp":1522800000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2018,4,4]],"date-time":"2018-04-04T00:00:00Z","timestamp":1522800000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digital Med"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Intracranial hemorrhage (ICH) requires prompt diagnosis to optimize patient outcomes. We hypothesized that machine learning algorithms could automatically analyze computed tomography (CT) of the head, prioritize radiology worklists and reduce time to diagnosis of ICH. 46,583 head CTs (~2 million images) acquired from 2007\u20132017 were collected from several facilities across Geisinger. A deep convolutional neural network was trained on 37,074 studies and subsequently evaluated on 9499 unseen studies. The predictive model was implemented prospectively for 3 months to re-prioritize \u201croutine\u201d head CT studies as \u201cstat\u201d on realtime radiology worklists if an ICH was detected. Time to diagnosis was compared between the re-prioritized \u201cstat\u201d and \u201croutine\u201d studies. A neuroradiologist blinded to the study reviewed false positive studies to determine whether the dictating radiologist overlooked ICH. The model achieved an area under the ROC curve of 0.846 (0.837\u20130.856). During implementation, 94 of 347 \u201croutine\u201d studies were re-prioritized to \u201cstat\u201d, and 60\/94 had ICH identified by the radiologist. Five new cases of ICH were identified, and median time to diagnosis was significantly reduced (<jats:italic>p<\/jats:italic>\u2009&lt;\u20090.0001) from 512 to 19\u2009min. In particular, one outpatient with vague symptoms on anti-coagulation was found to have an ICH which was treated promptly with reversal of anticoagulation, resulting in a good clinical outcome. Of the 34 false positives, the blinded over-reader identified four probable ICH cases overlooked in original interpretation. In conclusion, an artificial intelligence algorithm can prioritize radiology worklists to reduce time to diagnosis of new outpatient ICH by 96% and may also identify subtle ICH overlooked by radiologists. This demonstrates the positive impact of advanced machine learning in radiology workflow optimization.<\/jats:p>","DOI":"10.1038\/s41746-017-0015-z","type":"journal-article","created":{"date-parts":[[2018,1,30]],"date-time":"2018-01-30T11:11:31Z","timestamp":1517310691000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":380,"title":["Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration"],"prefix":"10.1038","volume":"1","author":[{"given":"Mohammad R.","family":"Arbabshirani","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6231-9442","authenticated-orcid":false,"given":"Brandon K.","family":"Fornwalt","sequence":"additional","affiliation":[]},{"given":"Gino J.","family":"Mongelluzzo","sequence":"additional","affiliation":[]},{"given":"Jonathan D.","family":"Suever","sequence":"additional","affiliation":[]},{"given":"Brandon D.","family":"Geise","sequence":"additional","affiliation":[]},{"given":"Aalpen A.","family":"Patel","sequence":"additional","affiliation":[]},{"given":"Gregory J.","family":"Moore","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2018,4,4]]},"reference":[{"key":"15_CR1","doi-asserted-by":"publisher","first-page":"1632","DOI":"10.1016\/S0140-6736(09)60371-8","volume":"373","author":"AI Qureshi","year":"2009","unstructured":"Qureshi, A. I., Mendelow, A. D. & Hanley, D. F. Intracerebral haemorrhage. Lancet 373, 1632\u20131644 (2009).","journal-title":"Lancet"},{"key":"15_CR2","doi-asserted-by":"publisher","first-page":"2108","DOI":"10.1161\/STR.0b013e3181ec611b","volume":"41","author":"LB Morgenstern","year":"2010","unstructured":"Morgenstern, L. B. et al. Guidelines for the management of spontaneous intracerebral hemorrhage. Stroke 41, 2108\u20132129 (2010).","journal-title":"Stroke"},{"key":"15_CR3","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1016\/S1474-4422(09)70340-0","volume":"9","author":"CJJ van Asch","year":"2010","unstructured":"van Asch, C. J. J. et al. Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis. Lancet Neurol. 9, 167\u2013176 (2010).","journal-title":"Lancet Neurol."},{"key":"15_CR4","doi-asserted-by":"publisher","first-page":"2180","DOI":"10.1212\/WNL.0000000000000519","volume":"82","author":"DB Zahuranec","year":"2014","unstructured":"Zahuranec, D. B. et al. Intracerebral hemorrhage mortality is not changing despite declining incidence. Neurology 82, 2180\u20132186 (2014).","journal-title":"Neurology"},{"key":"15_CR5","doi-asserted-by":"publisher","first-page":"1750","DOI":"10.1212\/01.WNL.0000035748.91128.C2","volume":"59","author":"SA Mayer","year":"2002","unstructured":"Mayer, S. A. et al. Global and domain-specific cognitive impairment and outcome after subarachnoid hemorrhage. Neurology 59, 1750\u20131758 (2002).","journal-title":"Neurology"},{"key":"15_CR6","doi-asserted-by":"publisher","first-page":"658","DOI":"10.1212\/WNL.55.5.658","volume":"55","author":"ML Hackett","year":"2000","unstructured":"Hackett, M. L. & Anderson, C. S. on Subarachnoid Hemorrhage Study (ACROSS) Group, A. C. R. & others. Health outcomes 1 year after subarachnoid hemorrhage an international population-based study. Neurology 55, 658\u2013662 (2000).","journal-title":"Neurology"},{"key":"15_CR7","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1016\/S1474-4422(03)00266-7","volume":"2","author":"VL Feigin","year":"2003","unstructured":"Feigin, V. L., Lawes, C. M. M., Bennett, D. A. & Anderson, C. S. Stroke epidemiology: a review of population-based studies of incidence, prevalence, and case-fatality in the late 20th century. Lancet Neurol. 2, 43\u201353 (2003).","journal-title":"Lancet Neurol."},{"key":"15_CR8","doi-asserted-by":"publisher","first-page":"2180","DOI":"10.1161\/STROKEAHA.106.467506","volume":"38","author":"AI Qureshi","year":"2007","unstructured":"Qureshi, A. I. et al. Changes in cost and outcome among US patients with stroke hospitalized in 1990 to 1991 and those hospitalized in 2000 to 2001. Stroke 38, 2180\u20132184 (2007).","journal-title":"Stroke"},{"key":"15_CR9","doi-asserted-by":"publisher","first-page":"1419","DOI":"10.1213\/ANE.0b013e3181d568c8","volume":"110","author":"J Elliott","year":"2010","unstructured":"Elliott, J. & Smith, M. The acute management of intracerebral hemorrhage: a clinical review. Anesth. Analg. 110, 1419\u20131427 (2010).","journal-title":"Anesth. Analg."},{"key":"15_CR10","doi-asserted-by":"publisher","first-page":"518","DOI":"10.3171\/jns.1990.73.4.0518","volume":"73","author":"K Fujitsu","year":"1990","unstructured":"Fujitsu, K. et al. Indications for surgical treatment of putaminal hemorrhage: comparative study based on serial CT and time-course analysis. J. Neurosurg. 73, 518\u2013525 (1990).","journal-title":"J. Neurosurg."},{"key":"15_CR11","doi-asserted-by":"publisher","first-page":"897","DOI":"10.7326\/0003-4819-120-11-199406010-00001","volume":"120","author":"EM Hylek","year":"1994","unstructured":"Hylek, E. M. & Singer, D. E. Risk factors for intracranial hemorrhage in outpatients taking warfarin. Ann. Intern. Med. 120, 897\u2013902 (1994).","journal-title":"Ann. Intern. Med."},{"key":"15_CR12","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"15_CR13","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1016\/S0140-6736(17)31764-6","volume":"390","author":"G Quer","year":"2017","unstructured":"Quer, G., Muse, E. D., Nikzad, N., Topol, E. J. & Steinhubl, S. R. Augmenting diagnostic vision with AI. Lancet 390, 221 (2017).","journal-title":"Lancet"},{"key":"15_CR14","doi-asserted-by":"publisher","first-page":"2402","DOI":"10.1001\/jama.2016.17216","volume":"316","author":"V Gulshan","year":"2016","unstructured":"Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402\u20132410 (2016).","journal-title":"JAMA"},{"key":"15_CR15","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115\u2013118 (2017).","journal-title":"Nature"},{"key":"15_CR16","doi-asserted-by":"publisher","first-page":"1160","DOI":"10.1109\/TMI.2016.2536809","volume":"35","author":"AAA Setio","year":"2016","unstructured":"Setio, A. A. A. et al. Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks. IEEE Trans. Med. Imaging 35, 1160\u20131169 (2016).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"15_CR17","doi-asserted-by":"publisher","first-page":"1182","DOI":"10.1109\/TMI.2016.2528129","volume":"35","author":"Q Dou","year":"2016","unstructured":"Dou, Q. et al. Automatic detection of cerebral microbleeds from MR images via 3D convolutional neural networks. IEEE Trans. Med. Imaging 35, 1182\u20131195 (2016).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"15_CR18","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1016\/j.compmedimag.2007.02.002","volume":"31","author":"K Doi","year":"2007","unstructured":"Doi, K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput. Med. Imaging Graph. 31, 198\u2013211 (2007).","journal-title":"Comput. Med. Imaging Graph."},{"key":"15_CR19","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1148\/89.2.211","volume":"89","author":"F Winsberg","year":"1967","unstructured":"Winsberg, F., Elkin, M., Macy, J. Jr, Bordaz, V. & Weymouth, W. Detection of radiographic abnormalities in mammograms by means of optical scanning and computer analysis 1. Radiology 89, 211\u2013215 (1967).","journal-title":"Radiology"},{"key":"15_CR20","doi-asserted-by":"publisher","first-page":"1651","DOI":"10.2214\/ajr.171.6.9843307","volume":"171","author":"L Monnier-Cholley","year":"1998","unstructured":"Monnier-Cholley, L. et al. Computer-aided diagnosis for detection of interstitial opacities on chest radiographs. Ajr. Am. J. Roentgenol. 171, 1651\u20131656 (1998).","journal-title":"Ajr. Am. J. Roentgenol."},{"key":"15_CR21","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1148\/radiol.2222010506","volume":"222","author":"H Yoshida","year":"2002","unstructured":"Yoshida, H., Masutani, Y., Maceneaney, P., Rubin, D. T. & Dachman, A. H. Computerized detection of colonic polyps at ct colonography on the basis of volumetric features: pilot study 1. Radiology 222, 327\u2013336 (2002).","journal-title":"Radiology"},{"key":"15_CR22","doi-asserted-by":"crossref","unstructured":"Arbabshirani, M. R., Plis, S., Sui, J. & Calhoun, V. D. Single subject prediction of brain disorders in neuroimaging: promises and pitfalls. Neuroimage 145, 137\u2013165 (2016).","DOI":"10.1016\/j.neuroimage.2016.02.079"},{"key":"15_CR23","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1148\/radiol.2203001282","volume":"220","author":"TW Freer","year":"2001","unstructured":"Freer, T. W. & Ulissey, M. J. Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center 1. Radiology 220, 781\u2013786 (2001).","journal-title":"Radiology"},{"key":"15_CR24","doi-asserted-by":"publisher","first-page":"1399","DOI":"10.1056\/NEJMoa066099","volume":"356","author":"JJ Fenton","year":"2007","unstructured":"Fenton, J. J. et al. Influence of computer-aided detection on performance of screening mammography. N. Engl. J. Med. 356, 1399\u20131409 (2007).","journal-title":"N. Engl. J. Med."},{"key":"15_CR25","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision 1026\u20131034 (IEEE Computer Society, Washington, DC, 2015).","DOI":"10.1109\/ICCV.2015.123"},{"key":"15_CR26","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1109\/TMI.2016.2553401","volume":"35","author":"H Greenspan","year":"2016","unstructured":"Greenspan, H., van Ginneken, B. & Summers, R. M. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35, 1153\u20131159 (2016).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"15_CR27","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T. & Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. IEEE Comp. Soc. Conf. Comput. Vis. Pattern Recognit. 580\u2013587 (2014). https:\/\/doi.org\/10.1109\/CVPR.2014.81.","DOI":"10.1109\/CVPR.2014.81"},{"key":"15_CR28","unstructured":"Selvaraju, R. R. et al. Grad-C AM: visual explanations from deep networks via gradient-based localization. https:\/\/doi.org\/10.1109\/ICCV.2017(2014)."},{"key":"15_CR29","unstructured":"Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on \n                           Neural Information Processing Systems, Vol. 1, pp. 1097\u20131105 (Lake Tahoe, Nevada, 2012)."},{"key":"15_CR30","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T. & Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 580\u2013587 (Columbus, OH, 2014).","DOI":"10.1109\/CVPR.2014.81"},{"key":"15_CR31","doi-asserted-by":"crossref","unstructured":"Jia, Y. et al. Caffe: convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia 675\u2013678 (Orlando, Florida, 2014).","DOI":"10.1145\/2647868.2654889"},{"key":"15_CR32","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1093\/biomet\/26.4.404","volume":"26","author":"CJ Clopper","year":"1934","unstructured":"Clopper, C. J. & Pearson, E. S. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 404\u2013413 (1934).","journal-title":"Biometrika"},{"key":"15_CR33","doi-asserted-by":"publisher","first-page":"837","DOI":"10.2307\/2531595","volume":"44","author":"ER DeLong","year":"1988","unstructured":"DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837\u2013845 (1988).","journal-title":"Biometrics"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-017-0015-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-017-0015-z","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-017-0015-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T12:36:14Z","timestamp":1671626174000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-017-0015-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,4,4]]},"references-count":33,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,12]]}},"alternative-id":["15"],"URL":"https:\/\/doi.org\/10.1038\/s41746-017-0015-z","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,4,4]]},"assertion":[{"value":"18 September 2017","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 December 2017","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 December 2017","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 April 2018","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing financial interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"9"}}