{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T18:16:03Z","timestamp":1772216163484,"version":"3.50.1"},"reference-count":23,"publisher":"Public Library of Science (PLoS)","issue":"3","license":[{"start":{"date-parts":[[2023,3,13]],"date-time":"2023-03-13T00:00:00Z","timestamp":1678665600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Machine learning (ML) algorithms to detect critical findings on head CTs may expedite patient management. Most ML algorithms for diagnostic imaging analysis utilize dichotomous classifications to determine whether a specific abnormality is present. However, imaging findings may be indeterminate, and algorithmic inferences may have substantial uncertainty. We incorporated awareness of uncertainty into an ML algorithm that detects intracranial hemorrhage or other urgent intracranial abnormalities and evaluated prospectively identified, 1000 consecutive noncontrast head CTs assigned to Emergency Department Neuroradiology for interpretation. The algorithm classified the scans into high (IC+) and low (IC-) probabilities for intracranial hemorrhage or other urgent abnormalities. All other cases were designated as No Prediction (NP) by the algorithm. The positive predictive value for IC+ cases (N = 103) was 0.91 (CI: 0.84\u20130.96), and the negative predictive value for IC- cases (N = 729) was 0.94 (0.91\u20130.96). Admission, neurosurgical intervention, and 30-day mortality rates for IC+ was 75% (63\u201384), 35% (24\u201347), and 10% (4\u201320), compared to 43% (40\u201347), 4% (3\u20136), and 3% (2\u20135) for IC-. There were 168 NP cases, of which 32% had intracranial hemorrhage or other urgent abnormalities, 31% had artifacts and postoperative changes, and 29% had no abnormalities. An ML algorithm incorporating uncertainty classified most head CTs into clinically relevant groups with high predictive values and may help accelerate the management of patients with intracranial hemorrhage or other urgent intracranial abnormalities.<\/jats:p>","DOI":"10.1371\/journal.pone.0281900","type":"journal-article","created":{"date-parts":[[2023,3,13]],"date-time":"2023-03-13T13:30:06Z","timestamp":1678714206000},"page":"e0281900","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":9,"title":["Incorporating algorithmic uncertainty into a clinical machine deep learning algorithm for urgent head CTs"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2929-5677","authenticated-orcid":true,"given":"Byung C.","family":"Yoon","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stuart R.","family":"Pomerantz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nathaniel D.","family":"Mercaldo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Swati","family":"Goyal","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eric M.","family":"L\u2019Italien","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael H.","family":"Lev","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karen A.","family":"Buch","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bradley R.","family":"Buchbinder","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John W.","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Conklin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rajiv","family":"Gupta","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"George J.","family":"Hunter","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shahmir C.","family":"Kamalian","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hillary R.","family":"Kelly","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5038-4183","authenticated-orcid":true,"given":"Otto","family":"Rapalino","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sandra P.","family":"Rincon","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Javier M.","family":"Romero","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julian","family":"He","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pamela W.","family":"Schaefer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Synho","family":"Do","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0234-3897","authenticated-orcid":true,"given":"Ramon Gilberto","family":"Gonz\u00e1lez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2023,3,13]]},"reference":[{"issue":"2","key":"pone.0281900.ref001","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1148\/radiol.2020191479","article-title":"Computer-aided Detection of Brain Metastases in T1-weighted MRI for Stereotactic Radiosurgery Using Deep Learning Single-Shot Detectors.","volume":"295","author":"Z Zhou","year":"2020","journal-title":"Radiology"},{"issue":"3","key":"pone.0281900.ref002","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1148\/radiol.2020200334","article-title":"Detecting Large Vessel Occlusion at Multiphase CT Angiography by Using a Deep Convolutional Neural Network.","volume":"297","author":"MT Stib","year":"2020","journal-title":"Radiology"},{"issue":"9","key":"pone.0281900.ref003","article-title":"Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration.","volume":"1","author":"MR Arbabshirani","year":"2018","journal-title":"NPJ Digit Med"},{"key":"pone.0281900.ref004","doi-asserted-by":"crossref","first-page":"e209","DOI":"10.1016\/j.wneu.2021.02.134","article-title":"Assessment of an Artificial Intelligence Algorithm for Detection of Intracranial Hemorrhage.","volume":"150","author":"RA Rava","year":"2021","journal-title":"World Neurosurg"},{"issue":"2","key":"pone.0281900.ref005","doi-asserted-by":"crossref","first-page":"e210168","DOI":"10.1148\/ryai.210168","article-title":"Utilization of Artificial Intelligence-based Intracranial Hemorrhage Detection on Emergent Noncontrast CT Images in Clinical Workflow.","volume":"4","author":"M Seyam","year":"2022","journal-title":"Radiol Artif Intell."},{"issue":"2","key":"pone.0281900.ref006","doi-asserted-by":"crossref","first-page":"273","DOI":"10.3174\/ajnr.A6926","article-title":"Automated Cerebral Hemorrhage Detection Using RAPID.","volume":"42","author":"JJ Heit","year":"2021","journal-title":"American Journal of Neuroradiology"},{"key":"pone.0281900.ref007","first-page":"137","volume-title":"Three-Way Decision for Handling Uncertainty in Machine Learning: A Narrative Review.","author":"A Campagner","year":"2020"},{"issue":"3","key":"pone.0281900.ref008","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","article-title":"Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.","volume":"110","author":"E H\u00fcllermeier","year":"2021","journal-title":"Machine Learning."},{"issue":"1","key":"pone.0281900.ref009","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-020-00367-3","article-title":"Second opinion needed: communicating uncertainty in medical machine learning.","volume":"4","author":"B Kompa","year":"2021","journal-title":"npj Digital Medicine."},{"key":"pone.0281900.ref010","article-title":"Very Deep Convolutional Networks for Large-Scale Image Recognition2014 September","volume":"01","author":"K Simonyan","year":"2014"},{"key":"pone.0281900.ref011","first-page":"770","author":"K He","year":"2016","journal-title":"Deep Residual Learning for Image Recognition"},{"key":"pone.0281900.ref012","article-title":"Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning2016 February","volume":"01","author":"C Szegedy","year":"2016"},{"key":"pone.0281900.ref013","first-page":"2818","article-title":"Rethinking the Inception Architecture for Computer Vision.","author":"C Szegedy","year":"2016"},{"issue":"3","key":"pone.0281900.ref014","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1038\/s41551-018-0324-9","article-title":"An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets","volume":"3","author":"H Lee","year":"2019","journal-title":"Nature Biomedical Engineering"},{"key":"pone.0281900.ref015","volume":"01","author":"M Lin","year":"2013","journal-title":"Network In Network2013 December"},{"key":"pone.0281900.ref016","first-page":"543","article-title":"A method for unconstrained convex minimization problem with the rate of convergence o(1\/k^2)","volume":"269","author":"Y. Nesterov","year":"1983","journal-title":"Doklady AN USSR"},{"key":"pone.0281900.ref017","volume":"01","author":"M Abadi","year":"2016","journal-title":"TensorFlow: A system for large-scale machine learning"},{"key":"pone.0281900.ref018","first-page":"3462","article-title":"ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases.","author":"X Wang","year":"2017"},{"key":"pone.0281900.ref019","article-title":"Multi-label Class-imbalanced Action Recognition in Hockey Videos via 3D Convolutional Neural Networks2017 September","volume":"01","author":"K Sozykin","year":"2017"},{"key":"pone.0281900.ref020","article-title":"The R package GEEPACK for generalized estimating equations","volume":"15","author":"S H\u00f8jsgaard","year":"2005","journal-title":"Journal of Statistical Software"},{"issue":"5","key":"pone.0281900.ref021","doi-asserted-by":"crossref","first-page":"e0176622","DOI":"10.1371\/journal.pone.0176622","article-title":"Comparative accuracy of CT perfusion in diagnosing acute ischemic stroke: A systematic review of 27 trials.","volume":"12","author":"J Shen","year":"2017","journal-title":"PLoS One."},{"issue":"1","key":"pone.0281900.ref022","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1038\/s41598-023-27496-5","article-title":"Head CT deep learning model is highly accurate for early infarct estimation.","volume":"13","author":"R Gauriau","year":"2023","journal-title":"Sci Rep."},{"issue":"2","key":"pone.0281900.ref023","doi-asserted-by":"crossref","first-page":"e200024","DOI":"10.1148\/ryai.2020200024","article-title":"Active Reprioritization of the Reading Worklist Using Artificial Intelligence Has a Beneficial Effect on the Turnaround Time for Interpretation of Head CT with Intracranial Hemorrhage.","volume":"3","author":"TJ O\u2019Neill","year":"2021","journal-title":"Radiol Artif Intell"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0281900","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,13]],"date-time":"2023-03-13T13:30:27Z","timestamp":1678714227000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0281900"}},"subtitle":[],"editor":[{"given":"Tommaso","family":"Martino","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2023,3,13]]},"references-count":23,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,3,13]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0281900","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.07.19.22277808","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,13]]}}}