{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T22:55:42Z","timestamp":1793228142183,"version":"4.5.0"},"reference-count":42,"publisher":"American Medical Association (AMA)","issue":"8","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"Importance<\/jats:title>Clinical text reports from head computed tomography (CT) represent rich, incompletely utilized information regarding acute brain injuries and neurologic outcomes. CT reports are unstructured; thus, extracting information at scale requires automated natural language processing (NLP). However, designing new NLP algorithms for each individual injury category is an unwieldy proposition. An NLP tool that summarizes all injuries in head CT reports would facilitate exploration of large data sets for clinical significance of neuroradiological findings.<\/jats:p><\/jats:sec>Objective<\/jats:title>To automatically extract acute brain pathological data and their features from head CT reports.<\/jats:p><\/jats:sec>Design, Setting, and Participants<\/jats:title>This diagnostic study developed a 2-part named entity recognition (NER) NLP model to extract and summarize data on acute brain injuries from head CT reports. The model, termed BrainNERD<\/jats:italic>, extracts and summarizes detailed brain injury information for research applications. Model development included building and comparing 2 NER models using a custom dictionary of terms, including lesion type, location, size, and age, then designing a rule-based decoder using NER outputs to evaluate for the presence or absence of injury subtypes. BrainNERD was evaluated against independent test data sets of manually classified reports, including 2 external validation sets. The model was trained on head CT reports from 1152 patients generated by neuroradiologists at the Yale Acute Brain Injury Biorepository. External validation was conducted using reports from 2 outside institutions. Analyses were conducted from May 2020 to December 2021.<\/jats:p><\/jats:sec>Main Outcomes and Measures<\/jats:title>Performance of the BrainNERD model was evaluated using precision, recall, and F1 scores based on manually labeled independent test data sets.<\/jats:p><\/jats:sec>Results<\/jats:title>A total of 1152 patients (mean [SD] age, 67.6 [16.1] years; 586 [52%] men), were included in the training set. NER training using transformer architecture and bidirectional encoder representations from transformers was significantly faster than spaCy. For all metrics, the 10-fold cross-validation performance was 93% to 99%. The final test performance metrics for the NER test data set were 98.82% (95% CI, 98.37%-98.93%) for precision, 98.81% (95% CI, 98.46%-99.06%) for recall, and 98.81% (95% CI, 98.40%-98.94%) for the F<\/jats:italic> score. The expert review comparison metrics were 99.06% (95% CI, 97.89%-99.13%) for precision, 98.10% (95% CI, 97.93%-98.77%) for recall, and 98.57% (95% CI, 97.78%-99.10%) for the F<\/jats:italic> score. The decoder test set metrics were 96.06% (95% CI, 95.01%-97.16%) for precision, 96.42% (95% CI, 94.50%-97.87%) for recall, and 96.18% (95% CI, 95.151%-97.16%) for the F<\/jats:italic> score. Performance in external institution report validation including 1053 head CR reports was greater than 96%.<\/jats:p><\/jats:sec>Conclusions and Relevance<\/jats:title>These findings suggest that the BrainNERD model accurately extracted acute brain injury terms and their properties from head CT text reports. This freely available new tool could advance clinical research by integrating information in easily gathered head CT reports to expand knowledge of acute brain injury radiographic phenotypes.<\/jats:p><\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2022.27109","type":"journal-article","created":{"date-parts":[[2022,8,19]],"date-time":"2022-08-19T16:05:28Z","timestamp":1660925128000},"page":"e2227109","source":"Crossref","is-referenced-by-count":24,"title":["Development and Validation of a Model to Identify Critical Brain Injuries Using Natural Language Processing of Text Computed Tomography Reports"],"prefix":"10.1001","volume":"5","author":[{"given":"Victor M.","family":"Torres-Lopez","sequence":"first","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Grace E.","family":"Rovenolt","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angelo J.","family":"Olcese","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriella E.","family":"Garcia","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah M.","family":"Chacko","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amber","family":"Robinson","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edward","family":"Gaiser","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julian","family":"Acosta","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alison L.","family":"Herman","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lindsey R.","family":"Kuohn","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Megan","family":"Leary","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandria L.","family":"Soto","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Safoora","family":"Fatima","sequence":"additional","affiliation":[{"name":"Department of Neurology, University of Wisconsin, Madison"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guido J.","family":"Falcone","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seyedmehdi","family":"Payabvash","sequence":"additional","affiliation":[{"name":"Department of Radiology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richa","family":"Sharma","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron F.","family":"Struck","sequence":"additional","affiliation":[{"name":"Department of Neurology, University of Wisconsin, Madison"},{"name":"William S Middleton Veterans Hospital, Madison, Wisconsin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin N.","family":"Sheth","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M. Brandon","family":"Westover","sequence":"additional","affiliation":[{"name":"Department of Neurology, Massachusetts General Hospital, Boston"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer A.","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Neurology, Yale University, New Haven, Connecticut"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2022,8,16]]},"reference":[{"issue":"9","key":"zoi220765r1","doi-asserted-by":"publisher","first-page":"e139","DOI":"10.1161\/CIR.0000000000000757","article-title":"Heart disease and stroke statistics\u20142020 update: a report from the American Heart Association.","volume":"141","author":"Virani","year":"2020","journal-title":"Circulation"},{"issue":"4","key":"zoi220765r3","doi-asserted-by":"publisher","first-page":"438","DOI":"10.1177\/1747493016632244","article-title":"Performance of e-ASPECTS software in comparison to that of stroke physicians on assessing CT scans of acute ischemic stroke patients.","volume":"11","author":"Herweh","year":"2016","journal-title":"Int J Stroke"},{"issue":"1","key":"zoi220765r4","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1148\/radiol.2015142346","article-title":"Automated detection, localization, and classification of traumatic vertebral body fractures in the thoracic and lumbar spine at CT.","volume":"278","author":"Burns","year":"2016","journal-title":"Radiology"},{"key":"zoi220765r5","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1186\/s41199-020-00053-7","article-title":"Applications of radiomics in precision diagnosis, prognostication and treatment planning of head and neck squamous cell carcinomas.","volume":"5","author":"Haider","year":"2020","journal-title":"Cancers Head Neck"},{"issue":"4","key":"zoi220765r6","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1016\/j.ejca.2011.11.036","article-title":"Radiomics: extracting more information from medical images using advanced feature analysis.","volume":"48","author":"Lambin","year":"2012","journal-title":"Eur J Cancer"},{"key":"zoi220765r7","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1016\/j.mri.2017.02.008","article-title":"A novel approach to segmentation and measurement of medical image using level set methods.","volume":"39","author":"Chen","year":"2017","journal-title":"Magn Reson Imaging"},{"key":"zoi220765r8","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.media.2016.05.004","article-title":"Brain tumor segmentation with Deep Neural Networks.","volume":"35","author":"Havaei","year":"2017","journal-title":"Med Image Anal"},{"key":"zoi220765r9","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.media.2016.10.004","article-title":"Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation.","volume":"36","author":"Kamnitsas","year":"2017","journal-title":"Med Image Anal"},{"issue":"2","key":"zoi220765r10","doi-asserted-by":"publisher","first-page":"774","DOI":"10.1016\/j.neuroimage.2012.01.021","article-title":"FreeSurfer.","volume":"62","author":"Fischl","year":"2012","journal-title":"Neuroimage"},{"issue":"3 Pt B","key":"zoi220765r12","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1016\/j.jacr.2017.12.028","article-title":"Machine learning in medical imaging.","volume":"15","author":"Giger","year":"2018","journal-title":"J Am Coll Radiol"},{"key":"zoi220765r13","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis.","volume":"42","author":"Litjens","year":"2017","journal-title":"Med Image Anal"},{"issue":"2","key":"zoi220765r14","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1148\/radiol.2018171820","article-title":"Current applications and future impact of machine learning in radiology.","volume":"288","author":"Choy","year":"2018","journal-title":"Radiology"},{"key":"zoi220765r15","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1016\/j.artmed.2015.09.007","article-title":"Information extraction from multi-institutional radiology reports.","volume":"66","author":"Hassanpour","year":"2016","journal-title":"Artif Intell Med"},{"issue":"1","key":"zoi220765r16","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1186\/s12911-021-01533-7","article-title":"A systematic review of natural language processing applied to radiology reports.","volume":"21","author":"Casey","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"zoi220765r19","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.cosrev.2018.06.001","article-title":"Recent named entity recognition and classification techniques: a systematic review.","volume":"29","author":"Goyal","year":"2018","journal-title":"Comput Sci Rev"},{"issue":"1","key":"zoi220765r20","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1075\/li","article-title":"A survey of named entity recognition and classification.","volume":"30","author":"Nadeau","year":"2007","journal-title":"Lingvistic\u00e6 Investigationes"},{"key":"zoi220765r21","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1186\/s13326-019-0211-7","article-title":"Text mining brain imaging reports.","volume":"10","author":"Alex","year":"2019","journal-title":"J Biomed Semantics"},{"issue":"1","key":"zoi220765r22","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1186\/s12911-019-0908-7","article-title":"A validated natural language processing algorithm for brain imaging phenotypes from radiology reports in UK electronic health records.","volume":"19","author":"Wheater","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"issue":"3","key":"zoi220765r23","doi-asserted-by":"publisher","first-page":"429","DOI":"10.3174\/ajnr.A6961","article-title":"Analysis of stroke detection during the COVID-19 pandemic using natural language processing of radiology reports.","volume":"42","author":"Li","year":"2021","journal-title":"AJNR Am J Neuroradiol"},{"issue":"6","key":"zoi220765r24","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0234908","article-title":"Machine learning and natural language processing methods to identify ischemic stroke, acuity and location from radiology reports.","volume":"15","author":"Ong","year":"2020","journal-title":"PLoS One"},{"issue":"2","key":"zoi220765r25","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1212\/WNL.0000000000003490","article-title":"Large-scale identification of patients with cerebral aneurysms using natural language processing.","volume":"88","author":"Castro","year":"2017","journal-title":"Neurology"},{"issue":"3","key":"zoi220765r27","doi-asserted-by":"publisher","first-page":"845","DOI":"10.1148\/radiol.2017171115","article-title":"Deep learning to classify radiology free-text reports.","volume":"286","author":"Chen","year":"2018","journal-title":"Radiology"},{"issue":"2","key":"zoi220765r28","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1148\/radiol.16142770","article-title":"Natural language processing in radiology: a systematic review.","volume":"279","author":"Pons","year":"2016","journal-title":"Radiology"},{"key":"zoi220765r35","first-page":"3104","article-title":"Sequence to sequence learning with neural networks.","volume":"4","author":"Sutskever","year":"2014","journal-title":"Adv Neural Inf Process Syst"},{"issue":"5","key":"zoi220765r36","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries.","volume":"34","author":"Chapman","year":"2001","journal-title":"J Biomed Inform"},{"key":"zoi220765r38","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1038\/s43856-021-00043-x","article-title":"Development and multicenter validation of chest X-ray radiography interpretations based on natural language processing.","volume":"1","author":"Zhang","year":"2021","journal-title":"Commun Med"},{"key":"zoi220765r18","first-page":"363","volume-title":"Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics","author":"Finkel","year":"2005"},{"key":"zoi220765r26","first-page":"35","volume-title":"Proceedings of The First Workshop Medical Informatics and Healthcare Held With the 23rd SIGKDD Conference on Knowledge Discovery and Data Mining","author":"Tiwari","year":"2017"},{"key":"zoi220765r32","first-page":"1480","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Yang","year":"2016"},{"key":"zoi220765r34","first-page":"5999","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"zoi220765r39","volume-title":"Natural Language Processing With Python","author":"Bird","year":"2009"},{"key":"zoi220765r2","unstructured":"Centers for Disease Control and Prevention. TBI data. Accessed July 13, 2022. https:\/\/www.cdc.gov\/traumaticbraininjury\/data\/tbi-hospitalizations.html."},{"key":"zoi220765r11","unstructured":"Papademetris? X, Jackowski? M, Rajeevan? N, Okuda? H, Constable? RT, Staib? LH. BioImage Suite website. Accessed July 14, 2022. http:\/\/www.bioimagesuite.org"},{"key":"zoi220765r17","doi-asserted-by":"crossref","unstructured":"Tjong Kim Sang? EF, de Meulder? F. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. arXiv. Preprint posted online June 12, 2003. doi:10.48550\/arXiv.cs\/0306050","DOI":"10.3115\/1119176.1119195"},{"key":"zoi220765r29","unstructured":"TagEditor. Accessed July 19, 2022. https:\/\/github.com\/d5555\/TagEditor"},{"key":"zoi220765r30","unstructured":"Honnibal? M, Montani? I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. Accessed July 13, 2022. https:\/\/spacy.io\/"},{"key":"zoi220765r31","unstructured":"Explosion. spaCy\u2019s entity recognition model: incremental parsing with Bloom embeddings & residual CNNs. Accessed July 13, 2022. https:\/\/www.youtube.com\/watch?v=sqDHBH9IjRU"},{"key":"zoi220765r33","unstructured":"Devlin? J, Chang? MW, Lee? K, Toutanova? K. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. Preprint posted online May 24, 2019. doi:10.48550\/arXiv.1810.04805"},{"key":"zoi220765r37","unstructured":"BrainNERD. Accessed July 25, 2022. https:\/\/github.com\/DrKimLab\/BrainNERD"},{"key":"zoi220765r40","unstructured":"spaCy. Facts & figures. Accessed July 13, 2022. https:\/\/spacy.io\/usage\/facts-figures"},{"key":"zoi220765r41","doi-asserted-by":"crossref","unstructured":"Wolf? T, Debut? L, Sanh? V, . HuggingFace\u2019s transformers: state-of-the-art natural language processing. arXiv. Preprint posted online July 14, 2020. doi:10.48550\/arXiv.1910.03771","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"zoi220765r42","unstructured":"spaCy. Training pipelines & models. Accessed July 13, 2022. https:\/\/spacy.io\/usage\/training"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2795179\/torreslopez_2022_oi_220765_1664286350.55909.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,29]],"date-time":"2022-09-29T15:00:45Z","timestamp":1664463645000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2795179"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,16]]},"references-count":42,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2022,8,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2022.27109","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,16]]}}}