{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T18:48:52Z","timestamp":1775155732518,"version":"3.50.1"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T00:00:00Z","timestamp":1727222400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T00:00:00Z","timestamp":1727222400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging. Inform. med."],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Natural language processing (NLP) is crucial to extract information accurately from unstructured text to provide insights for clinical decision-making, quality improvement, and medical research. This study compared the performance of a rule-based NLP system and a medical-domain transformer-based model to detect negated concepts in radiology reports. Using a corpus of 984 de-identified radiology reports from a large U.S.-based academic health system (1000 consecutive reports, excluding 16 duplicates), the investigators compared the rule-based medspaCy system and the Clinical Assertion and Negation Classification Bidirectional Encoder Representations from Transformers (CAN-BERT) system to detect negated expressions of terms from RadLex, the Unified Medical Language System Metathesaurus, and the Radiology Gamuts Ontology. Power analysis determined a sample size of 382 terms to achieve <jats:italic>\u03b1<\/jats:italic>\u2009=\u20090.05 and <jats:italic>\u03b2<\/jats:italic>\u2009=\u20090.8 for McNemar\u2019s test; based on an estimate of 15% negated terms, 2800 randomly selected terms were annotated manually as negated or not negated. Precision, recall, and F1 of the two models were compared using McNemar\u2019s test. Of the 2800 terms, 387 (13.8%) were negated. For negation detection, medspaCy attained a recall of 0.795, precision of 0.356, and F1 of 0.492. CAN-BERT achieved a recall of 0.785, precision of 0.768, and F1 of 0.777. Although recall was not significantly different, CAN-BERT had significantly better precision (<jats:italic>\u03c7<\/jats:italic>2\u2009=\u2009304.64; <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001). The transformer-based CAN-BERT model detected negated terms in radiology reports with high precision and recall; its precision significantly exceeded that of the rule-based medspaCy system. Use of this system will improve data extraction from textual reports to support information retrieval, AI model training, and discovery of causal relationships. <\/jats:p>","DOI":"10.1007\/s10278-024-01274-9","type":"journal-article","created":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T21:33:08Z","timestamp":1727299988000},"page":"1297-1303","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["A Large Language Model to Detect Negated Expressions in Radiology Reports"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7966-4683","authenticated-orcid":false,"given":"Yvonne","family":"Su","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9536-7504","authenticated-orcid":false,"given":"Yonatan B.","family":"Babore","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6654-7434","authenticated-orcid":false,"suffix":"Jr.","given":"Charles E.","family":"Kahn","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,9,25]]},"reference":[{"key":"1274_CR1","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1007\/s10115-022-01779-1","volume":"65","author":"MY Landolsi","year":"2023","unstructured":"Landolsi MY, Hlaoua L, Ben Romdhane L: Information extraction from electronic medical documents: state of the art and future research directions. Knowl Inf Syst 65:463-516, 2023","journal-title":"Knowl Inf Syst"},{"key":"1274_CR2","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1186\/s12911-021-01533-7","volume":"21","author":"A Casey","year":"2021","unstructured":"Casey A, et al.: A systematic review of natural language processing applied to radiology reports. BMC Med Inform Decis Mak 21:179, 2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"1274_CR3","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2022.104779","volume":"163","author":"N Linna","year":"2022","unstructured":"Linna N, Kahn CE Jr.: Applications of natural language processing in radiology: A systematic review. Int J Med Inform 163:104779, 2022","journal-title":"Int J Med Inform"},{"key":"1274_CR4","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1007\/s10278-011-9426-6","volume":"25","author":"P Lakhani","year":"2012","unstructured":"Lakhani P, Kim W, Langlotz CP: Automated detection of critical results in radiology reports. J Digit Imaging 25:30-36, 2012","journal-title":"J Digit Imaging"},{"key":"1274_CR5","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1148\/radiol.2241011118","volume":"224","author":"G Hripcsak","year":"2002","unstructured":"Hripcsak G, Austin JH, Alderson PO, Friedman C: Use of natural language processing to translate clinical information from a database of 889,921 chest radiographic reports. Radiology 224:157-163, 2002","journal-title":"Radiology"},{"key":"1274_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2023.105122","volume":"177","author":"D Fraile Navarro","year":"2023","unstructured":"Fraile Navarro D, et al.: Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review. Int J Med Inform 177:105122, 2023","journal-title":"Int J Med Inform"},{"key":"1274_CR7","doi-asserted-by":"crossref","unstructured":"Godoy E, et al.: A named entity recognition framework using transformers to identify relevant clinical findings from mammographic radiological reports: SPIE, 2023","DOI":"10.1117\/12.2670228"},{"key":"1274_CR8","doi-asserted-by":"publisher","DOI":"10.2196\/25378","volume":"23","author":"S Tsuji","year":"2021","unstructured":"Tsuji S, Wen A, Takahashi N, Zhang H, Ogasawara K, Jiang G: Developing a RadLex-based named entity recognition tool for mining textual radiology reports: development and performance evaluation study. J Med Internet Res 23:e25378, 2021","journal-title":"J Med Internet Res"},{"key":"1274_CR9","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","volume":"17","author":"GK Savova","year":"2010","unstructured":"Savova GK, et al.: Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Informatics Assoc 17:507-513, 2010","journal-title":"J Am Med Informatics Assoc"},{"key":"1274_CR10","first-page":"2013","volume":"149\u2013115","author":"HBS Liu","year":"2013","unstructured":"Liu H BS, Sohn S, et al.: An information extraction framework for cohort identification using electronic health record. AMIA Jt Summits Transl Sci Proc 2013:149-115, 2013","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"1274_CR11","first-page":"2021","volume":"438\u2013447","author":"H Eyre","year":"2021","unstructured":"Eyre H, et al.: Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python. AMIA Annu Symp Proc 2021:438-447, 2021","journal-title":"AMIA Annu Symp Proc"},{"key":"1274_CR12","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1186\/s13326-019-0211-7","volume":"10","author":"B Alex","year":"2019","unstructured":"Alex B, Grover C, Tobin R, Sudlow C, Mair G, Whiteley W: Text mining brain imaging reports. J Biomed Semantics 10:23, 2019","journal-title":"J Biomed Semantics"},{"key":"1274_CR13","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1016\/j.jbi.2015.02.010","volume":"54","author":"S Mehrabi","year":"2015","unstructured":"Mehrabi S, et al.: DEEPEN: A negation detection system for clinical text incorporating dependency relation into NegEx. J Biomed Inform 54:213-219, 2015","journal-title":"J Biomed Inform"},{"key":"1274_CR14","first-page":"2018","volume":"188\u2013196","author":"Y Peng","year":"2017","unstructured":"Peng Y, Wang X, Lu L, Bagheri M, Summers R, Lu Z: NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Jt Summits Transl Sci Proc 2017:188-196, 2018","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"1274_CR15","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1006\/jbin.2001.1029","volume":"34","author":"WW Chapman","year":"2001","unstructured":"Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG: A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 34:301-310, 2001","journal-title":"J Biomed Inform"},{"key":"1274_CR16","unstructured":"Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG: Evaluation of negation phrases in narrative clinical reports. Proc AMIA Symp:105\u2013109, 2001"},{"key":"1274_CR17","first-page":"187","volume":"136","author":"S Gindl","year":"2008","unstructured":"Gindl S, Kaiser K, Miksch S: Syntactical negation detection in clinical practice guidelines. Stud Health Technol Inform 136:187-192, 2008","journal-title":"Stud Health Technol Inform"},{"key":"1274_CR18","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1136\/jamia.2001.0080598","volume":"8","author":"PG Mutalik","year":"2001","unstructured":"Mutalik PG, Deshpande A, Nadkarni PM: Use of general-purpose negation detection to augment concept indexing of medical documents: a quantitative study using the UMLS. J Am Med Informatics Assoc 8:598-609, 2001","journal-title":"J Am Med Informatics Assoc"},{"key":"1274_CR19","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","volume":"29","author":"AJ Thirunavukarasu","year":"2023","unstructured":"Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW: Large language models in medicine. Nat Med 29:1930-1940, 2023","journal-title":"Nat Med"},{"key":"1274_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3605943","volume":"56","author":"B Min","year":"2023","unstructured":"Min B, et al.: Recent advances in natural language processing via large pre-trained language models: a survey. ACM Comput Surv 56:1-40, 2023","journal-title":"ACM Comput Surv"},{"key":"1274_CR21","unstructured":"Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren MP: CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. arXiv [preprint]:arXiv:2004.09167 [cs.CL]"},{"key":"1274_CR22","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K: BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv [preprint]:arXiv:1810.04805 [cs.CL], 2018"},{"key":"1274_CR23","doi-asserted-by":"publisher","first-page":"584","DOI":"10.1093\/jamia\/ocaa001","volume":"27","author":"C Lin","year":"2020","unstructured":"Lin C, Bethard S, Dligach D, Sadeque F, Savova G, Miller TA: Does BERT need domain adaptation for clinical negation detection? J Am Med Inform Assoc 27:584-591, 2020","journal-title":"J Am Med Inform Assoc"},{"key":"1274_CR24","unstructured":"Liu Y, et al.: RoBERTa: A robustly optimized BERT pretraining approach. arXiv [preprint]:arXiv:1907.11692 [cs.CL], 2019"},{"key":"1274_CR25","first-page":"196","volume":"158","author":"A Jaiswal","year":"2021","unstructured":"Jaiswal A, Tang L, Ghosh M, Rousseau JF, Peng Y, Ding Y: RadBERT-CL: Factually-aware contrastive learning for radiology report classification. Proceedings of Machine Learning Research 158:196-208, 2021","journal-title":"Proceedings of Machine Learning Research"},{"key":"1274_CR26","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1017\/S1351324920000509","volume":"27","author":"D Sykes","year":"2021","unstructured":"Sykes D, et al.: Comparison of rule-based and neural network models for negation detection in radiology reports. Natural Language Engineering 27:203-224, 2021","journal-title":"Natural Language Engineering"},{"key":"1274_CR27","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1186\/s12859-022-05130-x","volume":"24","author":"B van Es","year":"2023","unstructured":"van Es B, et al.: Negation detection in Dutch clinical texts: an evaluation of rule-based and machine learning methods. BMC Bioinformatics 24:10, 2023","journal-title":"BMC Bioinformatics"},{"key":"1274_CR28","doi-asserted-by":"publisher","first-page":"1595","DOI":"10.1148\/rg.266065168","volume":"26","author":"CP Langlotz","year":"2006","unstructured":"Langlotz CP: RadLex: a new method for indexing online educational materials. RadioGraphics 26:1595-1597, 2006","journal-title":"RadioGraphics"},{"key":"1274_CR29","volume-title":"The Unified Medical Language System (UMLS) Project","author":"SJ Nelson","year":"2002","unstructured":"Nelson SJ, Powell T, Humphreys BL: The Unified Medical Language System (UMLS) Project, New York: Marcel Dekker, Inc., 2002"},{"key":"1274_CR30","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1148\/rg.341135036","volume":"34","author":"JJ Budovec","year":"2014","unstructured":"Budovec JJ, Lam CA, Kahn CE Jr.: Radiology Gamuts Ontology: differential diagnosis for the Semantic Web. RadioGraphics 34:254-264, 2014","journal-title":"RadioGraphics"},{"key":"1274_CR31","unstructured":"van Aken B, et al.: Assertion detection in clinical notes: Medical language models to the rescue? Proc. Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations: City"},{"key":"1274_CR32","doi-asserted-by":"crossref","unstructured":"Kassner N, Sch\u00fctze H: Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. arXiv [preprint]:arXiv:1911.03343 [cs.CL], 2019","DOI":"10.18653\/v1\/2020.acl-main.698"},{"key":"1274_CR33","doi-asserted-by":"crossref","unstructured":"Truong TH, Baldwin T, Verspoor K, Cohn T: Language models are not naysayers: an analysis of language models on negation benchmarks. arXiv [preprint]:arXiv:2306.08189, 2023","DOI":"10.18653\/v1\/2023.starsem-1.10"},{"key":"1274_CR34","doi-asserted-by":"crossref","unstructured":"Garc\u00eda-Ferrero I, Altuna B, \u00c1lvez J, Gonzalez-Dios I, Rigau G: This is not a dataset: A large negation benchmark to challenge large language models. arXiv [preprint]:arXiv:2310.15941, 2023","DOI":"10.18653\/v1\/2023.emnlp-main.531"},{"key":"1274_CR35","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0112774","volume":"9","author":"S Wu","year":"2014","unstructured":"Wu S, et al.: Negation's not solved: generalizability versus optimizability in clinical natural language processing. PLoS One 9:e112774, 2014","journal-title":"PLoS One"},{"key":"1274_CR36","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103729","volume":"116","author":"K Sugimoto","year":"2021","unstructured":"Sugimoto K, et al.: Extracting clinical terms from radiology reports with deep learning. J Biomed Inform 116:103729, 2021","journal-title":"J Biomed Inform"},{"key":"1274_CR37","first-page":"569","volume":"310","author":"K Sugimoto","year":"2024","unstructured":"Sugimoto K, et al.: Classification of diagnostic certainty in radiology reports with deep learning. Stud Health Technol Inform 310:569-573, 2024","journal-title":"Stud Health Technol Inform"},{"key":"1274_CR38","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1097\/RTI.0000000000000622","volume":"37","author":"JA Irvin","year":"2022","unstructured":"Irvin JA, et al.: CheXED: Comparison of a Deep Learning Model to a Clinical Decision Support System for Pneumonia in the Emergency Department. J Thorac Imaging 37:162-167, 2022","journal-title":"J Thorac Imaging"},{"key":"1274_CR39","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.220055","volume":"4","author":"MA Fink","year":"2022","unstructured":"Fink MA, et al.: Deep learning-based assessment of oncologic outcomes from natural language processing of structured radiology reports. Radiol Artif Intell 4:e220055, 2022","journal-title":"Radiol Artif Intell"},{"key":"1274_CR40","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.220097","volume":"5","author":"D Nishigaki","year":"2023","unstructured":"Nishigaki D, et al.: BERT-based transfer learning in sentence-level anatomic classification of free-text radiology reports. Radiol Artif Intell 5:e220097, 2023","journal-title":"Radiol Artif Intell"},{"key":"1274_CR41","doi-asserted-by":"publisher","DOI":"10.2196\/46348","volume":"11","author":"KH Weng","year":"2023","unstructured":"Weng KH, Liu CF, Chen CJ: Deep learning approach for negation and speculation detection for automated important finding flagging and extraction in radiology report: internal validation and technique comparison study. JMIR Med Inform 11:e46348, 2023","journal-title":"JMIR Med Inform"},{"key":"1274_CR42","doi-asserted-by":"publisher","first-page":"1701","DOI":"10.1093\/jamia\/ocad119","volume":"30","author":"RA Sebro","year":"2023","unstructured":"Sebro RA, Kahn CE Jr: Automated detection of causal relationships among diseases and imaging findings in textual radiology reports. J Am Med Informatics Assoc 30:1701-1706, 2023","journal-title":"J Am Med Informatics Assoc"},{"key":"1274_CR43","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1007\/s10278-009-9250-4","volume":"24","author":"AS Wu","year":"2011","unstructured":"Wu AS, Do BH, Kim J, Rubin DL: Evaluation of negation and uncertainty detection and its impact on precision and recall in search. J Digit Imaging 24:234-242, 2011","journal-title":"J Digit Imaging"}],"container-title":["Journal of Imaging Informatics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-024-01274-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-024-01274-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-024-01274-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,20]],"date-time":"2025-05-20T17:27:42Z","timestamp":1747762062000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-024-01274-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,25]]},"references-count":43,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["1274"],"URL":"https:\/\/doi.org\/10.1007\/s10278-024-01274-9","relation":{},"ISSN":["2948-2933"],"issn-type":[{"value":"2948-2933","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,25]]},"assertion":[{"value":"23 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 August 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 September 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 September 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Study protocol approved by the University of Pennsylvania IRB. Informed consent from patients was waived.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}]}}