{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T09:55:55Z","timestamp":1792662955098,"version":"4.5.0"},"reference-count":24,"publisher":"Oxford University Press (OUP)","issue":"1149","license":[{"start":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T00:00:00Z","timestamp":1684972800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,8,1]]},"abstract":"<jats:sec>\n                  <jats:title>Objectives:<\/jats:title>\n                  <jats:p>Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods:<\/jats:title>\n                  <jats:p>We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis\/colitis on abdomen\/pelvis CT reports, ischemic infarct on brain CT\/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results:<\/jats:title>\n                  <jats:p>In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions:<\/jats:title>\n                  <jats:p>With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (&amp;lt;1000) and more imbalanced training data, traditional NLP techniques performed better.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Advances in knowledge:<\/jats:title>\n                  <jats:p>Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1259\/bjr.20220769","type":"journal-article","created":{"date-parts":[[2023,5,10]],"date-time":"2023-05-10T13:15:32Z","timestamp":1683724532000},"source":"Crossref","is-referenced-by-count":35,"title":["Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?"],"prefix":"10.1093","volume":"96","author":[{"given":"Eric","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Harvard Medical School , Boston, MA, USA"},{"name":"Department of Radiology, Massachusetts General Hospital, Harvard Medical School , Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew D","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Radiology and Diagnostic Imaging, University of Alberta , Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shruti","family":"Raghavan","sequence":"additional","affiliation":[{"name":"Department of Radiology, Massachusetts General Hospital, Harvard Medical School , Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francis","family":"Deng","sequence":"additional","affiliation":[{"name":"Department of Radiology, Massachusetts General Hospital, Harvard Medical School , Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Min","family":"Lang","sequence":"additional","affiliation":[{"name":"Department of Radiology, Massachusetts General Hospital, Harvard Medical School , Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marc D","family":"Succi","sequence":"additional","affiliation":[{"name":"Department of Radiology, Massachusetts General Hospital, Harvard Medical School , Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ambrose J","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Radiology, Massachusetts General Hospital, Harvard Medical School , Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jayashree","family":"Kalpathy-Cramer","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, University of Colorado , Aurora, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,5,25]]},"reference":[{"key":"2024050216083998900_b1","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1148\/radiol.16142770","article-title":"Natural language processing in radiology: a systematic review","volume":"279","author":"Pons","year":"2016","journal-title":"Radiology"},{"key":"2024050216083998900_b2","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1148\/radiol.2018171093","article-title":"Natural language-based machine learning models for the annotation of clinical radiology reports","volume":"287","author":"Zech","year":"2018","journal-title":"Radiology"},{"key":"2024050216083998900_b3","doi-asserted-by":"crossref","first-page":"1421","DOI":"10.1001\/jamaoncol.2019.1800","article-title":"Assessment of deep natural language processing in ascertaining oncologic outcomes from radiology reports","volume":"5","author":"Kehl","year":"2019","journal-title":"JAMA Oncol"},{"key":"2024050216083998900_b4","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.artmed.2018.11.004","article-title":"Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification","volume":"97","author":"Banerjee","year":"2019","journal-title":"Artif Intell Med"},{"key":"2024050216083998900_b5","doi-asserted-by":"crossref","first-page":"734","DOI":"10.2214\/AJR.18.19869","article-title":"Automatic disease annotation from radiology reports using artificial intelligence implemented by a recurrent neural network","volume":"212","author":"Lee","year":"2019","journal-title":"AJR Am J Roentgenol"},{"key":"2024050216083998900_b6","doi-asserted-by":"crossref","first-page":"750","DOI":"10.2214\/AJR.16.16128","article-title":"Performance of a machine learning classifier of knee MRI reports in two large academic radiology practices: a tool to estimate diagnostic yield","volume":"208","author":"Hassanpour","year":"2017","journal-title":"AJR Am J Roentgenol"},{"key":"2024050216083998900_b7","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.jbi.2017.11.012","article-title":"Radiology report annotation using intelligent word embeddings: applied to multi-institutional chest CT cohort","volume":"77","author":"Banerjee","year":"2018","journal-title":"J Biomed Inform"},{"key":"2024050216083998900_b8","author":"Vaswani","year":"2017","journal-title":"Attention Is All You Need"},{"key":"2024050216083998900_b9","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0229963","article-title":"Supervised and unsupervised language modelling in Chest X-Ray radiological reports","author":"Drozdov","year":"2020","journal-title":"PLOS ONE"},{"key":"2024050216083998900_b10","article-title":"Bow vs BERT: classification","author":"Chilakapati","year":"2019","journal-title":"Internet"},{"key":"2024050216083998900_b11","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2024050216083998900_b12","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.ajem.2021.05.057","article-title":"Automated tracking of emergency department abdominal CT findings during the COVID-19 pandemic using natural language processing","volume":"49","author":"Li","year":"2021","journal-title":"Am J Emerg Med"},{"key":"2024050216083998900_b13","doi-asserted-by":"crossref","first-page":"429","DOI":"10.3174\/ajnr.A6961","article-title":"Analysis of stroke detection during the COVID-19 pandemic using natural language processing of radiology reports","volume":"42","author":"Li","year":"2021","journal-title":"AJNR Am J Neuroradiol"},{"key":"2024050216083998900_b14","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1016\/j.acra.2021.01.017","article-title":"Automated radiology-arthroscopy correlation of knee meniscal tears using natural language processing algorithms","volume":"29","author":"Li","year":"2022","journal-title":"Acad Radiol"},{"key":"2024050216083998900_b15","volume-title":"Natural language processing with Python","author":"Bird","year":"2009","edition":"1st"},{"key":"2024050216083998900_b16","first-page":"785","volume-title":". In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining [Internet].","author":"Chen","year":"2016"},{"key":"2024050216083998900_b17","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1162\/EVCO_a_00168","article-title":"LM-cma: an alternative to l-bfgs for large-scale black box optimization","volume":"25","author":"Loshchilov","year":"2017","journal-title":"Evol Comput"},{"key":"2024050216083998900_b18","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-60548-3_18","article-title":"Federated learning for breast density classification: a real-world implementation","author":"Roth","year":"2020"},{"key":"2024050216083998900_b19","doi-asserted-by":"crossref","first-page":"1247","DOI":"10.1007\/s00234-020-02420-0","article-title":"Comparison of deep learning models for natural language processing-based classification of non-English head CT reports","volume":"62","author":"Barash","year":"2020","journal-title":"Neuroradiology"},{"key":"2024050216083998900_b20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1200\/CCI.18.00138","article-title":"Natural language processing for automated quantification of brain metastases reported in free-text radiology reports","author":"Senders","year":"2019","journal-title":"JCO Clinical Cancer Informatics"},{"key":"2024050216083998900_b21","doi-asserted-by":"crossref","first-page":"1422","DOI":"10.1016\/j.acra.2018.03.008","article-title":"Comparison of natural language processing rules-based and machine-learning systems to identify lumbar spine imaging findings related to low back pain","volume":"25","author":"Tan","year":"2018","journal-title":"Acad Radiol"},{"key":"2024050216083998900_b22","doi-asserted-by":"crossref","first-page":"S188","DOI":"10.1016\/j.acra.2021.09.005","article-title":"A comparison of natural language processing methods for the classification of lumbar spine imaging findings related to lower back pain","volume":"29 Suppl 3","author":"Jujjavarapu","year":"2022","journal-title":"Acad Radiol"},{"key":"2024050216083998900_b23","doi-asserted-by":"crossref","DOI":"10.1186\/s12911-021-01533-7","article-title":"A systematic review of natural language processing applied to radiology reports","volume":"21","author":"Casey","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"2024050216083998900_b24","doi-asserted-by":"crossref","DOI":"10.1186\/s12880-021-00671-8","article-title":"The reporting quality of natural language processing studies: systematic review of studies of radiology reports","volume":"21","author":"Davidson","year":"2021","journal-title":"BMC Med Imaging"}],"container-title":["The British Journal of Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bjr\/article-pdf\/96\/1149\/20220769\/57370836\/bjr.20220769.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bjr\/article-pdf\/96\/1149\/20220769\/57370836\/bjr.20220769.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,2]],"date-time":"2024-05-02T18:31:52Z","timestamp":1714674712000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bjr\/article\/doi\/10.1259\/bjr.20220769\/7469147"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,25]]},"references-count":24,"journal-issue":{"issue":"1149","published-online":{"date-parts":[[2023,5,25]]},"published-print":{"date-parts":[[2023,8,1]]}},"URL":"https:\/\/doi.org\/10.1259\/bjr.20220769","relation":{},"ISSN":["0007-1285","1748-880X"],"issn-type":[{"value":"0007-1285","type":"print"},{"value":"1748-880X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,8,1]]},"published":{"date-parts":[[2023,5,25]]},"article-number":"20220769"}}