{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T20:46:29Z","timestamp":1781901989605,"version":"3.54.5"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,10,2]],"date-time":"2021-10-02T00:00:00Z","timestamp":1633132800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,2]],"date-time":"2021-10-02T00:00:00Z","timestamp":1633132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100012338","name":"Alan Turing Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100012338","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]},{"name":"HDR-UK"},{"DOI":"10.13039\/501100000589","name":"Chief Scientist Office","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000589","id-type":"DOI","asserted-by":"publisher"}]},{"name":"alzheimer\u2019s society"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Imaging"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Automated language analysis of radiology reports using natural language processing (NLP) can provide valuable information on patients\u2019 health and disease. With its rapid development, NLP studies should have transparent methodology to allow comparison of approaches and reproducibility. This systematic review aims to summarise the characteristics and reporting quality of studies applying NLP to radiology reports.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We searched Google Scholar for studies published in English that applied NLP to radiology reports of any imaging modality between January 2015 and October 2019. At least two reviewers independently performed screening and completed data extraction. We specified 15 criteria relating to data source, datasets, ground truth, outcomes, and reproducibility for quality assessment. The primary NLP performance measures were precision, recall and F1 score.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Of the 4,836 records retrieved, we included 164 studies that used NLP on radiology reports. The commonest clinical applications of NLP were disease information or classification (28%) and diagnostic surveillance (27.4%). Most studies used English radiology reports (86%). Reports from mixed imaging modalities were used in 28% of the studies. Oncology (24%) was the most frequent disease area. Most studies had dataset size\u2009&gt;\u2009200 (85.4%) but the proportion of studies that described their annotated, training, validation, and test set were 67.1%, 63.4%, 45.7%, and 67.7% respectively. About half of the studies reported precision (48.8%) and recall (53.7%). Few studies reported external validation performed (10.8%), data availability (8.5%) and code availability (9.1%). There was no pattern of performance associated with the overall reporting quality.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>There is a range of potential clinical applications for NLP of radiology reports in health services and research. However, we found suboptimal reporting quality that precludes comparison, reproducibility, and replication. Our results support the need for development of reporting standards specific to clinical NLP studies.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12880-021-00671-8","type":"journal-article","created":{"date-parts":[[2021,10,4]],"date-time":"2021-10-04T14:45:42Z","timestamp":1633358742000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["The reporting quality of natural language processing studies: systematic review of studies of radiology reports"],"prefix":"10.1186","volume":"21","author":[{"given":"Emma M.","family":"Davidson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael T. C.","family":"Poon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arlene","family":"Casey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreas","family":"Grivas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Duma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hang","family":"Dong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"V\u00edctor","family":"Su\u00e1rez-Paniagua","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claire","family":"Grover","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Tobin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heather","family":"Whalley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Honghan","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Beatrice","family":"Alex","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"Whiteley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,10,2]]},"reference":[{"issue":"1","key":"671_CR1","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1148\/rg.2016150080","volume":"36","author":"T Cai","year":"2016","unstructured":"Cai T, Giannopoulos AA, Yu S, Kelil T, Ripley B, Kumamaru KK, et al. Natural language processing technologies in radiology research and clinical applications. Radiographics. 2016;36(1):176\u201391.","journal-title":"Radiographics"},{"key":"671_CR2","first-page":"16927","volume":"368","author":"S Vollmer","year":"2020","unstructured":"Vollmer S, Mateen BA, Bohner G, Kir\u00e1ly FJ, Ghani R, Jonsson P, et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ. 2020;368:16927.","journal-title":"BMJ"},{"issue":"10","key":"671_CR3","doi-asserted-by":"publisher","first-page":"e549","DOI":"10.1016\/S2589-7500(20)30219-3","volume":"2","author":"S Cruz Rivera","year":"2020","unstructured":"Cruz Rivera S, Liu X, Chan A-W, Denniston AK, Calvert MJ, Ashrafian H, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit Health. 2020;2(10):e549\u201360.","journal-title":"Lancet Digit Health"},{"issue":"10","key":"671_CR4","doi-asserted-by":"publisher","first-page":"e537","DOI":"10.1016\/S2589-7500(20)30218-1","volume":"2","author":"X Liu","year":"2020","unstructured":"Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, Ashrafian H, et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Lancet Digit Health. 2020;2(10):e537\u201348.","journal-title":"Lancet Digit Health"},{"issue":"3","key":"671_CR5","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1148\/radiol.2019192515","volume":"294","author":"DA Bluemke","year":"2019","unstructured":"Bluemke DA, Moy L, Bredella MA, Ertl-Wagner BB, Fowler KJ, Goh VJ, et al. Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers\u2014from the radiology editorial board. Radiology. 2019;294(3):487\u20139.","journal-title":"Radiology"},{"issue":"3","key":"671_CR6","doi-asserted-by":"publisher","first-page":"e034568-e","DOI":"10.1136\/bmjopen-2019-034568","volume":"10","author":"M Yusuf","year":"2020","unstructured":"Yusuf M, Atal I, Li J, Smith P, Ravaud P, Fergie M, et al. Reporting quality of studies using machine learning models for medical diagnosis: a systematic review. BMJ Open. 2020;10(3):e034568-e.","journal-title":"BMJ Open"},{"issue":"10181","key":"671_CR7","doi-asserted-by":"publisher","first-page":"1577","DOI":"10.1016\/S0140-6736(19)30037-6","volume":"393","author":"GS Collins","year":"2019","unstructured":"Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet. 2019;393(10181):1577\u20139.","journal-title":"Lancet"},{"key":"671_CR8","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.jbi.2018.10.005","volume":"88","author":"S Velupillai","year":"2018","unstructured":"Velupillai S, Suominen H, Liakata M, Roberts A, Shah AD, Morley K, et al. Using clinical natural language processing for health outcomes research: overview and actionable suggestions for future advances. J Biomed Inform. 2018;88:11\u20139.","journal-title":"J Biomed Inform"},{"issue":"2","key":"671_CR9","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1148\/radiol.2019191586","volume":"293","author":"JR Geis","year":"2019","unstructured":"Geis JR, Brady AP, Wu CC, Spencer J, Ranschaert E, Jaremko JL, et al. Ethics of artificial intelligence in radiology: summary of the Joint European and North American Multisociety Statement. Radiology. 2019;293(2):436\u201340.","journal-title":"Radiology"},{"key":"671_CR10","doi-asserted-by":"publisher","first-page":"m689","DOI":"10.1136\/bmj.m689","volume":"368","author":"M Nagendran","year":"2020","unstructured":"Nagendran M, Chen Y, Lovejoy CA, Gordon AC, Komorowski M, Harvey H, et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ. 2020;368:m689.","journal-title":"BMJ"},{"issue":"2","key":"671_CR11","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1148\/radiol.16142770","volume":"279","author":"E Pons","year":"2016","unstructured":"Pons E, Braun LMM, Hunink MGM, Kors JA. Natural language processing in radiology: a systematic review. Radiology. 2016;279(2):329\u201343.","journal-title":"Radiology"},{"issue":"e1","key":"671_CR12","doi-asserted-by":"publisher","first-page":"e113","DOI":"10.1093\/jamia\/ocv155","volume":"23","author":"J Bates","year":"2016","unstructured":"Bates J, Fodeh SJ, Brandt CA, Womack JA. Classification of radiology reports for falls in an HIV study cohort. J Am Med Inform Assoc JAMIA. 2016;23(e1):e113\u20137.","journal-title":"J Am Med Inform Assoc JAMIA"},{"key":"671_CR13","doi-asserted-by":"crossref","unstructured":"Casey A, Davidson E, Poon M, Dong H, Duma D, Grivas A, et al. A systematic review of natural language processing applied to radiology reports. 2021.","DOI":"10.1186\/s12911-021-01533-7"},{"key":"671_CR14","unstructured":"NLP of radiology reports: systematic review protocol [Internet]. 2020. https:\/\/www.protocols.io\/view\/nlp-of-radiology-reports-systematic-review-protoco-bmwhk7b6."},{"issue":"7","key":"671_CR15","doi-asserted-by":"publisher","first-page":"e100097","DOI":"10.1371\/journal.pmed.1000097","volume":"6","author":"D Moher","year":"2009","unstructured":"Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e100097.","journal-title":"PLoS Med"},{"key":"671_CR16","unstructured":"Harzing.com. Publish or Perish 2020. https:\/\/harzing.com\/resources\/publish-or-perish."},{"issue":"12","key":"671_CR17","doi-asserted-by":"publisher","first-page":"1495","DOI":"10.1016\/j.ijsu.2014.07.013","volume":"12","author":"E von Elm","year":"2014","unstructured":"von Elm E, Altman DG, Egger M, Pocock SJ, G\u00f8tzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495\u20139.","journal-title":"Int J Surg"},{"key":"671_CR18","doi-asserted-by":"crossref","unstructured":"Dodge J, Gururangan S, Card D, Schwartz R, Smith NAJA. Show your work: improved reporting of experimental results. 2019; abs\/1909.03004.","DOI":"10.18653\/v1\/D19-1224"},{"key":"671_CR19","doi-asserted-by":"publisher","first-page":"m363","DOI":"10.1136\/bmj.m363","volume":"368","author":"P Noor","year":"2020","unstructured":"Noor P. Can we trust AI not to further embed racial bias and prejudice? BMJ. 2020;368:m363.","journal-title":"BMJ"},{"issue":"12","key":"671_CR20","doi-asserted-by":"publisher","first-page":"866","DOI":"10.7326\/M18-1990","volume":"169","author":"A Rajkomar","year":"2018","unstructured":"Rajkomar A, Hardt M, Howell MD, Corrado G, Chin MH. Ensuring fairness in machine learning to advance health equity. Ann Intern Med. 2018;169(12):866\u201372.","journal-title":"Ann Intern Med"},{"issue":"2","key":"671_CR21","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1093\/jamia\/ocv080","volume":"23","author":"D Demner-Fushman","year":"2016","unstructured":"Demner-Fushman D, Kohli MD, Rosenman MB, Shooshan SE, Rodriguez L, Antani S, et al. Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc JAMIA. 2016;23(2):304\u201310.","journal-title":"J Am Med Inform Assoc JAMIA"},{"key":"671_CR22","doi-asserted-by":"crossref","unstructured":"Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. 2019.","DOI":"10.1609\/aaai.v33i01.3301590"},{"issue":"1","key":"671_CR23","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/s41597-019-0322-0","volume":"6","author":"AEW Johnson","year":"2019","unstructured":"Johnson AEW, Pollard TJ, Berkowitz SJ, Greenbaum NR, Lungren MP, Deng C, et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci Data. 2019;6(1):317.","journal-title":"Sci Data"},{"issue":"2","key":"671_CR24","doi-asserted-by":"publisher","first-page":"e0212778","DOI":"10.1371\/journal.pone.0212778","volume":"14","author":"C Kim","year":"2019","unstructured":"Kim C, Zhu V, Obeid J, Lenert L. Natural language processing and machine learning algorithm to identify brain MRI reports with acute ischemic stroke. PLoS ONE. 2019;14(2):e0212778.","journal-title":"PLoS ONE"},{"issue":"7829","key":"671_CR25","doi-asserted-by":"publisher","first-page":"E14","DOI":"10.1038\/s41586-020-2766-y","volume":"586","author":"B Haibe-Kains","year":"2020","unstructured":"Haibe-Kains B, Adam GA, Hosny A, Khodakarami F, Shraddha T, Kusko R, et al. Transparency and reproducibility in artificial intelligence. Nature. 2020;586(7829):E14\u20136.","journal-title":"Nature"},{"issue":"7829","key":"671_CR26","doi-asserted-by":"publisher","first-page":"E17","DOI":"10.1038\/s41586-020-2767-x","volume":"586","author":"SM McKinney","year":"2020","unstructured":"McKinney SM, Karthikesalingam A, Tse D, Kelly CJ, Liu Y, Corrado GS, et al. Reply to: Transparency and reproducibility in artificial intelligence. Nature. 2020;586(7829):E17\u20138.","journal-title":"Nature"},{"key":"671_CR27","unstructured":"HDRUK. National Implementation Project: National Text Analytics Resource [cited 2021 18th January]. https:\/\/www.hdruk.org\/projects\/national-text-analytics-project\/."},{"key":"671_CR28","unstructured":"Pineau J, Vincent-Lamarre P, Sinha K, Larivi\u00e8re V, Beygelzimer A, d'Alch\u00e9-Buc F, et al. Improving reproducibility in machine learning research (A Report from the NeurIPS 2019 Reproducibility Program). 2020;abs\/2003.12206."},{"key":"671_CR29","unstructured":"Enhancing the QUAlity and Transparency Of health Research EQUATOR network [cited 2020 4th November]. https:\/\/www.equator-network.org\/."},{"key":"671_CR30","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/1472-6947-13-7","volume":"13","author":"JF Gehanno","year":"2013","unstructured":"Gehanno JF, Rollin L, Darmoni S. Is the coverage of Google Scholar enough to be used alone for systematic reviews. BMC Med Inform Decis Mak. 2013;13:7.","journal-title":"BMC Med Inform Decis Mak"}],"container-title":["BMC Medical Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12880-021-00671-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12880-021-00671-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12880-021-00671-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,4]],"date-time":"2021-10-04T14:58:20Z","timestamp":1633359500000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedimaging.biomedcentral.com\/articles\/10.1186\/s12880-021-00671-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,2]]},"references-count":30,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["671"],"URL":"https:\/\/doi.org\/10.1186\/s12880-021-00671-8","relation":{},"ISSN":["1471-2342"],"issn-type":[{"value":"1471-2342","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,2]]},"assertion":[{"value":"1 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"142"}}