{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,7]],"date-time":"2025-11-07T09:40:03Z","timestamp":1762508403090,"version":"3.37.3"},"reference-count":24,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,3,4]],"date-time":"2021-03-04T00:00:00Z","timestamp":1614816000000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2021,3,4]],"date-time":"2021-03-04T00:00:00Z","timestamp":1614816000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005416","name":"Norges Forskningsr\u00e5d","doi-asserted-by":"publisher","award":["259055"],"award-info":[{"award-number":["259055"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>With a motivation of quality assurance, machine learning techniques were trained to classify Norwegian radiology reports of paediatric CT examinations according to their description of abnormal findings.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>13.506 reports from CT-scans of children, 1000 reports from CT scan of adults and 1000 reports from X-ray examination of adults were classified as positive or negative by a radiologist, according to the presence of abnormal findings. Inter-rater reliability was evaluated by comparison with a clinician\u2019s classifications of 500 reports. Test\u2013retest reliability of the radiologist was performed on the same 500 reports. A convolutional neural network model (CNN), a bidirectional recurrent neural network model (bi-LSTM) and a support vector machine model (SVM) were trained on a random selection of the children\u2019s data set. Models were evaluated on the remaining CT-children reports and the adult data sets.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Test\u2013retest reliability: Cohen\u2019s Kappa\u2009=\u20090.86 and F1\u2009=\u20090.919. Inter-rater reliability: Kappa\u2009=\u20090.80 and F1\u2009=\u20090.885. Model performances on the Children-CT data were as follows. CNN: (AUC\u2009=\u20090.981, F1\u2009=\u20090.930), bi-LSTM: (AUC\u2009=\u20090.978, F1\u2009=\u20090.927), SVM: (AUC\u2009=\u20090.975, F1\u2009=\u20090.912). On the adult data sets, the models had AUC around 0.95 and F1 around 0.91.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>The models performed close to perfectly on its defined domain, and also performed convincingly on reports pertaining to a different patient group and a different modality. The models were deemed suitable for classifying radiology reports for future quality assurance purposes, where the fraction of the examinations with abnormal findings for different sub-groups of patients is a parameter of interest.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-021-01451-8","type":"journal-article","created":{"date-parts":[[2021,3,4]],"date-time":"2021-03-04T17:02:44Z","timestamp":1614877364000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Neural classification of Norwegian radiology reports: using NLP to detect findings in CT-scans of children"],"prefix":"10.1186","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6835-0985","authenticated-orcid":false,"given":"Fredrik A.","family":"Dahl","sequence":"first","affiliation":[]},{"given":"Taraka","family":"Rama","sequence":"additional","affiliation":[]},{"given":"Petter","family":"Hurlen","sequence":"additional","affiliation":[]},{"given":"P\u00e5l H.","family":"Brekke","sequence":"additional","affiliation":[]},{"given":"Haldor","family":"Husby","sequence":"additional","affiliation":[]},{"given":"Tore","family":"Gundersen","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8163-2362","authenticated-orcid":false,"given":"\u00d8ystein","family":"Nytr\u00f8","sequence":"additional","affiliation":[]},{"given":"Lilja","family":"\u00d8vrelid","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,3,4]]},"reference":[{"unstructured":"Oatway WB, Jones AL, Holmes S, Watson S, Cabianca T. Ionising Radiation Exposure of the UK Population: 2010 Review. Public Health of England. 2010.","key":"1451_CR1"},{"doi-asserted-by":"crossref","unstructured":"Mathews JD, Forsythe AV, Brady Z, et al. Cancer risk in 680,000 people exposed to computed tomography scans in childhood or adolescence: data linkage study of 11 million Australians. BMJ. (Clinical research ed.), 2013; 346.","key":"1451_CR2","DOI":"10.1136\/bmj.f2360"},{"issue":"9","key":"1451_CR3","doi-asserted-by":"publisher","first-page":"2354","DOI":"10.1038\/bjc.2014.103","volume":"110","author":"WY Huang","year":"2014","unstructured":"Huang WY, Muo CH, Lin CY, et al. Paediatric head CT scan and subsequent risk of malignancy and benign brain tumour: a nation-wide population-based cohort study. Br J Cancer. 2014;110(9):2354\u201360.","journal-title":"Br J Cancer"},{"issue":"3","key":"1451_CR4","doi-asserted-by":"publisher","first-page":"256","DOI":"10.1093\/jnci\/djy104","volume":"111","author":"JM Meulepas","year":"2018","unstructured":"Meulepas JM, Ronckers CM, Smets AMJB, et al. Radiation exposure from pediatric CT scans and subsequent cancer risk in the Netherlands. J Natl Cancer Inst. 2018;111(3):256\u201363.","journal-title":"J Natl Cancer Inst"},{"doi-asserted-by":"crossref","unstructured":"Lumbreras B, Salinas JM, Gonzalez-Alvarez I. Cumulative exposure to ionising radiation from diagnostic imaging tests: a 12-year follow-up population-based analysis in Spain. BMJ Open. 2019;18;9(9).","key":"1451_CR5","DOI":"10.1136\/bmjopen-2019-030905"},{"issue":"Suppl 3","key":"1451_CR6","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1007\/s00247-014-3122-x","volume":"44","author":"K Frush","year":"2014","unstructured":"Frush K. Why and when to use CT in children: perspective of a pediatric emergency medicine physician. Pediatr Radiol. 2014;44(Suppl 3):409\u201313.","journal-title":"Pediatr Radiol"},{"issue":"6","key":"1451_CR7","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1007\/s10140-015-1332-7","volume":"22","author":"M Muhm","year":"2015","unstructured":"Muhm M, Danko T, Henzler T, Luiz T, Winkler H, Ruffing T. Pediatric trauma care with computed tomography\u2013criteria for CT scanning. Emerg Radiol. 2015;22(6):613\u201321.","journal-title":"Emerg Radiol"},{"unstructured":"Perrault R, Shoham Y, Brynjolfsson E, et al. \u201cThe AI Index 2019 Annual Report\u201d, AI Index Steering Committee, Human-Centered AI Institute, Stanford University, Stanford, CA, December 2019","key":"1451_CR8"},{"issue":"5","key":"1451_CR9","doi-asserted-by":"publisher","first-page":"760","DOI":"10.1016\/j.jbi.2009.08.007","volume":"42","author":"D Demner-Fushman","year":"2009","unstructured":"Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform. 2009;42(5):760\u201372.","journal-title":"J Biomed Inform"},{"issue":"2","key":"1451_CR10","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1136\/jamia.1994.95236146","volume":"1","author":"C Friedman","year":"1994","unstructured":"Friedman C, Alderson PO, Austin JHM, Cimino JJ, Johnson SB. A general natural-language text processor for clinical radiology. J Am Med Inform Assoc. 1994;1(2):161\u201374.","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"1451_CR11","doi-asserted-by":"publisher","first-page":"313","DOI":"10.2214\/AJR.07.3508","volume":"191","author":"PA Dang","year":"2008","unstructured":"Dang PA, Kalra MK, Blake MA, Schultz TJ, Halpern EF, Dreyer KJ. Extraction of recommendation features in radiology with natural language processing: exploratory study. Am J Roentgenol. 2008;191(2):313\u201320.","journal-title":"Am J Roentgenol"},{"issue":"6","key":"1451_CR12","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1136\/jamia.2000.0070593","volume":"7","author":"M Fiszman","year":"2000","unstructured":"Fiszman M, Chapman WW, Aronsky D, Scott Evans R, Haug PJ. Automatic detection of acute bacterial pneumonia from chest x-ray reports. J Am Med Inform Assoc. 2000;7(6):593\u2013604.","journal-title":"J Am Med Inform Assoc"},{"unstructured":"Elkin PL, Froehling D, Wahner-Roedler D, et al. NLP-based identification of pneumonia cases from free-text radiological reports. AMIA Annual Symposium proceedings, 2008; 172\u2013176.","key":"1451_CR13"},{"issue":"5","key":"1451_CR14","doi-asserted-by":"publisher","first-page":"728","DOI":"10.1016\/j.jbi.2011.03.011","volume":"44","author":"BE Chapman","year":"2011","unstructured":"Chapman BE, Lee S, Kang HP, Chapman WW. Document-level classification of CT pulmonary angiography reports based on an extension of the ConText algorithm. J Biomed Inform. 2011;44(5):728\u201337.","journal-title":"J Biomed Inform"},{"issue":"3","key":"1451_CR15","doi-asserted-by":"publisher","first-page":"845","DOI":"10.1148\/radiol.2017171115","volume":"286","author":"MC Chen","year":"2018","unstructured":"Chen MC, Ball RL, Yang L, et al. Deep learning to classify radiology free-text reports. Radiology. 2018;286(3):845\u201352.","journal-title":"Radiology"},{"doi-asserted-by":"crossref","unstructured":"Lee C, Kim Y, Kim YS, Jang J. Automatic disease annotation from radiology reports using artificial intelligence implemented by a recurrent neural network. Am J Roentgenol. 2019; 212: 734\u201340.","key":"1451_CR16","DOI":"10.2214\/AJR.18.19869"},{"issue":"1","key":"1451_CR17","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1186\/s13326-018-0179-8","volume":"9","author":"A N\u00e9v\u00e9ol","year":"2018","unstructured":"N\u00e9v\u00e9ol A, Dalianis H, Velupillai S, Savova G, Zweigenbaum P. Clinical natural language processing in languages other than english: opportunities and challenges. J Biomed Semant. 2018;9(1):12.","journal-title":"J Biomed Semant"},{"key":"1451_CR18","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-4541-9","volume-title":"An introduction to the bootstrap","author":"B Efron","year":"1993","unstructured":"Efron B, Tibshirani R. An introduction to the bootstrap. New York: Chapman & Hall; 1993."},{"unstructured":"Van Rijsbergen CJ. Information Retrieval (2nd ed.). Butterworth-Heinemann. 1979.","key":"1451_CR19"},{"unstructured":"\u00d8vrelid L, Hohle P. Universal Dependencies for Norwegian. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association. 2016; 1579 \u2013 1585.","key":"1451_CR20"},{"key":"1451_CR21","doi-asserted-by":"publisher","first-page":"981","DOI":"10.3390\/app8060981","volume":"8","author":"V Menger","year":"2018","unstructured":"Menger V, Scheepers F, Spruit M. Comparing deep learning and classical machine learning approaches for predicting inpatient violence incidents from clinical text. Appl Sci. 2018;8:981.","journal-title":"Appl Sci"},{"issue":"8","key":"1451_CR22","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735\u201380.","journal-title":"Neural Comput"},{"unstructured":"Zhou P, Qi Z, Zheng S, Xu J, Bao H, Xu B. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. arXiv preprint 1611.06639, 2016.","key":"1451_CR23"},{"doi-asserted-by":"crossref","unstructured":"Kim Y. Convolutional neural networks for sentence classification. arXiv preprint 1408.5882, 2014.","key":"1451_CR24","DOI":"10.3115\/v1\/D14-1181"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01451-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-021-01451-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01451-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,4]],"date-time":"2021-03-04T17:08:34Z","timestamp":1614877714000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-021-01451-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,4]]},"references-count":24,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1451"],"URL":"https:\/\/doi.org\/10.1186\/s12911-021-01451-8","relation":{},"ISSN":["1472-6947"],"issn-type":[{"type":"electronic","value":"1472-6947"}],"subject":[],"published":{"date-parts":[[2021,3,4]]},"assertion":[{"value":"3 June 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 February 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 March 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declaration"}},{"value":"The study was approved as a quality assurance project by the hospital's Privacy Ombudsman (Approval number 2016_113). Under Norwegian law (Helsepersonelloven \u00a726), this approval gives exemption from consent by the participants.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"84"}}