{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,23]],"date-time":"2025-10-23T21:05:42Z","timestamp":1761253542713,"version":"build-2065373602"},"reference-count":33,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,3,24]],"date-time":"2021-03-24T00:00:00Z","timestamp":1616544000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Training medical image analysis models traditionally requires large amounts of expertly annotated imaging data which is time-consuming and expensive to obtain. One solution is to automatically extract scan-level labels from radiology reports. Previously, we showed that, by extending BERT with a per-label attention mechanism, we can train a single model to perform automatic extraction of many labels in parallel. However, if we rely on pure data-driven learning, the model sometimes fails to learn critical features or learns the correct answer via simplistic heuristics (e.g., that \u201clikely\u201d indicates positivity), and thus fails to generalise to rarer cases which have not been learned or where the heuristics break down (e.g., \u201clikely represents prominent VR space or lacunar infarct\u201d which indicates uncertainty over two differential diagnoses). In this work, we propose template creation for data synthesis, which enables us to inject expert knowledge about unseen entities from medical ontologies, and to teach the model rules on how to label difficult cases, by producing relevant training examples. Using this technique alongside domain-specific pre-training for our underlying BERT architecture i.e., PubMedBERT, we improve F1 micro from 0.903 to 0.939 and F1 macro from 0.512 to 0.737 on an independent test set for 33 labels in head CT reports for stroke patients. Our methodology offers a practical way to combine domain knowledge with machine learning for text classification tasks.<\/jats:p>","DOI":"10.3390\/make3020015","type":"journal-article","created":{"date-parts":[[2021,3,24]],"date-time":"2021-03-24T21:36:51Z","timestamp":1616621811000},"page":"299-317","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Templated Text Synthesis for Expert-Guided Multi-Label Extraction from Radiology Reports"],"prefix":"10.3390","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2484-6855","authenticated-orcid":false,"given":"Patrick","family":"Schrempf","sequence":"first","affiliation":[{"name":"Canon Medical Research Europe, Edinburgh EH6 5NP, UK"},{"name":"School of Computer Science, University of St Andrews, St Andrews KY16 9SX, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2728-0386","authenticated-orcid":false,"given":"Hannah","family":"Watson","sequence":"additional","affiliation":[{"name":"Canon Medical Research Europe, Edinburgh EH6 5NP, UK"}]},{"given":"Eunsoo","family":"Park","sequence":"additional","affiliation":[{"name":"Canon Medical Research Europe, Edinburgh EH6 5NP, UK"}]},{"given":"Maciej","family":"Pajak","sequence":"additional","affiliation":[{"name":"Canon Medical Research Europe, Edinburgh EH6 5NP, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6796-6905","authenticated-orcid":false,"given":"Hamish","family":"MacKinnon","sequence":"additional","affiliation":[{"name":"Canon Medical Research Europe, Edinburgh EH6 5NP, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9535-022X","authenticated-orcid":false,"given":"Keith W.","family":"Muir","sequence":"additional","affiliation":[{"name":"Institute of Neuroscience &amp; Psychology, University of Glasgow, Glasgow G12 8QB, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0740-3668","authenticated-orcid":false,"given":"David","family":"Harris-Birtill","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of St Andrews, St Andrews KY16 9SX, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8371-0603","authenticated-orcid":false,"given":"Alison Q.","family":"O\u2019Neil","sequence":"additional","affiliation":[{"name":"Canon Medical Research Europe, Edinburgh EH6 5NP, UK"},{"name":"School of Engineering, University of Edinburgh, Edinburgh EH9 3JL, UK"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,24]]},"reference":[{"key":"ref_1","unstructured":"Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., and Shpanskaya, K. (February, January 27). Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_2","unstructured":"Radiological Society of North America (2020, November 01). RSNA Intracranial Hemorrhage Detection (Kaggle Challenge). Available online: https:\/\/www.kaggle.com\/c\/rsna-intracranial-hemorrhage-detection\/overview."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Cardoso, J., Van Nguyen, H., Heller, N., Henriques Abreu, P., Isgum, I., Silva, W., Cruz, R., Pereira Amorim, J., Patel, V., and Roysam, B. (2020). Paying Per-Label Attention for Multi-label Extraction from Radiology Reports. Interpretable and Annotation-Efficient Learning for Medical Image Computing, Springer International Publishing.","DOI":"10.1007\/978-3-030-61166-8_30"},{"key":"ref_4","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 3\u20135). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA."},{"key":"ref_5","first-page":"1101","article-title":"Explainable Prediction of Medical Codes from Clinical Text","volume":"Volume 1","author":"Mullenbach","year":"2018","journal-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Cardoso, J., Van Nguyen, H., Heller, N., Henriques Abreu, P., Isgum, I., Silva, W., Cruz, R., Pereira Amorim, J., Patel, V., and Roysam, B. (2020). Labelling Imaging Datasets on the Basis of Neuroradiology Reports: A Validation Study. Interpretable and Annotation-Efficient Learning for Medical Image Computing, Springer International Publishing.","DOI":"10.1007\/978-3-030-61166-8_30"},{"key":"ref_7","unstructured":"McCoy, T., Pavlick, E., and Linzen, T. (August, January 28). Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"267D","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): Integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. (2020). Domain-specific language model pretraining for biomedical natural language processing. arXiv.","DOI":"10.1145\/3458754"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1016\/j.jbi.2012.12.005","article-title":"A text processing pipeline to extract recommendations from radiology reports","volume":"46","author":"Gunn","year":"2013","journal-title":"J. Biomed. Inform."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Grivas, A., Alex, B., Grover, C., Tobin, R., and Whiteley, W. (2020, January 16\u201320). Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports. Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.louhi-1.4"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1148\/radiol.2018171093","article-title":"Natural language\u2013based machine learning models for the annotation of clinical radiology reports","volume":"287","author":"Zech","year":"2018","journal-title":"Radiology"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1111\/acem.12859","article-title":"Automated Outcome Classification of Computed Tomography Imaging Reports for Pediatric Traumatic Brain Injury","volume":"23","author":"Yadav","year":"2016","journal-title":"Acad. Emerg. Med."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E. (2016, January 12\u201317). Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.","DOI":"10.18653\/v1\/N16-1174"},{"key":"ref_15","unstructured":"Banerjee, S., Akkaya, C., Perez-Sorrosal, F., and Tsioutsiouliklis, K. (August, January 28). Hierarchical Transfer Learning for Multi-label Text Classification. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Drozdov, I., Forbes, D., Szubert, B., Hall, M., Carlin, C., and Lowe, D.J. (2020). Supervised and unsupervised language modelling in Chest X-ray radiological reports. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0229963"},{"key":"ref_17","unstructured":"Wood, D., Guilhem, E., Montvila, A., Varsavsky, T., Kiik, M., Siddiqui, J., Kafiabadi, S., Gadapa, N., Busaidi, A.A., and Townend, M. Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM). Proceedings of the Third Conference on Medical Imaging with Deep Learning; Montr\u00e9al, QC, Canada, 6\u20139 July 2020; Proceedings of Machine Learning Research, Montr\u00e9al, QC, Canada, 2020."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A.Y., and Lungren, M.P. (2020, January 16\u201320). CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.117"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Falis, M., Pajak, M., Lisowska, A., Schrempf, P., Deckers, L., Mikhael, S., Tsaftaris, S., and O\u2019Neil, A. (2019, January 3). Ontological attention ensembles for capturing semantic concepts in ICD code prediction from clinical text. Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), Hong Kong, China.","DOI":"10.18653\/v1\/D19-6220"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Alsentzer, E., Murphy, J., Boag, W., Weng, W.H., Jindi, D., Naumann, T., and McDermott, M. (, January 6\u20137). Publicly Available Clinical BERT Embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, MN, USA.","DOI":"10.18653\/v1\/W19-1909"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wei, J., and Zou, K. (2019, January 3\u20137). EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1670"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kryscinski, W., McCann, B., Xiong, C., and Socher, R. Evaluating the Factual Consistency of Abstractive Text Summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP); Online, 16\u201320 November 2020.","DOI":"10.18653\/v1\/2020.emnlp-main.750"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Michalowski, M., and Moskovitch, R. (2020). Divide to Better Classify. Artificial Intelligence in Medicine, Springer International Publishing.","DOI":"10.1007\/978-3-030-59137-3"},{"key":"ref_24","first-page":"881","article-title":"Paraphrasing Revisited with Neural Machine Translation","volume":"Volume 1","author":"Mallinson","year":"2017","journal-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics"},{"key":"ref_25","first-page":"1875","article-title":"Adversarial Example Generation with Syntactically Controlled Paraphrase Networks","volume":"Volume 1","author":"Iyyer","year":"2018","journal-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"},{"key":"ref_26","unstructured":"Appelgren, M., Schrempf, P., Falis, M., Ikeda, S., and O\u2019Neil, A.Q. (2019). Language Transfer for Early Warning of Epidemics from Social Media. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"IST-3 Collaborative Group (2015). Association between brain imaging signs, early and late outcomes, and response to intravenous alteplase after acute ischaemic stroke in the third International Stroke Trial (IST-3): Secondary analysis of a randomised controlled trial. Lancet Neurol., 14, 485\u2013496.","DOI":"10.1016\/S1474-4422(15)00012-5"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Loper, E., and Bird, S. (2002, January 6\u20137). NLTK: The Natural Language Toolkit. Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), Philadelphia, PA, USA.","DOI":"10.3115\/1118108.1118117"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Manning, C.D., Raghavan, P., and Sch\u00fctze, H. (2008). Introduction to Information Retrieval, Cambridge University Press.","DOI":"10.1017\/CBO9780511809071"},{"key":"ref_30","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR, Conference Track Proceedings, San Diego, CA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2019). HuggingFace\u2019s Transformers: State-of-the-art Natural Language Processing. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"ref_32","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7\u20139). Neural Machine Translation by Jointly Learning to Align and Translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR, Conference Track Proceedings, San Diego, CA, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13326-019-0211-7","article-title":"Text mining brain imaging reports","volume":"10","author":"Alex","year":"2019","journal-title":"J. Biomed. Semant."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/2\/15\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:40:33Z","timestamp":1760161233000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/2\/15"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,24]]},"references-count":33,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["make3020015"],"URL":"https:\/\/doi.org\/10.3390\/make3020015","relation":{},"ISSN":["2504-4990"],"issn-type":[{"type":"electronic","value":"2504-4990"}],"subject":[],"published":{"date-parts":[[2021,3,24]]}}}