{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T19:08:57Z","timestamp":1790017737522,"version":"4.0.1"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2023,3,11]],"date-time":"2023-03-11T00:00:00Z","timestamp":1678492800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,11]],"date-time":"2023-03-11T00:00:00Z","timestamp":1678492800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Universit\u00e4tsklinikum Bonn"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"abstract":"<jats:title>Abstract\n<\/jats:title><jats:sec>\n                <jats:title>Objectives<\/jats:title>\n                <jats:p>To provide insights for on-site development of transformer-based structuring of free-text report databases by investigating different labeling and pre-training strategies.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>A total of 93,368 German chest X-ray reports from 20,912 intensive care unit (ICU) patients were included. Two labeling strategies were investigated to tag six findings of the attending radiologist. First, a system based on human-defined rules was applied for annotation of all reports (termed \u201csilver labels\u201d). Second, 18,000 reports were manually annotated in 197\u00a0h (termed \u201cgold labels\u201d) of which 10% were used for testing. An on-site pre-trained model (T<jats:sub>mlm<\/jats:sub>) using masked-language modeling (MLM) was compared to a public, medically pre-trained model (T<jats:sub>med<\/jats:sub>). Both models were fine-tuned on silver labels only, gold labels only, and first with silver and then gold labels (hybrid training) for text classification, using varying numbers (<jats:italic>N<\/jats:italic>: 500, 1000, 2000, 3500, 7000, 14,580) of gold labels. Macro-averaged F1-scores (MAF1) in percent were calculated with 95% confidence intervals (CI).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>T<jats:sub>mlm,gold<\/jats:sub> (95.5 [94.5\u201396.3]) showed significantly higher MAF1 than T<jats:sub>med,silver<\/jats:sub> (75.0 [73.4\u201376.5]) and T<jats:sub>mlm,silver<\/jats:sub> (75.2 [73.6\u201376.7]), but not significantly higher MAF1 than T<jats:sub>med,gold<\/jats:sub> (94.7 [93.6\u201395.6]), T<jats:sub>med,hybrid<\/jats:sub> (94.9 [93.9\u201395.8]), and T<jats:sub>mlm,hybrid<\/jats:sub> (95.2 [94.3\u201396.0]). When using 7000 or less gold-labeled reports, T<jats:sub>mlm,gold<\/jats:sub> (<jats:italic>N<\/jats:italic>: 7000, 94.7 [93.5\u201395.7]) showed significantly higher MAF1 than T<jats:sub>med,gold<\/jats:sub> (<jats:italic>N<\/jats:italic>: 7000, 91.5 [90.0\u201392.8]). With at least 2000 gold-labeled reports, utilizing silver labels did not lead to significant improvement of T<jats:sub>mlm,hybrid<\/jats:sub> (<jats:italic>N<\/jats:italic>: 2000, 91.8 [90.4\u201393.2]) over T<jats:sub>mlm,gold<\/jats:sub> (<jats:italic>N<\/jats:italic>: 2000, 91.4 [89.9\u201392.8]).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Custom pre-training of transformers and fine-tuning on manual annotations promises to be an efficient strategy to unlock report databases for data-driven medicine.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Key Points<\/jats:title>\n                <jats:p>\u2022 <jats:italic>On-site development of natural language processing methods that retrospectively unlock free-text databases of radiology clinics for data-driven medicine is of great interest.<\/jats:italic><\/jats:p>\n                <jats:p>\u2022 <jats:italic>For clinics seeking to develop methods on-site for retrospective structuring of a report database of a certain department, it remains unclear which of previously proposed strategies for labeling reports and pre-training models is the most appropriate in context of, e.g., available annotator time.<\/jats:italic><\/jats:p>\n                <jats:p>\u2022 <jats:italic>Using a custom pre-trained transformer model, along with a little annotation effort, promises to be an efficient way to retrospectively structure radiological databases, even if not millions of reports are available for pre-training.<\/jats:italic><\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s00330-023-09526-y","type":"journal-article","created":{"date-parts":[[2023,3,11]],"date-time":"2023-03-11T07:02:55Z","timestamp":1678518175000},"page":"4228-4236","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Transformer-based structuring of free-text radiology report databases"],"prefix":"10.1007","volume":"33","author":[{"given":"S.","family":"Nowak","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"D.","family":"Biesner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Y. C.","family":"Layer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M.","family":"Theis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"H.","family":"Schneider","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"W.","family":"Block","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"B.","family":"Wulff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"U. I.","family":"Attenberger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"R.","family":"Sifa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"A. M.","family":"Sprinkart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,3,11]]},"reference":[{"issue":"1","key":"9526_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13244-019-0831-6","volume":"11","author":"JM Nobel","year":"2020","unstructured":"Nobel JM, Kok EM, Robben SG (2020) Redefining the structure of structured reporting in radiology. Insights Imaging 11(1):1\u20135","journal-title":"Insights Imaging"},{"key":"9526_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13244-017-0588-8","volume":"9","author":"European Society of Radiology (ESR)","year":"2018","unstructured":"European Society of Radiology (ESR) (2018) ESR paper on structured reporting in radiology. Insights Imaging 9:1\u20137","journal-title":"Insights Imaging"},{"issue":"1","key":"9526_CR3","doi-asserted-by":"publisher","first-page":"590","DOI":"10.1609\/aaai.v33i01.3301590","volume":"33","author":"J Irvin","year":"2019","unstructured":"Irvin J, Rajpurkar P, Ko M et al (2019) Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence 33(1):590\u2013597","journal-title":"In Proceedings of the AAAI conference on artificial intelligence"},{"key":"9526_CR4","unstructured":"Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. In Advances in neural information processing systems 30"},{"key":"9526_CR5","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805"},{"key":"9526_CR6","doi-asserted-by":"crossref","unstructured":"Wahab A, Sifa R (2021) Dibert: Dependency injected bidirectional encoder representations from transformers. In 2021 IEEE Symposium Series on Computational Intelligence 1\u20138","DOI":"10.1109\/SSCI50451.2021.9659898"},{"key":"9526_CR7","doi-asserted-by":"crossref","unstructured":"Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren MP (2020) CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. arXiv preprint arXiv:2004.09167","DOI":"10.18653\/v1\/2020.emnlp-main.117"},{"key":"9526_CR8","unstructured":"McDermott MB, Hsu TMH, Wenig WH, Ghassemi M, Szolovits P (2020) Chexpert++: approximating the chexpert labeler for speed, differentiability, and probabilistic output. In Proceedings of the 5th Machine Learning for Healthcare Conference 913\u2013927"},{"issue":"21","key":"9526_CR9","doi-asserted-by":"publisher","first-page":"5255","DOI":"10.1093\/bioinformatics\/btaa668","volume":"36","author":"KK Bressem","year":"2020","unstructured":"Bressem KK, Adams LC, Gaudin RA et al (2020) Highly accurate classification of chest radiographic reports using a deep learning natural language model pre-trained on 3.8 million text reports. Bioinformatics 36(21):5255\u20135261","journal-title":"Bioinformatics"},{"key":"9526_CR10","unstructured":"Richter-Pechanski P, Amr A, Katus HA, Dieterich C (2019) Deep learning approaches outperform conventional strategies in de-identification of German medical reports. In GMDS 101\u2013109"},{"key":"9526_CR11","unstructured":"Carlini N, Tramer F, Wallace E et al (2021) Extracting training data from large language models. In 30th USENIX Security Symposium 2633\u20132650"},{"issue":"2","key":"9526_CR12","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1007\/s41060-021-00285-x","volume":"13","author":"D Biesner","year":"2022","unstructured":"Biesner D, Ramamurthy R, Stenzel R et al (2022) Anonymization of German financial documents using neural network-based language models with contextual word representations. Int J Data Sci Anal 13(2):151\u2013161","journal-title":"Int J Data Sci Anal"},{"key":"9526_CR13","unstructured":"Nakayama H, Kubo T, Kamura J, Taniguchi Y, Liang X (2018) doccano: Text annotation tool for human. Available via https:\/\/github.com\/doccano\/doccano. Accessed 28 Jul 2022"},{"key":"9526_CR14","volume-title":"Modern information retrieval","author":"R Baeza-Yates","year":"1999","unstructured":"Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval, 2nd edn. ACM Press, New York","edition":"2"},{"key":"9526_CR15","doi-asserted-by":"crossref","unstructured":"Wolf T, Debut L, Sanh V et al (2019) Huggingface\u2019s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"9526_CR16","unstructured":"Deepset (2021) German BERT. Available via https:\/\/huggingface.co\/bert-base-german-cased. Accessed 28 Jul 2022"},{"key":"9526_CR17","unstructured":"Shrestha M (2021) German Medical BERT. Available via https:\/\/huggingface.co\/smanjil\/German-MedBERT. Accessed 28 Jul 2022"},{"key":"9526_CR18","doi-asserted-by":"crossref","unstructured":"Sun C, Qiu X, Xu Y, Huang X (2019) How to fine-tune bert for text classification? In 2019 China national conference on Chinese computational linguistics 194\u2013206","DOI":"10.1007\/978-3-030-32381-3_16"},{"issue":"11","key":"9526_CR19","doi-asserted-by":"publisher","first-page":"8807","DOI":"10.1007\/s00330-021-07858-1","volume":"31","author":"S Nowak","year":"2021","unstructured":"Nowak S, Mesropyan N, Faron A et al (2021) Detection of liver cirrhosis in standard T2-weighted MRI using deep transfer learning. Eur Radiol 31(11):8807\u20138815","journal-title":"Eur Radiol"},{"issue":"1","key":"9526_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-022-12410-2","volume":"12","author":"JA Luetkens","year":"2022","unstructured":"Luetkens JA, Nowak S, Mesropyan N et al (2022) Deep learning supports the differentiation of alcoholic and other-than-alcoholic cirrhosis based on MRI. Sci Rep 12(1):1\u20138","journal-title":"Sci Rep"},{"issue":"4","key":"9526_CR21","doi-asserted-by":"publisher","first-page":"1084","DOI":"10.3390\/make1040061","volume":"1","author":"L Gan","year":"2019","unstructured":"Gan L, Yuen B, Lu T (2019) Multi-label classification with optimal thresholding for multi-composition spectroscopic analysis. Mach Learn Knowl Extr 1(4):1084\u20131099","journal-title":"Mach Learn Knowl Extr"},{"key":"9526_CR22","unstructured":"Loshchilov I, Hutter F (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101"},{"key":"9526_CR23","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A et al (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825\u20132830","journal-title":"J Mach Learn Res"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-023-09526-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-023-09526-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-023-09526-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,12]],"date-time":"2023-05-12T12:15:37Z","timestamp":1683893737000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-023-09526-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,11]]},"references-count":23,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["9526"],"URL":"https:\/\/doi.org\/10.1007\/s00330-023-09526-y","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,11]]},"assertion":[{"value":"16 September 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 January 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 February 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 March 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The scientific guarantor of this publication is PD Dr.-Ing. Alois Martin Sprinkart.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"No complex statistical methods were necessary for this paper.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"Written informed consent was waived by the Institutional Review Board (University of Bonn).","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained by the local ethics committees at the Medical Faculty of the Rheinische Friedrich-Wilhelms-Universit\u00e4t Bonn (AZ 411\/21).","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"\u2022 retrospective\u2022 experimental study\u2022 performed at one institution","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}