{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T04:10:51Z","timestamp":1776053451272,"version":"3.50.1"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,11,7]],"date-time":"2023-11-07T00:00:00Z","timestamp":1699315200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,7]],"date-time":"2023-11-07T00:00:00Z","timestamp":1699315200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01|S18038B"],"award-info":[{"award-number":["01|S18038B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01KX2121"],"award-info":[{"award-number":["01KX2121"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Universit\u00e4tsklinikum Bonn"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Objectives<\/jats:title>\n                <jats:p>To investigate the potential and limitations of utilizing transformer-based report annotation for on-site development of image-based diagnostic decision support systems (DDSS).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>The study included 88,353 chest X-rays from 19,581 intensive care unit (ICU) patients. To label the presence of six typical findings in 17,041 images, the corresponding free-text reports of the attending radiologists were assessed by medical research assistants (\u201cgold labels\u201d). Automatically generated \u201csilver\u201d labels were extracted for all reports by transformer models trained on gold labels. To investigate the benefit of such silver labels, the image-based models were trained using three approaches: with gold labels only (M<jats:sub>G<\/jats:sub>), with silver labels first, then with gold labels (M<jats:sub>S\/G<\/jats:sub>), and with silver and gold labels together (M<jats:sub>S+G<\/jats:sub>). To investigate the influence of invested annotation effort, the experiments were repeated with different numbers (<jats:italic>N<\/jats:italic>) of gold-annotated reports for training the transformer and image-based models and tested on 2099 gold-annotated images. Significant differences in macro-averaged area under the receiver operating characteristic curve (AUC) were assessed by non-overlapping 95% confidence intervals.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Utilizing transformer-based silver labels showed significantly higher macro-averaged AUC than training solely with gold labels (<jats:italic>N<\/jats:italic>\u2009=\u20091000: M<jats:sub>G<\/jats:sub> 67.8 [66.0\u201369.6], M<jats:sub>S\/G<\/jats:sub> 77.9 [76.2\u201379.6]; <jats:italic>N<\/jats:italic>\u2009=\u200914,580: M<jats:sub>G<\/jats:sub> 74.5 [72.8\u201376.2], M<jats:sub>S\/G<\/jats:sub> 80.9 [79.4\u201382.4]). Training with silver and gold labels together was beneficial using only 500 gold labels (M<jats:sub>S+G<\/jats:sub> 76.4 [74.7\u201378.0], M<jats:sub>S\/G<\/jats:sub> 75.3 [73.5\u201377.0]).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Transformer-based annotation has potential for unlocking free-text report databases for the development of image-based DDSS. However, on-site development of image-based DDSS could benefit from more sophisticated annotation pipelines including further information than a single radiological report.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Clinical relevance statement<\/jats:title>\n                <jats:p>Leveraging clinical databases for on-site development of artificial intelligence (AI)\u2013based diagnostic decision support systems by text-based transformers could promote the application of AI in clinical practice by circumventing highly regulated data exchanges with third parties.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Key Points<\/jats:title>\n                <jats:p><jats:italic>\u2022 The amount of data from a database that can be used to develop AI-assisted diagnostic decision systems is often limited by the need for time-consuming identification of pathologies by radiologists.<\/jats:italic><\/jats:p>\n                <jats:p><jats:italic>\u2022 The transformer-based structuring of free-text radiological reports shows potential to unlock corresponding image databases for on-site development of image-based diagnostic decision support systems.<\/jats:italic><\/jats:p>\n                <jats:p><jats:italic>\u2022 However, the quality of image annotations generated solely on the content of a single radiology report may be limited by potential inaccuracies and incompleteness of this report.<\/jats:italic><\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s00330-023-10373-0","type":"journal-article","created":{"date-parts":[[2023,11,7]],"date-time":"2023-11-07T06:01:25Z","timestamp":1699336885000},"page":"2895-2904","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Development of image-based decision support systems utilizing information extracted from radiological free-text report databases with text-based transformers"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5175-5559","authenticated-orcid":false,"given":"Sebastian","family":"Nowak","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Helen","family":"Schneider","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yannik C.","family":"Layer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maike","family":"Theis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Biesner","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wolfgang","family":"Block","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Wulff","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ulrike I.","family":"Attenberger","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rafet","family":"Sifa","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alois M.","family":"Sprinkart","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,11,7]]},"reference":[{"issue":"8","key":"10373_CR1","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1038\/s41568-018-0016-5","volume":"18","author":"A Hosny","year":"2018","unstructured":"Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJ (2018) Artificial intelligence in radiology. Nat Rev Cancer 18(8):500\u2013510","journal-title":"Nat Rev Cancer"},{"issue":"8","key":"10373_CR2","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1097\/RLI.0000000000000771","volume":"56","author":"SM Niehues","year":"2021","unstructured":"Niehues SM, Adams LC, Gaudin RA et al (2021) Deep-learning-based diagnosis of bedside chest X-ray in intensive care and emergency medicine. Invest Radiol 56(8):525\u2013534","journal-title":"Invest Radiol"},{"issue":"6","key":"10373_CR3","doi-asserted-by":"publisher","first-page":"1445","DOI":"10.2214\/AJR.19.21872","volume":"214","author":"VL Mango","year":"2020","unstructured":"Mango VL, Sun M, Wynn RT, Ha R (2020) Should we ignore, follow, or biopsy? Impact of artificial intelligence decision support on breast ultrasound lesion assessment. AJR Am J Roentgenol 214(6):1445\u20131452","journal-title":"AJR Am J Roentgenol"},{"key":"10373_CR4","doi-asserted-by":"publisher","first-page":"101","DOI":"10.3233\/SHTI190813","volume":"267","author":"P Richter-Pechanski","year":"2019","unstructured":"Richter-Pechanski P, Amr A, Katus HA, Dieterich C (2019) Deep learning approaches outperform conventional strategies in de-identification of German medical reports. Stud Health Technol Inform 267:101\u2013109. https:\/\/doi.org\/10.3233\/SHTI190813","journal-title":"Stud Health Technol Inform"},{"key":"10373_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13244-019-0831-6","volume":"11","author":"JM Nobel","year":"2020","unstructured":"Nobel JM, Kok EM, Robben SG (2020) Redefining the structure of structured reporting in radiology. Insights Imaging 11:1\u20135","journal-title":"Insights Imaging"},{"key":"10373_CR6","doi-asserted-by":"crossref","unstructured":"Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren MP (2020) CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. arXiv preprint arXiv:2004.09167","DOI":"10.18653\/v1\/2020.emnlp-main.117"},{"key":"10373_CR7","first-page":"913","volume":"126","author":"MB McDermott","year":"2020","unstructured":"McDermott MB, Hsu TMH, Wenig WH, Ghassemi M, Szolovits P (2020) Chexpert++: Approximating the chexpert labeler for speed, differentiability, and probabilistic output. Proceedings of PMLR 126:913\u2013927","journal-title":"Proceedings of PMLR"},{"issue":"21","key":"10373_CR8","doi-asserted-by":"publisher","first-page":"5255","DOI":"10.1093\/bioinformatics\/btaa668","volume":"36","author":"KK Bressem","year":"2020","unstructured":"Bressem KK, Adams LC, Gaudin RA et al (2020) Highly accurate classification of chest radiographic reports using a deep learning natural language model pre-trained on 3.8 million text reports. Bioinformatics 36(21):5255\u20135261","journal-title":"Bioinformatics"},{"key":"10373_CR9","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-023-09526-y","author":"S Nowak","year":"2023","unstructured":"Nowak S, Biesner D, Layer YC et al (2023) Transformer-based structuring of free-text radiology report databases. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-023-09526-y","journal-title":"Eur Radiol"},{"key":"10373_CR10","unstructured":"Jain S, Smit A, Ng AY, Rajpurkar P (2021) Effect of radiology report labeler quality on deep learning models for chest X-ray interpretation. arXiv preprint arXiv:2104.00793"},{"key":"10373_CR11","doi-asserted-by":"crossref","unstructured":"Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In proceedings of MICCAI 2015 18:234-241","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"10373_CR12","doi-asserted-by":"crossref","unstructured":"Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In proceedings of CVPR 2017 4700-4708","DOI":"10.1109\/CVPR.2017.243"},{"issue":"1","key":"10373_CR13","first-page":"590","volume":"33","author":"J Irvin","year":"2019","unstructured":"Irvin J, Rajpurkar P, Ko M et al (2019) Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Proc AAAI Conf Artif Intell 33(1):590\u2013597","journal-title":"Proc AAAI Conf Artif Intell"},{"key":"10373_CR14","unstructured":"Smith LN (2018) A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820"},{"issue":"11","key":"10373_CR15","doi-asserted-by":"publisher","first-page":"8807","DOI":"10.1007\/s00330-021-07858-1","volume":"31","author":"S Nowak","year":"2021","unstructured":"Nowak S, Mesropyan N, Faron A et al (2021) Detection of liver cirrhosis in standard T2-weighted MRI using deep transfer learning. Eur Radiol 31(11):8807\u20138815","journal-title":"Eur Radiol"},{"issue":"1","key":"10373_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-022-12410-2","volume":"12","author":"JA Luetkens","year":"2022","unstructured":"Luetkens JA, Nowak S, Mesropyan N et al (2022) Deep learning supports the differentiation of alcoholic and other-than-alcoholic cirrhosis based on MRI. Sci Rep 12(1):1\u20138","journal-title":"Sci Rep"},{"issue":"2","key":"10373_CR17","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1002\/sim.3471","volume":"28","author":"G Cumming","year":"2009","unstructured":"Cumming G (2009) Inference by eye: Reading the overlap of independent confidence intervals. Stat Med 28(2):205\u2013220","journal-title":"Stat Med"},{"issue":"4","key":"10373_CR18","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1177\/1751143715580141","volume":"16","author":"R Spiritoso","year":"2015","unstructured":"Spiritoso R, Padley S, Singh S (2015) Chest X-ray interpretation in UK intensive care units: A survey 2014. J Intensive Care Soc 16(4):339\u2013344","journal-title":"J Intensive Care Soc"},{"key":"10373_CR19","doi-asserted-by":"crossref","unstructured":"Jain S, Smit A, Truong SQ et al (2021) VisualCheXbert: addressing the discrepancy between radiology report labels and image labels. In Proceedings of CHIL 2021 105-115","DOI":"10.1145\/3450439.3451862"},{"key":"10373_CR20","unstructured":"Thulasidasan S, Bhattacharya T, Bilmes J, Chennupati G, Mohd-Yusof J (2019) Combating label noise in deep learning using abstention. arXiv preprint arXiv:1905.10964"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-023-10373-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-023-10373-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-023-10373-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,24]],"date-time":"2024-05-24T18:10:53Z","timestamp":1716574253000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-023-10373-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,7]]},"references-count":20,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["10373"],"URL":"https:\/\/doi.org\/10.1007\/s00330-023-10373-0","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,7]]},"assertion":[{"value":"12 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 August 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 September 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 November 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The scientific guarantor of this publication is PD Dr.-Ing. Alois Martin Sprinkart.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"No complex statistical methods were necessary for this paper.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"Written informed consent was waived by the Institutional Review Board (University of Bonn).","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained by the local Ethics Committees at the Medical Faculty of the Rheinische Friedrich-Wilhelms-Universit\u00e4t Bonn (AZ 411\/21).","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"The present study is a follow-up to a study published in European Radiology on the transformer-based structuring of free-text radiology databases (Nowak, S., Biesner, D., Layer, Y.C. et al Eur Radiol (2023). ). The manual and transformer-based report annotations generated in the previous study were used in the current study to annotate the corresponding ICU radiographs to investigate the value of these report-based annotations for the development of image-based DDSS systems.","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Study subjects or cohorts overlap"}},{"value":"\u2022 retrospective\u2022 experimental study\u2022 performed at one institution","order":8,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}