{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T11:48:18Z","timestamp":1782906498644,"version":"3.54.5"},"reference-count":19,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T00:00:00Z","timestamp":1779840000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T00:00:00Z","timestamp":1779840000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol Exp"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To compare open-weight and proprietary large language models (LLMs),\na rule-based extractor (RBE) and radiologists for labelling pulmonary embolism\nCT reports, and to test whether a hybrid RBE\u2013LLM workflow improves\nlabelling performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>This single-centre retrospective study included structured CT\nreports from October 2021 to March 2025. Three labelling pipelines were\nevaluated: an RBE; a model-agnostic LLM extractor (18 open-weight, four GPT-4\nvariants); and a hybrid pipeline routing only RBE failures to an LLM. Ground\ntruth was defined at the report-text level by deterministic schema matching for\ninitially RBE-valid fields and blinded adjudication of RBE-invalid fields by two\nattending radiologists. Eight radiologists provided a human baseline. Outcomes\nincluded F1 scores, accuracy, LLM-based salvage of RBE failures, and labelling\ntime.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      In total, 2,923 reports from 2,923 patients (mean age 66\u2009\u00b1\u200917 years;\n1,465 women) were included. Falcon3-10b and GPT-4.1-mini achieved similar\nitem-level performance (F1 0.98 [95% CI, 0.97\u20130.98] for both;\n                      <jats:italic>p<\/jats:italic>\n                      \u2009=\u20090.70) and both exceeded the RBE (F1 0.81 [95%\nCI, 0.80\u20130.82];\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.001).\nSalvage of RBE failures was comparable between open-weight and proprietary\nmodels (88.1%\n                      <jats:italic>vs<\/jats:italic>\n                      91.9%;\n                      <jats:italic>p<\/jats:italic>\n                      \u2009=\u20090.12). The hybrid RBE\u2013LLM workflow\nachieved 99.8% accuracy and F1 0.99 (0.98\u20130.99), exceeding both the RBE\nand pooled radiologists (F1 0.92 [95% CI, 0.90\u20130.93]; all\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.001).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Schema-constrained open-weight and proprietary LLMs exceeded\nrule-based extraction and, at the upper end of performance, matched a pooled\nradiologist label-transfer baseline. A rules-first, targeted LLM workflow\nenabled near-perfect extraction from finalised structured pulmonary embolism CT\nreports.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Relevance statement<\/jats:title>\n                    <jats:p>A rules-first LLM workflow can automate high-fidelity extraction of\nstructured CT findings from finalised radiology reports, enabling scalable,\nauditable, and more consistent cohort curation for clinical research,\nregistries, and quality improvement.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Key Points<\/jats:title>\n                    <jats:p>\n                      <jats:list list-type=\"bullet\">\n                        <jats:list-item>\n                          <jats:p>A hybrid rules-first workflow combining a rule-based\nextractor (RBE) with targeted large language model (LLM) salvage\nachieved the highest overall performance for labelling of pulmonary\nembolism CT reports (F1, 0.99; accuracy, 99.8%).<\/jats:p>\n                        <\/jats:list-item>\n                        <jats:list-item>\n                          <jats:p>The top standalone open-weight and proprietary LLMs\n(Falcon3-10b and GPT-4.1-mini) both exceeded the RBE and, at the\nupper end of performance, matched a pooled radiologist\nlabel-transfer baseline.<\/jats:p>\n                        <\/jats:list-item>\n                        <jats:list-item>\n                          <jats:p>The hybrid workflow reduced cohort-curation time from\n32.2\u2009h for radiologists to 1.0\u2009h while reducing LLM calls by 85.6%,\nbecause the LLM was only triggered for rule-failed fields.<\/jats:p>\n                        <\/jats:list-item>\n                      <\/jats:list>\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s41747-026-00738-7","type":"journal-article","created":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T08:59:57Z","timestamp":1779872397000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Who labels best? Radiologists, rules, or large language models\nfor CT reports on pulmonary embolism"],"prefix":"10.1186","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0189-7070","authenticated-orcid":false,"given":"Matthias A.","family":"Fink","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arved","family":"Bischoff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edem","family":"Atsiatorme","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Kremer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonas","family":"Kroschke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Moll","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick","family":"Stein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Veronika","family":"Riebl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Timo","family":"Leichenich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hans-Ulrich","family":"Kauczor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kai","family":"Schlamp","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,27]]},"reference":[{"key":"738_CR1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2024.102814","volume":"149","author":"S Pereira","year":"2024","unstructured":"Pereira S, Mendon\u00e7a AM, Campilho A, Sousa P, Teixeira Lopes C\n(2024) Automated image label extraction from radiology reports\u2014a review. Artif\nIntell Med 149:102814. https:\/\/doi.org\/10.1016\/j.artmed.2024.102814","journal-title":"Artif Intell Med"},{"key":"738_CR2","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1148\/radiol.2021211013","volume":"302","author":"MA Fink","year":"2022","unstructured":"Fink MA, Mayer VL, Schneider T et al (2022) CT angiography clot\nburden score from data mining of structured reports for pulmonary embolism.\nRadiology 302:175\u2013184. https:\/\/doi.org\/10.1148\/radiol.2021211013","journal-title":"Radiology"},{"key":"738_CR3","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-021-01533-7","volume":"21","author":"A Casey","year":"2021","unstructured":"Casey A, Davidson E, Poon M et al (2021) A systematic review of\nnatural language processing applied to radiology reports. BMC Med Inform Decis\nMak 21:179. https:\/\/doi.org\/10.1186\/s12911-021-01533-7","journal-title":"BMC Med Inform Decis Mak"},{"key":"738_CR4","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1038\/s41746-023-00773-3","volume":"6","author":"A Sylolypavan","year":"2023","unstructured":"Sylolypavan A, Sleeman D, Wu H, Sim M (2023) The impact of\ninconsistent human annotations on AI driven clinical decision making. NPJ Digit\nMed 6:26. https:\/\/doi.org\/10.1038\/s41746-023-00773-3","journal-title":"NPJ Digit Med"},{"key":"738_CR5","doi-asserted-by":"publisher","unstructured":"Lee RC, Hadidchi R, Coard MC et al (2025) Use of large language\nmodels on radiology reports: a scoping review. J Am Coll Radiol. https:\/\/doi.org\/10.1016\/j.jacr.2025.10.005","DOI":"10.1016\/j.jacr.2025.10.005"},{"key":"738_CR6","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1038\/s41746-024-01219-0","volume":"7","author":"D Reichenpfader","year":"2024","unstructured":"Reichenpfader D, M\u00fcller H, Denecke K (2024) A scoping review of\nlarge language model based approaches for information extraction from radiology\nreports. NPJ Digit Med 7:222. https:\/\/doi.org\/10.1038\/s41746-024-01219-0","journal-title":"NPJ Digit Med"},{"key":"738_CR7","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231362","volume":"308","author":"MA Fink","year":"2023","unstructured":"Fink MA, Bischoff A, Fink CA et al (2023) Potential of ChatGPT and\nGPT-4 for data mining of free-text CT reports on lung cancer. Radiology\n308:e231362. https:\/\/doi.org\/10.1148\/radiol.231362","journal-title":"Radiology"},{"key":"738_CR8","doi-asserted-by":"publisher","unstructured":"Van Veen D, Van Uden C, Attias M et al (2023) RadAdapt: radiology\nreport summarization via lightweight domain adaptation of large language models.\nIn: Proceedings of the 22nd Workshop on Biomedical Natural Language Processing\nand BioNLP Shared Tasks. Association for Computational Linguistics, Toronto, pp\n449\u2013460. https:\/\/doi.org\/10.18653\/v1\/2023.bionlp-1.42","DOI":"10.18653\/v1\/2023.bionlp-1.42"},{"key":"738_CR9","doi-asserted-by":"publisher","unstructured":"Moassefi M, Houshmand S, Faghani S et al (2025) Cross-institutional\nevaluation of large language models for radiology diagnosis extraction: a\nprompt-engineering perspective. J Digit Imaging Inform Med. https:\/\/doi.org\/10.1007\/s10278-025-01523-5","DOI":"10.1007\/s10278-025-01523-5"},{"key":"738_CR10","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.241073","volume":"314","author":"CH Savage","year":"2025","unstructured":"Savage CH, Kanhere A, Parekh V et al (2025) Open-source large\nlanguage models in radiology: a review and tutorial for practical research and\nclinical deployment. Radiology 314:e241073. https:\/\/doi.org\/10.1148\/radiol.241073","journal-title":"Radiology"},{"key":"738_CR11","doi-asserted-by":"publisher","first-page":"188","DOI":"10.1016\/j.jcct.2017.02.008","volume":"11","author":"BO Sabel","year":"2017","unstructured":"Sabel BO, Plum JL, Kneidinger N et al (2017) Structured reporting\nof CT examinations in acute pulmonary embolism. J Cardiovasc Comput Tomogr\n11:188\u2013195. https:\/\/doi.org\/10.1016\/j.jcct.2017.02.008","journal-title":"J Cardiovasc Comput Tomogr"},{"key":"738_CR12","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1186\/s13244-023-01408-7","volume":"14","author":"T Jorg","year":"2023","unstructured":"Jorg T, Halfmann MC, Arnhold G et al (2023) Implementation of\nstructured reporting in clinical routine: a review of 7 years of institutional\nexperience. Insights Imaging 14:61. https:\/\/doi.org\/10.1186\/s13244-023-01408-7","journal-title":"Insights Imaging"},{"key":"738_CR13","doi-asserted-by":"publisher","unstructured":"Fink MA, Bischoff A, Atsiatorme E et al (2026) How green are large\nlanguage models for radiology report labelling? Comparing human, rule-based and\nhybrid workflows. Insights Imaging. https:\/\/doi.org\/10.1186\/s13244-026-02289-2","DOI":"10.1186\/s13244-026-02289-2"},{"key":"738_CR14","doi-asserted-by":"publisher","first-page":"1297","DOI":"10.1007\/s10278-024-01274-9","volume":"38","author":"Y Su","year":"2025","unstructured":"Su Y, Babore YB, Kahn CE (2025) A large language model to detect\nnegated expressions in radiology reports. J Digit Imaging Inf Med\n38:1297\u20131303. https:\/\/doi.org\/10.1007\/s10278-024-01274-9","journal-title":"J Digit Imaging Inf Med"},{"key":"738_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104428","volume":"144","author":"S Kierner","year":"2023","unstructured":"Kierner S, Kucharski J, Kierner Z (2023) Taxonomy of hybrid\narchitectures involving rule-based reasoning and machine learning in clinical\ndecision systems: a scoping review. J Biomed Inform 144:104428. https:\/\/doi.org\/10.1016\/j.jbi.2023.104428","journal-title":"J Biomed Inform"},{"key":"738_CR16","doi-asserted-by":"publisher","first-page":"e55318","DOI":"10.2196\/55318","volume":"12","author":"S Sivarajkumar","year":"2024","unstructured":"Sivarajkumar S, Kelley M, Samolyk-Mazzanti A, Visweswaran S, Wang Y\n(2024) An empirical evaluation of prompting strategies for large language models\nin zero-shot clinical natural language processing: algorithm development and\nvalidation study. JMIR Med Inform 12:e55318. https:\/\/doi.org\/10.2196\/55318","journal-title":"JMIR Med Inform"},{"key":"738_CR17","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1007\/s10462-024-10921-0","volume":"57","author":"D Wang","year":"2024","unstructured":"Wang D, Zhang S (2024) Large language models in medical and\nhealthcare fields: applications, advances, and challenges. Artif Intell Rev\n57:299. https:\/\/doi.org\/10.1007\/s10462-024-10921-0","journal-title":"Artif Intell Rev"},{"key":"738_CR18","doi-asserted-by":"publisher","first-page":"2018","DOI":"10.1007\/s00330-024-11074-y","volume":"35","author":"P Wo\u017anicki","year":"2025","unstructured":"Wo\u017anicki P, Laqua C, Fiku I et al (2025) Automatic structuring of\nradiology reports with on-premise open-source large language models. Eur Radiol\n35:2018\u20132029. https:\/\/doi.org\/10.1007\/s00330-024-11074-y","journal-title":"Eur Radiol"},{"key":"738_CR19","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.230364","volume":"6","author":"B Le Guellec","year":"2024","unstructured":"Le Guellec B, Lef\u00e8vre A, Geay C et al (2024) Performance of an\nopen-source large language model in extracting information from free-text\nradiology reports. Radiol Artif Intell 6:e230364. https:\/\/doi.org\/10.1148\/ryai.230364","journal-title":"Radiol Artif Intell"}],"container-title":["European Radiology Experimental"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41747-026-00738-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s41747-026-00738-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s41747-026-00738-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T11:06:03Z","timestamp":1782903963000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s41747-026-00738-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,27]]},"references-count":19,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["738"],"URL":"https:\/\/doi.org\/10.1186\/s41747-026-00738-7","relation":{},"ISSN":["2509-9280"],"issn-type":[{"value":"2509-9280","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,27]]},"assertion":[{"value":"18 January 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 March 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 April 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This retrospective study of pseudonymised CT pulmonary angiography\nreports was approved by the Ethics Committee of the Medical Faculty of\nHeidelberg University (S-236\/2020). The requirement for individual informed\nconsent was waived because the study used existing routine clinical data without\ndirect patient contact.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable. This manuscript does not contain identifiable\nindividual person data that would require specific consent for\npublication.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"74"}}