{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T00:12:58Z","timestamp":1791245578934,"version":"4.1.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,8,19]],"date-time":"2024-08-19T00:00:00Z","timestamp":1724025600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,19]],"date-time":"2024-08-19T00:00:00Z","timestamp":1724025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging. Inform. med."],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Expert feedback on trainees\u2019 preliminary reports is crucial for radiologic training, but real-time feedback can be challenging due to non-contemporaneous, remote reading and increasing imaging volumes. Trainee report revisions contain valuable educational feedback, but synthesizing data from raw revisions is challenging. Generative AI models can potentially analyze these revisions and provide structured, actionable feedback. This study used the OpenAI GPT-4 Turbo API to analyze paired synthesized and open-source analogs of preliminary and finalized reports, identify discrepancies, categorize their severity and type, and suggest review topics. Expert radiologists reviewed the output by grading discrepancies, evaluating the severity and category accuracy, and suggested review topic relevance. The reproducibility of discrepancy detection and maximal discrepancy severity was also examined. The model exhibited high sensitivity, detecting significantly more discrepancies than radiologists (<jats:italic>W<\/jats:italic>\u2009=\u200919.0, <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001) with a strong positive correlation (<jats:italic>r<\/jats:italic>\u2009=\u20090.778, <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001). Interrater reliability for severity and type were fair (Fleiss\u2019 kappa\u2009=\u20090.346 and 0.340, respectively; weighted kappa\u2009=\u20090.622 for severity). The LLM achieved a weighted F1 score of 0.66 for severity and 0.64 for type. Generated teaching points were considered relevant in\u2009~\u200985% of cases, and relevance correlated with the maximal discrepancy severity (Spearman <jats:italic>\u03c1<\/jats:italic>\u2009=\u20090.76, <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001). The reproducibility was moderate to good (ICC (2,1)\u2009=\u20090.690) for the number of discrepancies and substantial for maximal discrepancy severity (Fleiss\u2019 kappa\u2009=\u20090.718; weighted kappa\u2009=\u20090.94). Generative AI models can effectively identify discrepancies in report revisions and generate relevant educational feedback, offering promise for enhancing radiology training.<\/jats:p>","DOI":"10.1007\/s10278-024-01233-4","type":"journal-article","created":{"date-parts":[[2024,8,19]],"date-time":"2024-08-19T16:02:15Z","timestamp":1724083335000},"page":"1265-1279","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5999-6194","authenticated-orcid":false,"given":"Shawn","family":"Lyo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4025-115X","authenticated-orcid":false,"given":"Suyash","family":"Mohan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1292-5265","authenticated-orcid":false,"given":"Alvand","family":"Hassankhani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3969-6351","authenticated-orcid":false,"given":"Abass","family":"Noor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4765-9358","authenticated-orcid":false,"given":"Farouk","family":"Dako","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6007-7267","authenticated-orcid":false,"given":"Tessa","family":"Cook","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,8,19]]},"reference":[{"key":"1233_CR1","doi-asserted-by":"crossref","unstructured":"Dako, F. and O.A. Awan, The radiology readout: How much does it matter? 2021, Radiological Society of North America. p. 316\u2013317.","DOI":"10.1148\/rg.2021200023"},{"key":"1233_CR2","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1007\/s10278-011-9419-5","volume":"25","author":"RE Sharpe","year":"2012","unstructured":"Sharpe, R.E., et al., Radiology report comparator: a novel method to augment resident education. Journal of digital imaging, 2012. 25: p. 330-336.","journal-title":"Journal of digital imaging"},{"issue":"6","key":"1233_CR3","doi-asserted-by":"publisher","first-page":"872","DOI":"10.1016\/j.acra.2020.04.028","volume":"27","author":"CH Li","year":"2020","unstructured":"Li, C.H., et al., Virtual read-out: radiology education for the 21st century during the COVID-19 pandemic. Academic Radiology, 2020. 27(6): p. 872-881.","journal-title":"Academic Radiology"},{"key":"1233_CR4","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1007\/s10278-015-9840-2","volume":"29","author":"AD Kalaria","year":"2016","unstructured":"Kalaria, A.D. and R.W. Filice, Comparison-bot: an automated preliminary-final report comparison system. Journal of digital imaging, 2016. 29: p. 325-330.","journal-title":"Journal of digital imaging"},{"issue":"7","key":"1233_CR5","doi-asserted-by":"publisher","first-page":"2127","DOI":"10.1148\/rg.327105701","volume":"32","author":"T O\u2019Connell","year":"2012","unstructured":"O\u2019Connell, T. and D. Chang, Informatics in Radiology: Web-based Preliminary Reporting System for Radiology Residents with PACS Integration. RadioGraphics, 2012. 32(7): p. 2127-2134.","journal-title":"RadioGraphics"},{"issue":"10","key":"1233_CR6","doi-asserted-by":"publisher","first-page":"1242","DOI":"10.1016\/j.acra.2015.06.011","volume":"22","author":"P-H Chen","year":"2015","unstructured":"Chen, P.-H., Y.J. Chen, and T.S. Cook, Capricorn\u2013A Web-Based Automatic Case Log and Volume Analytics for Diagnostic Radiology Residents. Academic Radiology, 2015. 22(10): p. 1242-1251.","journal-title":"Academic Radiology"},{"issue":"5","key":"1233_CR7","doi-asserted-by":"publisher","first-page":"1443","DOI":"10.1148\/rg.2018170117","volume":"38","author":"AB Durojaiye","year":"2018","unstructured":"Durojaiye, A.B., et al., Radiology resident assessment and feedback dashboard. Radiographics, 2018. 38(5): p. 1443-1453.","journal-title":"Radiographics"},{"key":"1233_CR8","doi-asserted-by":"crossref","unstructured":"Akinci D\u2019Antonoli, T., et al., Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions. Diagnostic and Interventional Radiology, 2023: p. Epub ahead of print.","DOI":"10.4274\/dir.2023.232417"},{"key":"1233_CR9","unstructured":"Vaswani, A., et al., Attention is all you need. Advances in neural information processing systems, 2017. 30."},{"key":"1233_CR10","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown, T., et al., Language models are few-shot learners. Advances in neural information processing systems, 2020. 33: p. 1877-1901.","journal-title":"Advances in neural information processing systems"},{"key":"1233_CR11","doi-asserted-by":"crossref","unstructured":"Tippareddy, C., et al., Radiology reading room for the future: harnessing the power of large language models like ChatGPT. Current Problems in Diagnostic Radiology, 2023.","DOI":"10.1067\/j.cpradiol.2023.08.018"},{"issue":"4","key":"1233_CR12","doi-asserted-by":"publisher","first-page":"e230725","DOI":"10.1148\/radiol.230725","volume":"307","author":"LC Adams","year":"2023","unstructured":"Adams, L.C., et al., Leveraging GPT-4 for post hoc transformation of free-text radiology reports into structured reporting: a multilingual feasibility study. Radiology, 2023. 307(4): p. e230725.","journal-title":"Radiology"},{"key":"1233_CR13","doi-asserted-by":"crossref","unstructured":"Schmidt, R.A., et al., Generative Large Language Models for Detection of Speech Recognition Errors in Radiology Reports. Radiology: Artificial Intelligence, 2024. 6(2): p. e230205.","DOI":"10.1148\/ryai.230205"},{"issue":"1","key":"1233_CR14","doi-asserted-by":"publisher","first-page":"e231167","DOI":"10.1148\/radiol.231167","volume":"308","author":"J Kottlors","year":"2023","unstructured":"Kottlors, J., et al., Feasibility of differential diagnosis based on imaging patterns using a large language model. Radiology, 2023. 308(1): p. e231167.","journal-title":"Radiology"},{"issue":"1","key":"1233_CR15","doi-asserted-by":"publisher","first-page":"e232756","DOI":"10.1148\/radiol.232756","volume":"310","author":"R Bhayana","year":"2024","unstructured":"Bhayana, R., Chatbots and large language models in radiology: a practical primer for clinical and research applications. Radiology, 2024. 310(1): p. e232756.","journal-title":"Radiology"},{"issue":"3","key":"1233_CR16","doi-asserted-by":"publisher","first-page":"e231362","DOI":"10.1148\/radiol.231362","volume":"308","author":"MA Fink","year":"2023","unstructured":"Fink, M.A., et al., Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer. Radiology, 2023. 308(3): p. e231362.","journal-title":"Radiology"},{"issue":"1","key":"1233_CR17","doi-asserted-by":"publisher","first-page":"e232714","DOI":"10.1148\/radiol.232714","volume":"311","author":"RJ Gertz","year":"2024","unstructured":"Gertz, R.J., et al., Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy. Radiology, 2024. 311(1): p. e232714.","journal-title":"Radiology"},{"issue":"1103","key":"1233_CR18","doi-asserted-by":"publisher","first-page":"20190389","DOI":"10.1259\/bjr.20190389","volume":"92","author":"MT Duong","year":"2019","unstructured":"Duong, M.T., et al., Artificial intelligence for precision education in radiology. The British journal of radiology, 2019. 92(1103): p. 20190389.","journal-title":"The British journal of radiology"},{"issue":"11","key":"1233_CR19","doi-asserted-by":"publisher","first-page":"1388","DOI":"10.1016\/j.jacr.2020.09.028","volume":"17","author":"SA Simpson","year":"2020","unstructured":"Simpson, S.A. and T.S. Cook, Artificial intelligence and the trainee experience in radiology. Journal of the American College of Radiology, 2020. 17(11): p. 1388-1393.","journal-title":"Journal of the American College of Radiology"},{"key":"1233_CR20","doi-asserted-by":"crossref","unstructured":"Levy, M., A. Jacoby, and Y. Goldberg, Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848, 2024.","DOI":"10.18653\/v1\/2024.acl-long.818"},{"key":"1233_CR21","unstructured":"Dong, Z., et al., Exploring Context Window of Large Language Models via Decomposed Positional Vectors. arXiv preprint arXiv:2405.18009, 2024."},{"key":"1233_CR22","doi-asserted-by":"crossref","unstructured":"Krishna, S., et al., Evaluation of Reliability, Repeatability, Robustness, and Confidence of GPT-3.5 and GPT-4 on a Radiology Board\u2013style Examination. Radiology, 2024. 311(2): p. e232715.","DOI":"10.1148\/radiol.232715"},{"key":"1233_CR23","unstructured":"Yu, F., et al., Radiology Report Expert Evaluation (ReXVal) Dataset. 2023."},{"issue":"6","key":"1233_CR24","doi-asserted-by":"publisher","first-page":"694","DOI":"10.1016\/j.acra.2016.12.004","volume":"24","author":"JC Wildenberg","year":"2017","unstructured":"Wildenberg, J.C., et al., Attending radiologist variability and its effect on radiology resident discrepancy rates. Academic Radiology, 2017. 24(6): p. 694-699.","journal-title":"Academic Radiology"},{"issue":"5","key":"1233_CR25","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1177\/0284185114532922","volume":"56","author":"G Issa","year":"2015","unstructured":"Issa, G., et al., The discrepancy rate between preliminary and official reports of emergency radiology studies: a performance indicator and quality improvement method. Acta Radiologica, 2015. 56(5): p. 598-604.","journal-title":"Acta Radiologica"},{"issue":"4","key":"1233_CR26","doi-asserted-by":"publisher","first-page":"739","DOI":"10.2214\/AJR.16.16963","volume":"208","author":"S Waite","year":"2017","unstructured":"Waite, S., et al., Interpretive Error in Radiology. AJR Am J Roentgenol, 2017. 208(4): p. 739-749.","journal-title":"AJR Am J Roentgenol"},{"key":"1233_CR27","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1007\/978-3-031-03763-4_4","volume-title":"Statistical Methods for Annotation Analysis","author":"S Paun","year":"2022","unstructured":"Paun, S., R. Artstein, and M. Poesio, Probabilistic Models of Agreement, in Statistical Methods for Annotation Analysis. 2022, Springer International Publishing: Cham. p. 79-101."},{"key":"1233_CR28","unstructured":"@AnthropicAI, A. Anthropic on X: \"We fed Claude-Instant The Great Gatsby (72K tokens), except we modified one line to say that Mr. Carraway was \"a software engineer that works on machine learning tooling at Anthropic.\" We asked the model to spot what was added - it responded with the right answer in 22 seconds.\" \/ X [Tweet] 2023 11 May 2023 [cited 2024 7\/26\/2024]; Available from: https:\/\/x.com\/AnthropicAI\/status\/1656700154190389248."},{"key":"1233_CR29","unstructured":"Schulhoff, S., et al., The Prompt Report: A Systematic Survey of Prompting Techniques. arXiv preprint arXiv:2406.06608, 2024."},{"key":"1233_CR30","doi-asserted-by":"crossref","unstructured":"Tejani, A.S., et al., Performance of multiple pretrained BERT models to automate and accelerate data annotation for large datasets. Radiology: Artificial Intelligence, 2022. 4(4): p. e220007.","DOI":"10.1148\/ryai.220007"},{"key":"1233_CR31","doi-asserted-by":"crossref","unstructured":"Jin, Q., et al., Hidden flaws behind expert-level accuracy of gpt-4 vision in medicine. arXiv preprint arXiv:2401.08396, 2024.","DOI":"10.1038\/s41746-024-01185-7"}],"container-title":["Journal of Imaging Informatics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-024-01233-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-024-01233-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-024-01233-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,30]],"date-time":"2025-03-30T14:17:39Z","timestamp":1743344259000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-024-01233-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,19]]},"references-count":31,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["1233"],"URL":"https:\/\/doi.org\/10.1007\/s10278-024-01233-4","relation":{},"ISSN":["2948-2933"],"issn-type":[{"value":"2948-2933","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,19]]},"assertion":[{"value":"13 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 August 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 August 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 August 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"There are no conflicts of interest to report.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}]}}