{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T10:22:22Z","timestamp":1777630942019,"version":"3.51.4"},"reference-count":24,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T00:00:00Z","timestamp":1716422400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"NYU Langone Health"},{"DOI":"10.13039\/100007845","name":"MCIT","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100007845","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["3UL1TR001445-05"],"award-info":[{"award-number":["3UL1TR001445-05"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["#1928614"],"award-info":[{"award-number":["#1928614"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["#2129076"],"award-info":[{"award-number":["#2129076"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To evaluate the proficiency of a HIPAA-compliant version of GPT-4 in identifying actionable, incidental findings from unstructured radiology reports of Emergency Department patients. To assess appropriateness of artificial intelligence (AI)-generated, patient-facing summaries of these findings.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Radiology reports extracted from the electronic health record of a large academic medical center were manually reviewed to identify non-emergent, incidental findings with high likelihood of requiring follow-up, further sub-stratified as \u201cdefinitely actionable\u201d (DA) or \u201cpossibly actionable\u2014clinical correlation\u201d (PA-CC). Instruction prompts to GPT-4 were developed and iteratively optimized using a validation set of 50 reports. The optimized prompt was then applied to a test set of 430 unseen reports. GPT-4 performance was primarily graded on accuracy identifying either DA or PA-CC findings, then secondarily for DA findings alone. Outputs were reviewed for hallucinations. AI-generated patient-facing summaries were assessed for appropriateness via Likert scale.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>For the primary outcome (DA or PA-CC), GPT-4 achieved 99.3% recall, 73.6% precision, and 84.5% F-1. For the secondary outcome (DA only), GPT-4 demonstrated 95.2% recall, 77.3% precision, and 85.3% F-1. No findings were \u201challucinated\u201d outright. However, 2.8% of cases included generated text about recommendations that were inferred without specific reference. The majority of True Positive AI-generated summaries required no or minor revision.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>GPT-4 demonstrates proficiency in detecting actionable, incidental findings after refined instruction prompting. AI-generated patient instructions were most often appropriate, but rarely included inferred recommendations. While this technology shows promise to augment diagnostics, active clinician oversight via \u201chuman-in-the-loop\u201d workflows remains critical for clinical implementation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae117","type":"journal-article","created":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T08:38:43Z","timestamp":1716453523000},"page":"1983-1993","source":"Crossref","is-referenced-by-count":21,"title":["Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findings"],"prefix":"10.1093","volume":"31","author":[{"given":"Kar-mun C","family":"Woo","sequence":"first","affiliation":[{"name":"Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gregory W","family":"Simon","sequence":"additional","affiliation":[{"name":"Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Olumide","family":"Akindutire","sequence":"additional","affiliation":[{"name":"Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yindalon","family":"Aphinyanaphongs","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]},{"name":"Department of Health Informatics, Medical Center IT, NYU Langone Health , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonathan S","family":"Austrian","sequence":"additional","affiliation":[{"name":"Department of Health Informatics, Medical Center IT, NYU Langone Health , New York, NY 10016,","place":["United States"]},{"name":"Department of Medicine, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jung G","family":"Kim","sequence":"additional","affiliation":[{"name":"Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]},{"name":"Institute for Innovations in Medical Education, NYU Langone Health, New York, NY 10016 ,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicholas","family":"Genes","sequence":"additional","affiliation":[{"name":"Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]},{"name":"Department of Health Informatics, Medical Center IT, NYU Langone Health , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jacob A","family":"Goldenring","sequence":"additional","affiliation":[{"name":"Ronald O. Perelman Department of Emergency Medicine, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2604-3458","authenticated-orcid":false,"given":"Vincent J","family":"Major","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]},{"name":"Department of Health Informatics, Medical Center IT, NYU Langone Health , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chlo\u00e9 S","family":"Pariente","sequence":"additional","affiliation":[{"name":"Department of Health Informatics, Medical Center IT, NYU Langone Health , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edwin G","family":"Pineda","sequence":"additional","affiliation":[{"name":"MCIT Clinical Systems\u2014ASAP application, NYU Langone Health , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2402-3787","authenticated-orcid":false,"given":"Stella K","family":"Kang","sequence":"additional","affiliation":[{"name":"Department of Population Health, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]},{"name":"Department of Radiology, NYU Grossman School of Medicine , New York, NY 10016,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2024,5,23]]},"reference":[{"issue":"9","key":"2025052220014625200_ocae117-B1","doi-asserted-by":"crossref","first-page":"672","DOI":"10.1016\/j.jacr.2013.05.012","article-title":"Overview of white papers of the ACR incidental findings committee II on adnexal, vascular, splenic, nodal, gallbladder, and biliary findings","volume":"10","author":"Berland","year":"2013","journal-title":"J Am Coll Radiol"},{"issue":"3","key":"2025052220014625200_ocae117-B2","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1016\/j.annemergmed.2022.03.027","article-title":"Incidental radiology findings on CT studies in the ED: a systematic review and meta-analysis","volume":"80","author":"Evans","year":"2022","journal-title":"Ann Emerg Med"},{"issue":"988","key":"2025052220014625200_ocae117-B3","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1259\/bjr\/98067945","article-title":"Incidental findings in imaging diagnostic tests a systematic review","volume":"83","author":"Lumbreras","year":"2010","journal-title":"Br J Radiol"},{"issue":"4","key":"2025052220014625200_ocae117-B4","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1016\/j.jacr.2023.01.001","article-title":"White paper: best practices in the communication and management of actionable incidental findings in emergency department imaging","volume":"20","author":"Moore","year":"2023","journal-title":"J Am Coll Radiol"},{"key":"2025052220014625200_ocae117-B5","doi-asserted-by":"crossref","first-page":"k2387","DOI":"10.1136\/bmj.k2387","article-title":"Prevalence and outcomes of incidental imaging findings: umbrella review","volume":"361","author":"O'Sullivan","year":"2018","journal-title":"BMJ"},{"issue":"18","key":"2025052220014625200_ocae117-B6","doi-asserted-by":"crossref","first-page":"1821","DOI":"10.1056\/NEJMoa070972","article-title":"Incidental findings on brain MRI in the general population","volume":"357","author":"Vernooij","year":"2007","journal-title":"N Engl J Med"},{"issue":"9","key":"2025052220014625200_ocae117-B7","doi-asserted-by":"crossref","first-page":"843","DOI":"10.1001\/jama.2019.11456","article-title":"Trends in use of medical imaging in US health care systems and in Ontario, Canada, 2000-2016","volume":"322","author":"Smith-Bindman","year":"2019","journal-title":"JAMA"},{"issue":"3","key":"2025052220014625200_ocae117-B8","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.jemermed.2008.01.021","article-title":"Frequency and follow-up of incidental findings on trauma computed tomography scans: experience at a level one trauma center","volume":"38","author":"Munk","year":"2010","journal-title":"J Emerg Med"},{"key":"2025052220014625200_ocae117-B9","doi-asserted-by":"crossref","first-page":"624847","DOI":"10.1155\/2011\/624847","article-title":"Incidental findings on CT scans in the emergency department","volume":"2011","author":"Thompson","year":"2011","journal-title":"Emerg Med Int"},{"issue":"2","key":"2025052220014625200_ocae117-B10","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1016\/j.annemergmed.2013.02.001","article-title":"Automated detection using NLP and radiologists' recommendations for additional imaging of incidental findings","volume":"62","author":"Dutta","year":"2013","journal-title":"Ann Emerg Med"},{"key":"2025052220014625200_ocae117-B11","doi-asserted-by":"crossref","first-page":"109072","DOI":"10.1016\/j.ejrad.2020.109072","article-title":"Incidental findings on emergency CT scans: predictive factors and medico-economic impact","volume":"129","author":"Berge","year":"2020","journal-title":"Eur J Radiol"},{"issue":"6","key":"2025052220014625200_ocae117-B12","doi-asserted-by":"crossref","first-page":"781","DOI":"10.1016\/j.jacr.2018.11.010","article-title":"Location, location, location: the association between imaging setting and follow-up of findings of indeterminate malignant potential","volume":"16","author":"Liao","year":"2019","journal-title":"J Am Coll Radiol"},{"issue":"3","key":"2025052220014625200_ocae117-B13","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/j.annemergmed.2022.04.026","article-title":"Catching those who fall through the cracks: integrating a follow-up process for emergency department patients with incidental radiologic findings","volume":"80","author":"Barrett","year":"2022","journal-title":"Ann Emerg Med"},{"issue":"2","key":"2025052220014625200_ocae117-B14","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.jacr.2020.02.021","article-title":"Factors affecting adherence to recommendations for additional imaging of incidental findings in radiology reports","volume":"18","author":"Hansra","year":"2021","journal-title":"J Am Coll Radiol"},{"issue":"6","key":"2025052220014625200_ocae117-B15","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1016\/j.jacr.2020.12.027","article-title":"Incidental findings: a survey of radiologists and emergency physicians","volume":"18","author":"Moore","year":"2021","journal-title":"J Am Coll Radiol"},{"issue":"6","key":"2025052220014625200_ocae117-B16","doi-asserted-by":"crossref","first-page":"349","DOI":"10.12788\/jhm.3128","article-title":"Follow up of incidental high-risk pulmonary nodules on computed tomography pulmonary angiography at care transitions","volume":"14","author":"Kwan","year":"2019","journal-title":"J Hosp Med"},{"issue":"2","key":"2025052220014625200_ocae117-B17","doi-asserted-by":"crossref","first-page":"509.e7","DOI":"10.1053\/j.gastro.2023.04.033","article-title":"ChatGPT answers common patient questions about colonoscopy","volume":"165","author":"Lee","year":"2023","journal-title":"Gastroenterology"},{"issue":"11","key":"2025052220014625200_ocae117-B18","doi-asserted-by":"crossref","first-page":"2260","DOI":"10.1097\/CORR.0000000000002668","article-title":"Can artificial intelligence improve the readability of patient education materials?","volume":"481","author":"Kirchner","year":"2023","journal-title":"Clin Orthop Relat Res"},{"issue":"3","key":"2025052220014625200_ocae117-B19","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1016\/j.annemergmed.2022.08.450","article-title":"A NLP and ML approach to ID incidental radiology findings in trauma patients discharged from the ED","volume":"81","author":"Evans","year":"2023","journal-title":"Ann Emerg Med"},{"issue":"2","key":"2025052220014625200_ocae117-B20","doi-asserted-by":"crossref","first-page":"e12109","DOI":"10.2196\/12109","article-title":"Natural language processing for the identification of silent brain infarcts from neuroimaging reports","volume":"7","author":"Fu","year":"2019","journal-title":"JMIR Med Inform"},{"issue":"11","key":"2025052220014625200_ocae117-B21","doi-asserted-by":"crossref","first-page":"1587","DOI":"10.1016\/j.jacr.2019.04.026","article-title":"Natural language processing for identification of incidental pulmonary nodules in radiology reports","volume":"16","author":"Kang","year":"2019","journal-title":"J Am Coll Radiol"},{"issue":"1","key":"2025052220014625200_ocae117-B22","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1186\/1471-2105-15-266","article-title":"Natural language processing of radiology reports for the detection of thromboembolic diseases and clinically relevant incidental findings","volume":"15","author":"Pham","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2025052220014625200_ocae117-B23","author":"Nori","year":"2023"},{"issue":"3","key":"2025052220014625200_ocae117-B24","first-page":"e233065","article-title":"Use of GPT-4 with single-shot learning to identify incidental findings in radiology reports","volume":"222","author":"Bhayana","year":"2024","journal-title":"AJR Am J Roentgenol"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1983\/58868017\/ocae117.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/1983\/58868017\/ocae117.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,23]],"date-time":"2025-05-23T00:01:57Z","timestamp":1747958517000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/1983\/7680046"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,23]]},"references-count":24,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,5,23]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae117","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,5,23]]}}}