{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,14]],"date-time":"2026-11-14T11:41:19Z","timestamp":1794656479875,"version":"4.9.1"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T00:00:00Z","timestamp":1685664000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T00:00:00Z","timestamp":1685664000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Objective<\/jats:title>\n                <jats:p>To examine whether incorrect AI results impact radiologist performance, and if so, whether human factors can be optimized to reduce error.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>Multi-reader design, 6 radiologists interpreted 90 identical chest radiographs (follow-up CT needed: yes\/no) on four occasions (09\/20\u201301\/22). No AI result was provided for session 1. Sham AI results were provided for sessions 2\u20134, and AI for 12 cases were manipulated to be incorrect (8 false positives (FP), 4 false negatives (FN)) (0.87 ROC-AUC). In the Delete AI (No Box) condition, radiologists were told AI results would not be saved for the evaluation. In Keep AI (No Box) and Keep AI (Box), radiologists were told results would be saved. In Keep AI (Box), the ostensible AI program visually outlined the region of suspicion. AI results were constant between conditions.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Relative to the No AI condition (FN\u2009=\u20092.7%, FP\u2009=\u200951.4%), FN and FPs were higher in the Keep AI (No Box) (FN\u2009=\u200933.0%, FP\u2009=\u200986.0%), Delete AI (No Box) (FN\u2009=\u200926.7%, FP\u2009=\u200980.5%), and Keep AI (Box) (FN\u2009=\u2009to 20.7%, FP\u2009=\u200980.5%) conditions (all <jats:italic>p<\/jats:italic>s\u2009&lt;\u20090.05). FNs were higher in the Keep AI (No Box) condition (33.0%) than in the Keep AI (Box) condition (20.7%) (<jats:italic>p<\/jats:italic>\u2009=\u20090.04). FPs were higher in the Keep AI (No Box) (86.0%) condition than in the Delete AI (No Box) condition (80.5%) (<jats:italic>p<\/jats:italic>\u2009=\u20090.03).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>Incorrect AI causes radiologists to make incorrect follow-up decisions when they were correct without AI. This effect is mitigated when radiologists believe AI will be deleted from the patient\u2019s file or a box is provided around the region of interest.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Clinical relevance statement<\/jats:title>\n                <jats:p>When AI is wrong, radiologists make more errors than they would have without AI. Based on human factors psychology, our manuscript provides evidence for two AI implementation strategies that reduce the deleterious effects of incorrect AI.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Key Points<\/jats:title>\n                <jats:p>\u2022 <jats:italic>When AI provided incorrect results, false negative and false positive rates among the radiologists increased.<\/jats:italic><\/jats:p>\n                <jats:p>\u2022 <jats:italic>False positives decreased when AI results were deleted, versus kept, in the patient\u2019s record.<\/jats:italic><\/jats:p>\n                <jats:p>\u2022 <jats:italic>False negatives and false positives decreased when AI visually outlined the region of suspicion.<\/jats:italic><\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s00330-023-09747-1","type":"journal-article","created":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T07:10:55Z","timestamp":1685689855000},"page":"8263-8269","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":145,"title":["Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography"],"prefix":"10.1007","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7221-0697","authenticated-orcid":false,"given":"Michael H.","family":"Bernstein","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael K.","family":"Atalay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth H.","family":"Dibble","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron W. P.","family":"Maxwell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adib R.","family":"Karam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saurabh","family":"Agarwal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert C.","family":"Ward","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Terrance T.","family":"Healey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Grayson L.","family":"Baird","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,6,2]]},"reference":[{"key":"9747_CR1","unstructured":"American College of Radiology (2022). Available via https:\/\/aicentral.acrdsi.org\/. Accessed 12 Oct 2022"},{"key":"9747_CR2","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1016\/j.jacr.2021.04.002","volume":"18","author":"B Allen","year":"2021","unstructured":"Allen B, Agarwal S, Coombs L, Wald C, Dreyer K (2021) 2020 ACR Data Science Institute artificial intelligence survey. J Am Coll Radiol 18:1153\u20131159","journal-title":"J Am Coll Radiol"},{"key":"9747_CR3","doi-asserted-by":"crossref","unstructured":"Aggarwal R, Sounderajah V, Martin G et al (2021) Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ Digit Med 4:65","DOI":"10.1038\/s41746-021-00438-z"},{"key":"9747_CR4","doi-asserted-by":"publisher","first-page":"e496","DOI":"10.1016\/S2589-7500(21)00106-0","volume":"3","author":"JC Seah","year":"2021","unstructured":"Seah JC, Tang CH, Buchlak QD et al (2021) Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. Lancet Digit Health 3:e496\u2013e506","journal-title":"Lancet Digit Health"},{"key":"9747_CR5","doi-asserted-by":"publisher","first-page":"e2141096","DOI":"10.1001\/jamanetworkopen.2021.41096","volume":"4","author":"F Homayounieh","year":"2021","unstructured":"Homayounieh F, Digumarthy S, Ebrahimian S et al (2021) An artificial intelligence\u2013based chest X-ray model on human nodule detection accuracy from a multicenter study. JAMA Netw Open 4:e2141096\u2013e2141096","journal-title":"JAMA Netw Open"},{"key":"9747_CR6","doi-asserted-by":"publisher","first-page":"652","DOI":"10.1148\/radiol.2020200165","volume":"296","author":"S Jang","year":"2020","unstructured":"Jang S, Song H, Shin YJ et al (2020) Deep learning\u2013based automatic detection algorithm for reducing overlooked lung cancers on chest radiographs. Radiology 296:652\u2013661","journal-title":"Radiology"},{"key":"9747_CR7","doi-asserted-by":"publisher","first-page":"909","DOI":"10.1016\/j.acra.2004.05.012","volume":"11","author":"E Alberdi","year":"2004","unstructured":"Alberdi E, Povyakalo A, Strigini L, Ayton P (2004) Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography. Acad Radiol 11:909\u2013918","journal-title":"Acad Radiol"},{"key":"9747_CR8","unstructured":"American College of Radiology About ACR DSI. Available via https:\/\/www.acrdsi.org\/About-ACR-DSI. Accessed 30 Mar 2022"},{"key":"9747_CR9","unstructured":"InternationalErgonomicsAssociation What Is Ergonomics? Available via https:\/\/iea.cc\/what-is-ergonomics\/. Accessed 30 Sept 2021"},{"key":"9747_CR10","doi-asserted-by":"publisher","first-page":"1102","DOI":"10.1016\/j.jacr.2016.02.017","volume":"13","author":"RM Ratwani","year":"2016","unstructured":"Ratwani RM, Wang E, Fong A, Cooper CJ (2016) A human factors approach to understanding the types and sources of interruptions in radiology reading rooms. J Am Coll Radiol 13:1102\u20131105","journal-title":"J Am Coll Radiol"},{"key":"9747_CR11","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1016\/j.jacr.2020.09.041","volume":"18","author":"EP Larsen","year":"2021","unstructured":"Larsen EP, Hailu T, Sheldon L et al (2021) Optimizing radiology reading room design: the Eudaimonia Radiology Machine. J Am Coll Radiol 18:108\u2013120","journal-title":"J Am Coll Radiol"},{"key":"9747_CR12","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.jacr.2017.08.027","volume":"15","author":"RM Nishikawa","year":"2018","unstructured":"Nishikawa RM, Bae KT (2018) Importance of better human-computer interaction in the era of deep learning: mammography computer-aided diagnosis as a use case. J Am Coll Radiol 15:49\u201352","journal-title":"J Am Coll Radiol"},{"key":"9747_CR13","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1148\/radiol.210318","volume":"303","author":"MH Bernstein","year":"2022","unstructured":"Bernstein MH, Baird GL, Lourenco AP (2022) Digital breast tomosynthesis and digital mammography recall and false-positive rates by time of day and reader experience. Radiology 303:63\u201368","journal-title":"Radiology"},{"key":"9747_CR14","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1038\/s41591-018-0307-0","volume":"25","author":"J He","year":"2019","unstructured":"He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K (2019) The practical implementation of artificial intelligence technologies in medicine. Nat Med 25:30\u201336","journal-title":"Nat Med"},{"key":"9747_CR15","doi-asserted-by":"crossref","unstructured":"Philpotts LE (2022) Advancing artificial intelligence to meet breast imaging needs. Radiology\u00a0303:78\u201379","DOI":"10.1148\/radiol.213101"},{"key":"9747_CR16","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1148\/radiol.2020203782","volume":"298","author":"EA Krupinski","year":"2021","unstructured":"Krupinski EA (2021) Evaluating AI clinically\u2014it\u2019s not just ROC AUC! Radiology 298:47\u201348","journal-title":"Radiology"},{"key":"9747_CR17","doi-asserted-by":"publisher","first-page":"5525","DOI":"10.1007\/s00330-020-06946-y","volume":"30","author":"L Strohm","year":"2020","unstructured":"Strohm L, Hehakaya C, Ranschaert ER, Boon WPC, Moors EHM (2020) Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur Radiol 30:5525\u20135532","journal-title":"Eur Radiol"},{"key":"9747_CR18","doi-asserted-by":"publisher","first-page":"1752","DOI":"10.1001\/jama.1996.03540210060035","volume":"276","author":"TK Egglin","year":"1996","unstructured":"Egglin TK, Feinstein AR (1996) Context bias: a problem in diagnostic radiology. JAMA 276:1752\u20131755","journal-title":"JAMA"},{"key":"9747_CR19","unstructured":"Jodogne S (2016) ORTHANC. Available via https:\/\/www.orthanc-server.com\/download.php. Accessed 2 Apr 2022"},{"key":"9747_CR20","unstructured":"Hundall C (2021) Thinking about AI? Available via https:\/\/www.acr.org\/Practice-Management-Quality-Informatics\/ACR-Bulletin\/Articles\/June-2021\/Thinking-About-AI. Accessed 30 Mar 2022"},{"key":"9747_CR21","unstructured":"Walton D (2010) Appeal to expert opinion: arguments from authority. Penn State Press"},{"key":"9747_CR22","doi-asserted-by":"crossref","unstructured":"Stec N, Arje D, Moody AR, Krupinski EA, Tyrrell PN (2018) A systematic review of fatigue in radiology: is it a problem? AJR Am J Roentgenol 210:799\u2013806","DOI":"10.2214\/AJR.17.18613"},{"key":"9747_CR23","doi-asserted-by":"publisher","first-page":"1560","DOI":"10.1016\/j.acra.2010.07.004","volume":"17","author":"DS Feigin","year":"2010","unstructured":"Feigin DS (2010) Lateral chest radiograph: a systematic approach. Acad Radiol 17:1560\u20131566","journal-title":"Acad Radiol"},{"key":"9747_CR24","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1148\/radiol.2018181371","volume":"290","author":"A Rodr\u00edguez-Ruiz","year":"2019","unstructured":"Rodr\u00edguez-Ruiz A, Krupinski E, Mordang J-J et al (2019) Detection of breast cancer with mammography: effect of an artificial intelligence support system. Radiology 290:305\u2013314","journal-title":"Radiology"},{"key":"9747_CR25","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacr.2017.12.019","author":"TN Hanna","year":"2018","unstructured":"Hanna TN, Zygmont ME, Peterson R et al (2018) The effects of fatigue from overnight shifts on radiology search patterns and diagnostic performance. J Am Coll Radiol. https:\/\/doi.org\/10.1016\/j.jacr.2017.12.019","journal-title":"J Am Coll Radiol"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-023-09747-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-023-09747-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-023-09747-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,24]],"date-time":"2023-10-24T17:06:50Z","timestamp":1698167210000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-023-09747-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,2]]},"references-count":25,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,11]]}},"alternative-id":["9747"],"URL":"https:\/\/doi.org\/10.1007\/s00330-023-09747-1","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,2]]},"assertion":[{"value":"9 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 April 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 April 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 June 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The scientific guarantor of this publication is Michael Bernstein.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"The authors of this manuscript declare relationships with the following companies: Lunit. Authors MHB and GLB are forming a relationship with the AI company, Lunit. The study published here began prior to our work with Lunit. Lunit had no involvement in any aspect of this study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"One of the authors has significant statistical expertise.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"Written informed consent was waived by the Institutional Review Board.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"\u2022 prospective\u2022 experimental\u2022 performed at one institution","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}