{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T17:46:25Z","timestamp":1790876785509,"version":"4.1.0"},"reference-count":18,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T00:00:00Z","timestamp":1717372800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010663","name":"European Research Council","doi-asserted-by":"publisher","award":["MIA\u2010NORMAL 101083647"],"award-info":[{"award-number":["MIA\u2010NORMAL 101083647"]}],"id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["102431"],"award-info":[{"award-number":["102431"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100023232","name":"National Institute for Health Research Applied Research Collaboration South London","doi-asserted-by":"publisher","award":["NIHR300555"],"award-info":[{"award-number":["NIHR300555"]}],"id":[{"id":"10.13039\/501100023232","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100023232","name":"National Institute for Health Research Applied Research Collaboration South London","doi-asserted-by":"publisher","award":["NIHR301448"],"award-info":[{"award-number":["NIHR301448"]}],"id":[{"id":"10.13039\/501100023232","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["obgyn.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Ultrasound in Obstet &amp; Gyne"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:title>ABSTRACT<\/jats:title><jats:sec><jats:title>Objectives<\/jats:title><jats:p>Artificial intelligence (AI) has shown promise in improving the performance of fetal ultrasound screening in detecting congenital heart disease (CHD). The effect of giving AI advice to human operators has not been studied in this context. Giving additional information about AI model workings, such as confidence scores for AI predictions, may be a way of further improving performance. Our aims were to investigate whether AI advice improved overall diagnostic accuracy (using a single CHD lesion as an exemplar), and to determine what, if any, additional information given to clinicians optimized the overall performance of the clinician\u2013AI team.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>An AI model was trained to classify a single fetal CHD lesion (atrioventricular septal defect (AVSD)), using a retrospective cohort of 121\u2009130 cardiac four\u2010chamber images extracted from 173 ultrasound scan videos (98 with normal hearts, 75 with AVSD); a ResNet50 model architecture was used. Temperature scaling of model prediction probability was performed on a validation set, and gradient\u2010weighted class activation maps (grad\u2010CAMs) produced. Ten clinicians (two consultant fetal cardiologists, three trainees in pediatric cardiology and five fetal cardiac sonographers) were recruited from a center of fetal cardiology to participate. Each participant was shown 2000 fetal four\u2010chamber images in a random order (1000 normal and 1000 AVSD). The dataset comprised 500 images, each shown in four conditions: (1) image alone without AI output; (2) image with binary AI classification; (3) image with AI model confidence; and (4) image with grad\u2010CAM image overlays. The clinicians were asked to classify each image as normal or AVSD.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>A total of 20\u2009000 image classifications were recorded from 10 clinicians. The AI model alone achieved an accuracy of 0.798 (95%\u2009CI, 0.760\u20130.832), a sensitivity of 0.868 (95%\u2009CI, 0.834\u20130.902) and a specificity of 0.728 (95%\u2009CI, 0.702\u20130.754), and the clinicians without AI achieved an accuracy of 0.844 (95%\u2009CI, 0.834\u20130.854), a sensitivity of 0.827 (95%\u2009CI, 0.795\u20130.858) and a specificity of 0.861 (95%\u2009CI, 0.828\u20130.895). Showing a binary (normal or AVSD) AI model output resulted in significant improvement in accuracy to 0.865 (<jats:italic>P<\/jats:italic>\u2009&lt;\u20090.001). This effect was seen in both experienced and less\u2010experienced participants. Giving incorrect AI advice resulted in a significant deterioration in overall accuracy, from 0.761 to 0.693 (<jats:italic>P<\/jats:italic>\u2009&lt;\u20090.001), which was driven by an increase in both Type\u2010I and Type\u2010II errors by the clinicians. This effect was worsened by showing model confidence (accuracy, 0.649; <jats:italic>P<\/jats:italic>\u2009&lt;\u20090.001) or grad\u2010CAM (accuracy, 0.644; <jats:italic>P<\/jats:italic>\u2009&lt;\u20090.001).<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>AI has the potential to improve performance when used in collaboration with clinicians, even if the model performance does not reach expert level. Giving additional information about model workings such as model confidence and class activation map image overlays did not improve overall performance, and actually worsened performance for images for which the AI model was incorrect. \u00a9 2024 The Authors. <jats:italic>Ultrasound in Obstetrics &amp; Gynecology<\/jats:italic> published by John Wiley &amp; Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.<\/jats:p><\/jats:sec>","DOI":"10.1002\/uog.27577","type":"journal-article","created":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T13:27:36Z","timestamp":1704893256000},"page":"28-35","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Interaction between clinicians and artificial intelligence to detect fetal atrioventricular septal defects on ultrasound: how can we optimize collaborative performance?"],"prefix":"10.1002","volume":"64","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8391-7903","authenticated-orcid":false,"given":"T. G.","family":"Day","sequence":"first","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"},{"name":"Department of Congenital Heart Disease, Evelina London Children's Healthcare Guy's and St Thomas' NHS Foundation Trust  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4754-0322","authenticated-orcid":false,"given":"J.","family":"Matthew","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"S. F.","family":"Budd","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"L.","family":"Venturini","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"R.","family":"Wright","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"A.","family":"Farruggia","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"T. V.","family":"Vigneswaran","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"},{"name":"Department of Congenital Heart Disease, Evelina London Children's Healthcare Guy's and St Thomas' NHS Foundation Trust  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"V.","family":"Zidere","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"},{"name":"Department of Congenital Heart Disease, Evelina London Children's Healthcare Guy's and St Thomas' NHS Foundation Trust  London UK"},{"name":"Harris Birthright Research Centre King's College London NHS Foundation Trust  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J. V.","family":"Hajnal","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"R.","family":"Razavi","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"},{"name":"Department of Congenital Heart Disease, Evelina London Children's Healthcare Guy's and St Thomas' NHS Foundation Trust  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J. M.","family":"Simpson","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"},{"name":"Department of Congenital Heart Disease, Evelina London Children's Healthcare Guy's and St Thomas' NHS Foundation Trust  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"B.","family":"Kainz","sequence":"additional","affiliation":[{"name":"School of Biomedical Engineering and Imaging Sciences, Faculty of Life Sciences and Medicine King's College London  London UK"},{"name":"Department Artificial Intelligence in Biomedical Engineering Friedrich\u2010Alexander\u2010Universit\u00e4t, Erlangen\u2010N\u00fcrnberg  Germany"},{"name":"Department of Computing, Faculty of Engineering Imperial College London  London UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2024,6,3]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1542\/peds.107.6.1277"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpeds.2011.12.036"},{"key":"e_1_2_8_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.14882"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1111\/1471-0528.17287"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/pd.5892"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-021-01342-5"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.26238"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.3386\/w31422"},{"key":"e_1_2_8_10_1","unstructured":"Labelbox. Accessed July 27 2023.https:\/\/labelbox.com"},{"key":"e_1_2_8_11_1","unstructured":"Temperature scaling. Accessed August 23 2023.https:\/\/github.com\/gpleiss\/temperature_scaling"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-019-01228-7"},{"key":"e_1_2_8_13_1","unstructured":"Fetal Medicine Foundation.https:\/\/fetalmedicine.org\/education\/fetal\u2010echocardiography\u20101"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0189-7"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00385-9"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-28633-w"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720815581940"},{"key":"e_1_2_8_18_1","volume-title":"National Congenital Heart Disease Audit, Summary Report","year":"2021"},{"key":"e_1_2_8_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jtcvs.2015.05.011"}],"container-title":["Ultrasound in Obstetrics &amp; Gynecology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/uog.27577","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T19:20:34Z","timestamp":1735586434000},"score":1,"resource":{"primary":{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/10.1002\/uog.27577"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,3]]},"references-count":18,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["10.1002\/uog.27577"],"URL":"https:\/\/doi.org\/10.1002\/uog.27577","archive":["Portico"],"relation":{},"ISSN":["0960-7692","1469-0705"],"issn-type":[{"value":"0960-7692","type":"print"},{"value":"1469-0705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,3]]},"assertion":[{"value":"2023-09-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-30","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}