{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T16:20:35Z","timestamp":1792599635658,"version":"4.5.0"},"reference-count":63,"publisher":"SAGE Publications","issue":"3","license":[{"start":{"date-parts":[[2023,1,9]],"date-time":"2023-01-09T00:00:00Z","timestamp":1673222400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002141","name":"Public Health England","doi-asserted-by":"publisher","award":["ITT 4347"],"award-info":[{"award-number":["ITT 4347"]}],"id":[{"id":"10.13039\/501100002141","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["J Med Screen"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To systematically review the accuracy of artificial intelligence (AI)-based systems for grading of fundus images in diabetic retinopathy (DR) screening.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We searched MEDLINE, EMBASE, the Cochrane Library and the ClinicalTrials.gov from 1st January 2000 to 27th August 2021. Accuracy studies published in English were included if they met the pre-specified inclusion criteria. Selection of studies for inclusion, data extraction and quality assessment were conducted by one author with a second reviewer independently screening and checking 20% of titles. Results were analysed narratively.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Forty-three studies evaluating 15 deep learning (DL) and 4 machine learning (ML) systems were included. Nine systems were evaluated in a single study each. Most studies were judged to be at high or unclear risk of bias in at least one QUADAS-2 domain. Sensitivity for referable DR and higher grades was \u226585% while specificity varied and was &lt;80% for all ML systems and in 6\/31 studies evaluating DL systems. Studies reported high accuracy for detection of ungradable images, but the latter were analysed and reported inconsistently. Seven studies reported that AI was more sensitive but less specific than human graders.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>AI-based systems are more sensitive than human graders and could be safe to use in clinical practice but have variable specificity. However, for many systems evidence is limited, at high risk of bias and may not generalise across settings. Therefore, pre-implementation assessment in the target clinical pathway is essential to obtain reliable and applicable accuracy estimates.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/09691413221144382","type":"journal-article","created":{"date-parts":[[2023,1,9]],"date-time":"2023-01-09T03:13:40Z","timestamp":1673234020000},"page":"97-112","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":29,"title":["Test accuracy of artificial intelligence-based grading of fundus images in diabetic retinopathy screening: A systematic review"],"prefix":"10.1177","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0106-2401","authenticated-orcid":false,"given":"Zhivko","family":"Zhelev","sequence":"first","affiliation":[{"name":"Exeter Test Group, University of Exeter Medical School, University of Exeter, Exeter, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaime","family":"Peters","sequence":"additional","affiliation":[{"name":"Exeter Test Group, University of Exeter Medical School, University of Exeter, Exeter, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Morwenna","family":"Rogers","sequence":"additional","affiliation":[{"name":"NIHR ARC South West Peninsula (PenARC), University of Exeter Medical School, University of Exeter, Exeter, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Allen","sequence":"additional","affiliation":[{"name":"University of Exeter Medical School, University of Exeter, Exeter, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Goda","family":"Kijauskaite","sequence":"additional","affiliation":[{"name":"UK National Screening Committee, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Farah","family":"Seedat","sequence":"additional","affiliation":[{"name":"UK National Screening Committee, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth","family":"Wilkinson","sequence":"additional","affiliation":[{"name":"Northern Devon Healthcare NHS Trust, Devon, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7349-0616","authenticated-orcid":false,"given":"Christopher","family":"Hyde","sequence":"additional","affiliation":[{"name":"Exeter Test Group, University of Exeter Medical School, University of Exeter, Exeter, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,1,9]]},"reference":[{"key":"bibr1-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1038\/s41433-020-0961-6"},{"key":"bibr2-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1007\/s00592-017-0974-1"},{"key":"bibr3-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1001288"},{"key":"bibr4-09691413221144382","doi-asserted-by":"publisher","DOI":"10.2337\/dc12-2392"},{"key":"bibr5-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1111\/j.1464-5491.2009.02870.x"},{"key":"bibr6-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2013-004015"},{"key":"bibr7-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-014444"},{"key":"bibr8-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1159\/000502387"},{"key":"bibr9-09691413221144382","unstructured":"Broadbent DA-O, Wang AA-O, Cheyne CA-O, et al. Safety and cost-effectiveness of individualised screening for diabetic retinopathy: the ISDR open-label, equivalence RCT. 2021(1432-0428 (Electronic))."},{"key":"bibr10-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1007\/s11892-017-0928-6"},{"key":"bibr11-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1136\/bjo.2009.176784"},{"issue":"1","key":"bibr12-09691413221144382","first-page":"172","volume":"34","author":"Philip S","year":"2017","journal-title":"Diabetic Med"},{"key":"bibr13-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1159\/000368426"},{"key":"bibr14-09691413221144382","unstructured":"Zhelev ZPJ, Rogers M, Allen M, et al. Automated grading to replace level 1 graders in the Diabetic Eye Screening Programme: External review against programme appraisal criteria for the UK National Screening Committee. UK National Screening Committee, Public Health England; 2021 12 May 2021, https:\/\/assets.publishing.service.gov.uk\/government\/uploads\/system\/uploads\/attachment_data\/file\/1035903\/Evidence_summary_AI_in_DESP_2021.pdf."},{"key":"bibr15-09691413221144382","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-155-8-201110180-00009"},{"key":"bibr16-09691413221144382","unstructured":"Yang BWP, Davenport CF, Deeks JJ, et al. Guidance on how to use QUADAS-C, version 12 March 2020, https:\/\/www.bristol.ac.uk\/media-library\/sites\/social-community-medicine\/quadas\/Guidance%20on%20how%20to%20use%20QUADAS-C.pdf."},{"key":"bibr17-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1177\/1932296816628546"},{"key":"bibr18-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1089\/dia.2007.0239"},{"key":"bibr19-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2015.08.008"},{"key":"bibr20-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1136\/bjo.2008.149807"},{"key":"bibr21-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0027524"},{"key":"bibr22-09691413221144382","doi-asserted-by":"publisher","DOI":"10.3390\/jcm10112352"},{"key":"bibr23-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1111\/j.1600-0420.2004.00350.x"},{"key":"bibr24-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1159\/000330285"},{"key":"bibr25-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1136\/bjo.2007.119453"},{"key":"bibr26-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1111\/aos.12481"},{"key":"bibr27-09691413221144382","doi-asserted-by":"publisher","DOI":"10.3310\/hta20920"},{"key":"bibr28-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1038\/s41433-019-0562-4"},{"key":"bibr29-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1007\/s00417-020-04853-y"},{"key":"bibr30-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2018.2665"},{"key":"bibr31-09691413221144382","first-page":"13","volume":"13","author":"Gulshan V","year":"2019","journal-title":"JAMA Ophthalmol"},{"key":"bibr32-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2018.01.034"},{"key":"bibr33-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0099-8"},{"key":"bibr34-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1111\/aos.14306"},{"key":"bibr35-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.18152"},{"key":"bibr36-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-25138-w"},{"key":"bibr37-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1136\/bjophthalmol-2020-316594"},{"key":"bibr38-09691413221144382","first-page":"6","volume":"6","author":"Olvera-Barrios A","year":"2020","journal-title":"Br J Ophthalmol"},{"key":"bibr39-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1167\/tvst.9.2.44"},{"key":"bibr40-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1038\/s41433-020-01366-0"},{"key":"bibr41-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-23458-5"},{"key":"bibr42-09691413221144382","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics11081385"},{"key":"bibr43-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1089\/tmj.2019.0137"},{"key":"bibr44-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1089\/dia.2019.0164"},{"key":"bibr45-09691413221144382","unstructured":"FDA. EyeArt FDA approval letter, 510(k) Number: K200667 2020."},{"key":"bibr46-09691413221144382","first-page":"17","volume":"17","author":"Liu J","year":"2020","journal-title":"Ophthalmol Retina"},{"key":"bibr47-09691413221144382","first-page":"4330","volume":"8","author":"Keel S","year":"2018","journal-title":"Sci"},{"key":"bibr48-09691413221144382","doi-asserted-by":"publisher","DOI":"10.2337\/dc18-0147"},{"key":"bibr49-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1007\/s10792-020-01685-x"},{"key":"bibr50-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.17216"},{"key":"bibr51-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1167\/iovs.16-19964"},{"key":"bibr52-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0040-6"},{"key":"bibr53-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1177\/1932296820906212"},{"key":"bibr54-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1111\/aos.13613"},{"key":"bibr55-09691413221144382","doi-asserted-by":"publisher","DOI":"10.2337\/dc18-0148"},{"key":"bibr56-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(19)30004-4"},{"key":"bibr57-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1111\/ceo.13056"},{"key":"bibr58-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1080\/02713683.2020.1764975"},{"key":"bibr59-09691413221144382","unstructured":"Larson DB, Harvey H, Rubin DL, et al. Regulatory Frameworks for Development and Evaluation of Artificial Intelligence-Based Diagnostic Imaging Algorithms: Summary and Recommendations. (1558-349X (Electronic))."},{"key":"bibr60-09691413221144382","unstructured":"Committee MSA. Technical Guidelines for preparing assessment reports for the Medical Services Advisory Committee \u2013 Service Type: Investigative (Version 3.0). In: Health AGDo, editor.: Australian Government; 2017, https:\/\/consultations.health.gov.au\/mbd\/msac-investigative-technical-guidelines\/supporting_documents\/MSAC_Investigative_Guidelines2015%20final%20for%20consultationaccessible%20150415.docx."},{"key":"bibr61-09691413221144382","first-page":"786","author":"Early Treatment Diabetic Retinopathy Study Research Group","year":"1991","journal-title":"Ophthalmology"},{"key":"bibr62-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1007\/BF00410281"},{"key":"bibr63-09691413221144382","doi-asserted-by":"publisher","DOI":"10.1016\/S0161-6420(03)00475-5"}],"container-title":["Journal of Medical Screening"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/09691413221144382","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/09691413221144382","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/09691413221144382","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T23:25:37Z","timestamp":1777677937000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/09691413221144382"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,9]]},"references-count":63,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["10.1177\/09691413221144382"],"URL":"https:\/\/doi.org\/10.1177\/09691413221144382","relation":{},"ISSN":["0969-1413","1475-5793"],"issn-type":[{"value":"0969-1413","type":"print"},{"value":"1475-5793","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,9]]}}}