{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T17:10:15Z","timestamp":1782753015013,"version":"3.54.5"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,4,10]],"date-time":"2019-04-10T00:00:00Z","timestamp":1554854400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,4,10]],"date-time":"2019-04-10T00:00:00Z","timestamp":1554854400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Deep learning algorithms have been used to detect diabetic retinopathy (DR) with specialist-level accuracy. This study aims to validate one such algorithm on a large-scale clinical population, and compare the algorithm performance with that of human graders. A total of 25,326 gradable retinal images of patients with diabetes from the community-based, nationwide screening program of DR in Thailand were analyzed for DR severity and referable diabetic macular edema (DME). Grades adjudicated by a panel of international retinal specialists served as the reference standard. Relative to human graders, for detecting referable DR (moderate NPDR or worse), the deep learning algorithm had significantly higher sensitivity (0.97 vs. 0.74, <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001), and a slightly lower specificity (0.96 vs. 0.98, <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001). Higher sensitivity of the algorithm was also observed for each of the categories of severe or worse NPDR, PDR, and DME (<jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001 for all comparisons). The quadratic-weighted kappa for determination of DR severity levels by the algorithm and human graders was 0.85 and 0.78 respectively (<jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001 for the difference). Across different severity levels of DR for determining referable disease, deep learning significantly reduced the false negative rate (by 23%) at the cost of slightly higher false positive rates (2%). Deep learning algorithms may serve as a valuable tool for DR screening.<\/jats:p>","DOI":"10.1038\/s41746-019-0099-8","type":"journal-article","created":{"date-parts":[[2019,4,10]],"date-time":"2019-04-10T10:18:48Z","timestamp":1554891528000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":168,"title":["Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program"],"prefix":"10.1038","volume":"2","author":[{"given":"Paisan","family":"Ruamviboonsuk","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Krause","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peranut","family":"Chotcomwongse","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rory","family":"Sayres","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajiv","family":"Raman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kasumi","family":"Widner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bilson J. L.","family":"Campana","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sonia","family":"Phene","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kornwipa","family":"Hemarat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3337-0625","authenticated-orcid":false,"given":"Mongkol","family":"Tadarati","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sukhum","family":"Silpa-Archa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jirawut","family":"Limwattanayingyong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chetan","family":"Rao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oscar","family":"Kuruvilla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jesse","family":"Jung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9939-9506","authenticated-orcid":false,"given":"Jeffrey","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Surapong","family":"Orprayoon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chawawat","family":"Kangwanwongpaisan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ramase","family":"Sukumalpaiboon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chainarong","family":"Luengchaichawang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jitumporn","family":"Fuangkaew","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pipat","family":"Kongsap","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lamyong","family":"Chualinpha","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarawuth","family":"Saree","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Srirut","family":"Kawinpanitan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Korntip","family":"Mitvongsa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Siriporn","family":"Lawanasakol","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chaiyasit","family":"Thepchatri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lalita","family":"Wongpichedchai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Greg S.","family":"Corrado","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lily","family":"Peng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dale R.","family":"Webster","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,4,10]]},"reference":[{"key":"99_CR1","doi-asserted-by":"publisher","first-page":"2402","DOI":"10.1001\/jama.2016.17216","volume":"316","author":"V Gulshan","year":"2016","unstructured":"Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402\u20132410 (2016).","journal-title":"JAMA"},{"key":"99_CR2","doi-asserted-by":"publisher","first-page":"1264","DOI":"10.1016\/j.ophtha.2018.01.034","volume":"125","author":"J Krause","year":"2018","unstructured":"Krause, J. et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 1264\u20131272 (2018).","journal-title":"Ophthalmology"},{"key":"99_CR3","doi-asserted-by":"publisher","first-page":"2211","DOI":"10.1001\/jama.2017.18152","volume":"318","author":"DSW Ting","year":"2017","unstructured":"Ting, D. S. W. et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318, 2211\u20132223 (2017).","journal-title":"JAMA"},{"key":"99_CR4","first-page":"10","volume":"21","author":"W Jenchitr","year":"2007","unstructured":"Jenchitr, W. et al. The national survey of blindness low vision and visual impairment in thailand 2006\u20132007. Thai. J. Pub. Hlth. Ophthalmol. 21, 10 (2007).","journal-title":"Thai. J. Pub. Hlth. Ophthalmol."},{"key":"99_CR5","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0114245","volume":"9","author":"S Isipradit","year":"2014","unstructured":"Isipradit, S. et al. The First Rapid Assessment of Avoidable Blindness (RAAB) in Thailand. PLoS ONE 9, e114245 (2014).","journal-title":"PLoS ONE"},{"key":"99_CR6","unstructured":"Meenakshi Kumar, et al. Perceived barriers for diabetic retinopathy screening by the staff involved in screening in Thailand."},{"key":"99_CR7","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1136\/bmj.323.7303.4","volume":"323","author":"JV Ulrich Freudenstein","year":"2001","unstructured":"Ulrich Freudenstein, J. V. A national screening programme for diabetic retinopathy: needs to learn the lessons of existing screening programmes. BMJ 323, 4 (2001).","journal-title":"BMJ"},{"key":"99_CR8","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1001\/jamaophthalmol.2015.5332","volume":"134","author":"YT Wang","year":"2016","unstructured":"Wang, YT., Tadarati, M., Wolfson, Y., Bressler, S. B. & Bressler, N. M. Comparison of prevalence of diabetic macular edema based on monocular fundus photography vs optical coherence tomography. JAMA Ophthalmol. 134, 222\u2013228 (2016).","journal-title":"JAMA Ophthalmol."},{"key":"99_CR9","doi-asserted-by":"publisher","first-page":"597","DOI":"10.1016\/j.ajo.2004.10.064","volume":"139","author":"AA Cavallerano","year":"2005","unstructured":"Cavallerano, A. A. et al. A telemedicine program for diabetic retinopathy in a Veterans Affairs Medical Center\u2013the Joslin Vision Network Eye Health Care Model. Am. J. Ophthalmol. 139, 597\u2013604 (2005).","journal-title":"Am. J. Ophthalmol."},{"key":"99_CR10","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1682\/JRRD.2014.03.0077","volume":"52","author":"GL Tsan","year":"2015","unstructured":"Tsan, G. L. Assessment of diabetic teleretinal imaging program at the Portland Department of Veterans Affairs Medical Center. J. Rehabil. Res. Dev. 52, 193\u2013200 (2015).","journal-title":"J. Rehabil. Res. Dev."},{"key":"99_CR11","unstructured":"City & Guilds. Diabetic Retinopathy Screening qualifications and training courses. https:\/\/www.cityandguilds.com\/qualifications-and-apprenticeships\/health-and-social-care\/health\/7360-diabetic-retinopathy-screening#tab=information (2018). (Accessed 21 September 2018)."},{"key":"99_CR12","doi-asserted-by":"publisher","first-page":"1831","DOI":"10.2337\/diacare.26.6.1831","volume":"26","author":"P Namperumalsamy","year":"2003","unstructured":"Namperumalsamy, P., Nirmalan, P. K. & Ramasamy, K. Developing a screening program to detect sight-threatening diabetic retinopathy in South India. Diabetes Care 26, 1831\u20131835 (2003).","journal-title":"Diabetes Care"},{"key":"99_CR13","unstructured":"CPG Secretariat. Screening of Diabetic Retinopathy. Portal Rasmi Kemeterian Kesihatan Malaysia. www.moh.gov.my\/penerbitan\/CPG2017\/6601.pdf (2001)."},{"key":"99_CR14","doi-asserted-by":"publisher","first-page":"700","DOI":"10.7196\/SAMJ.8678","volume":"104","author":"S Cook","year":"2014","unstructured":"Cook, S. et al. Quality assurance in diabetic retinal screening in South Africa. S. Afr. Med. J. 104, 700\u2013704 (2014).","journal-title":"S. Afr. Med. J."},{"key":"99_CR15","doi-asserted-by":"publisher","first-page":"2571","DOI":"10.1016\/j.ophtha.2016.08.021","volume":"123","author":"HV Nguyen","year":"2016","unstructured":"Nguyen, H. V. Cost-effectiveness of a National Telemedicine Diabetic Retinopathy Screening Program in Singapore. Ophthalmology 123, 2571\u20132580 (2016).","journal-title":"Ophthalmology"},{"key":"99_CR16","doi-asserted-by":"publisher","DOI":"10.1136\/bmjgh-2018-000766","volume":"3","author":"IYH Wong","year":"2018","unstructured":"Wong, I. Y. H., Ni, M. Y., Wong, I. O. L., Fong, N. & Leung, G. M. Saving sight in China and beyond: the Lifeline Express model. BMJ Glob. Health 3, e000766 (2018).","journal-title":"BMJ Glob. Health"},{"key":"99_CR17","unstructured":"American Academy of Ophthalmology. Diabetic Retinopathy PPP - Updated 2017 (American Academy of Ophthalmology, 2017). https:\/\/www.aao.org\/preferred-practice-pattern\/diabetic-retinopathy-ppp-updated-2017."},{"key":"99_CR18","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1586\/eop.12.52","volume":"7","author":"R Chakrabarti","year":"2012","unstructured":"Chakrabarti, R., Harper, C. A. & Keeffe, J. E. Diabetic retinopathy management guidelines. Expert Rev. Ophthalmol. 7, 417\u2013439 (2012).","journal-title":"Expert Rev. Ophthalmol."},{"key":"99_CR19","doi-asserted-by":"publisher","first-page":"1608","DOI":"10.1016\/j.ophtha.2018.04.007","volume":"125","author":"TY Wong","year":"2018","unstructured":"Wong, T. Y. et al. Guidelines on Diabetic Eye Care: The International Council of Ophthalmology Recommendations for Screening, Follow-up, Referral, and Treatment Based on Resource Settings. Ophthalmology 125, 1608\u20131622 (2018).","journal-title":"Ophthalmology"},{"key":"99_CR20","unstructured":"American Academy of Ophthalmology. International Council of Ophthalmology: Resources. International Clinical Diabetic Retinopathy Disease Severity Scale, Detailed Table. http:\/\/www.icoph.org\/resources\/45\/International-Clinical-Diabetic-Retinopathy-Disease-Severity-Scale-Detailed-Table-.html (2010)."},{"key":"99_CR21","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1007\/s11892-017-0928-6","volume":"17","author":"PH Scanlon","year":"2017","unstructured":"Scanlon, P. H. Screening intervals for diabetic retinopathy and implications for care. Curr. Diab. Rep. 17, 96 (2017).","journal-title":"Curr. Diab. Rep."},{"key":"99_CR22","first-page":"CD008081","volume":"7","author":"G Virgili","year":"2011","unstructured":"Virgili, G. et al. Optical coherence tomography (OCT) for detection of macular oedema in patients with diabetic retinopathy. Cochrane Database Syst. Rev. 7, CD008081 (2011).","journal-title":"Cochrane Database Syst. Rev."},{"key":"99_CR23","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1016\/S0161-6420(99)00010-X","volume":"107","author":"GH Bresnick","year":"2000","unstructured":"Bresnick, G. H., Mukamel, D. B., Dickinson, J. C. & Cole, D. R. A screening approach to the surveillance of patients with diabetes for the presence of vision-threatening retinopathy. Ophthalmology 107, 19\u201324 (2000).","journal-title":"Ophthalmology"},{"key":"99_CR24","doi-asserted-by":"publisher","first-page":"814","DOI":"10.1089\/tmj.2011.0075","volume":"17","author":"HK Li","year":"2011","unstructured":"Li, H. K. et al. Telehealth practice recommendations for diabetic retinopathy, second edition. Telemed. e-Health 17, 814\u2013837 (2011).","journal-title":"Telemed. e-Health"},{"key":"99_CR25","first-page":"716","volume":"91","author":"S Supapluksakul","year":"2008","unstructured":"Supapluksakul, S., Ruamviboonsuk, P. & Chaowakul, W. The prevalence of diabetic retinopathy in Trang province determined by retinal photography and comprehensive eye examination. J. Med. Assoc. Thai. 91, 716\u2013722 (2008).","journal-title":"J. Med. Assoc. Thai."},{"key":"99_CR26","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V. & Alemi, A. Inception-v4, Inception-ResNet and the impact of residual connections on learning. Thirty-First AAAI Conference on Artificial Intelligence (AAAI, San Francisco, 2016).","DOI":"10.1609\/aaai.v31i1.11231"}],"updated-by":[{"DOI":"10.1038\/s41746-019-0146-5","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2019,7,23]],"date-time":"2019-07-23T00:00:00Z","timestamp":1563840000000}}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0099-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0099-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0099-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T18:28:56Z","timestamp":1671301736000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0099-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,10]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["99"],"URL":"https:\/\/doi.org\/10.1038\/s41746-019-0099-8","relation":{"correction":[{"id-type":"doi","id":"10.1038\/s41746-019-0146-5","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,4,10]]},"assertion":[{"value":"26 October 2018","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 March 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 April 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 July 2019","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"An amendment to this paper has been published and can be accessed via a link at the top of the paper.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"J.K., R.S., K.W., B.J.L.C., G.S.C., L.P., S.P. and D.R.W. are Google employees and receive salary and stock as a part of the standard compensation package. O.K., J.J. and J.T. are consultants for Google.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"25"}}