{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T05:15:21Z","timestamp":1790658921048,"version":"4.1.0"},"reference-count":67,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2019,11,26]],"date-time":"2019-11-26T00:00:00Z","timestamp":1574726400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Acta Ophthalmologica"],"published-print":{"date-parts":[[2020,6]]},"abstract":"Abstract<\/jats:title>\n \n Purpose<\/jats:title>\n \n To validate the performance of a commercially available,\n CE<\/jats:styled-content>\n \u2010certified deep learning (\n DL<\/jats:styled-content>\n ) system, Ret\n CAD<\/jats:styled-content>\n v.1.3.0 (Thirona, Nijmegen, The Netherlands), for the joint automatic detection of diabetic retinopathy (\n DR<\/jats:styled-content>\n ) and age\u2010related macular degeneration (\n AMD<\/jats:styled-content>\n ) in colour fundus (\n CF<\/jats:styled-content>\n ) images on a dataset with mixed presence of eye diseases.\n <\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n Evaluation of joint detection of referable\n DR<\/jats:styled-content>\n and\n AMD<\/jats:styled-content>\n was performed on a\n DR<\/jats:styled-content>\n \u2010\n AMD<\/jats:styled-content>\n dataset with 600 images acquired during routine clinical practice, containing referable and non\u2010referable cases of both diseases. Each image was graded for\n DR<\/jats:styled-content>\n and\n AMD<\/jats:styled-content>\n by an experienced ophthalmologist to establish the reference standard (\n RS<\/jats:styled-content>\n ), and by four independent observers for comparison with human performance. Validation was furtherly assessed on Messidor (1200 images) for individual identification of referable\n DR<\/jats:styled-content>\n , and the Age\u2010Related Eye Disease Study (\n AREDS<\/jats:styled-content>\n ) dataset (133\u00a0821 images) for referable\n AMD<\/jats:styled-content>\n , against the corresponding\n RS<\/jats:styled-content>\n .\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Regarding joint validation on the\n DR<\/jats:styled-content>\n \u2010\n AMD<\/jats:styled-content>\n dataset, the system achieved an area under the\n ROC<\/jats:styled-content>\n curve (\n AUC<\/jats:styled-content>\n ) of 95.1% for detection of referable\n DR<\/jats:styled-content>\n (\n SE<\/jats:styled-content>\n \u00a0=\u00a090.1%,\n SP<\/jats:styled-content>\n \u00a0=\u00a090.6%). For referable\n AMD<\/jats:styled-content>\n , the\n AUC<\/jats:styled-content>\n was 94.9% (\n SE<\/jats:styled-content>\n \u00a0=\u00a091.8%,\n SP<\/jats:styled-content>\n \u00a0=\u00a087.5%). Average human performance for\n DR<\/jats:styled-content>\n was\n SE<\/jats:styled-content>\n \u00a0=\u00a061.5% and\n SP<\/jats:styled-content>\n \u00a0=\u00a097.8%; for\n AMD<\/jats:styled-content>\n ,\n SE<\/jats:styled-content>\n \u00a0=\u00a076.5% and\n SP<\/jats:styled-content>\n \u00a0=\u00a096.1%. Regarding detection of referable\n DR<\/jats:styled-content>\n in Messidor,\n AUC<\/jats:styled-content>\n was 97.5% (\n SE<\/jats:styled-content>\n \u00a0=\u00a092.0%,\n SP<\/jats:styled-content>\n \u00a0=\u00a092.1%); for referable\n AMD<\/jats:styled-content>\n in\n AREDS<\/jats:styled-content>\n ,\n AUC<\/jats:styled-content>\n was 92.7% (\n SE<\/jats:styled-content>\n \u00a0=\u00a085.8%,\n SP<\/jats:styled-content>\n \u00a0=\u00a086.0%).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusion<\/jats:title>\n \n The validated system performs comparably to human experts at simultaneous detection of\n DR<\/jats:styled-content>\n and\n AMD<\/jats:styled-content>\n . This shows that\n DL<\/jats:styled-content>\n systems can facilitate access to joint screening of eye diseases and become a quick and reliable support for ophthalmological experts.\n <\/jats:p>\n <\/jats:sec>","DOI":"10.1111\/aos.14306","type":"journal-article","created":{"date-parts":[[2019,11,27]],"date-time":"2019-11-27T00:21:20Z","timestamp":1574814080000},"page":"368-377","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":104,"title":["Evaluation of a deep learning system for the joint automated detection of diabetic retinopathy and age\u2010related macular degeneration"],"prefix":"10.1111","volume":"98","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8413-3297","authenticated-orcid":false,"given":"Cristina","family":"Gonz\u00e1lez\u2010Gonzalo","sequence":"first","affiliation":[{"name":"A\u2010eye Research Group Radboud University Medical Center Nijmegen The Netherlands"},{"name":"Diagnostic Image Analysis Group Radboud University Medical Center Nijmegen The Netherlands"},{"name":"Donders Institute for Brain Cognition and Behaviour Radboud University Medical Center Nijmegen The Netherlands"},{"name":"Department of Ophthalmology Radboud University Medical Center Nijmegen The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ver\u00f3nica","family":"S\u00e1nchez\u2010Guti\u00e9rrez","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology University Hospital Ram\u00f3n y Cajal Ram\u00f3n y Cajal Health Research Institute (IRYCIS) Madrid Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paula","family":"Hern\u00e1ndez\u2010Mart\u00ednez","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology University Hospital Ram\u00f3n y Cajal Ram\u00f3n y Cajal Health Research Institute (IRYCIS) Madrid Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3524-9335","authenticated-orcid":false,"given":"In\u00e9s","family":"Contreras","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology University Hospital Ram\u00f3n y Cajal Ram\u00f3n y Cajal Health Research Institute (IRYCIS) Madrid Spain"},{"name":"Cl\u00ednica Rementer\u00eda Madrid Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yara T.","family":"Lechanteur","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology Radboud University Medical Center Nijmegen The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Artin","family":"Domanian","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology Radboud University Medical Center Nijmegen The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bram","family":"van Ginneken","sequence":"additional","affiliation":[{"name":"Diagnostic Image Analysis Group Radboud University Medical Center Nijmegen The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clara I.","family":"S\u00e1nchez","sequence":"additional","affiliation":[{"name":"A\u2010eye Research Group Radboud University Medical Center Nijmegen The Netherlands"},{"name":"Diagnostic Image Analysis Group Radboud University Medical Center Nijmegen The Netherlands"},{"name":"Donders Institute for Brain Cognition and Behaviour Radboud University Medical Center Nijmegen The Netherlands"},{"name":"Department of Ophthalmology Radboud University Medical Center Nijmegen The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2019,11,26]]},"reference":[{"key":"e_1_2_5_2_1","unstructured":"AAO(2015):PPP Retina\/Vitreous Panel Hoskins Center for Quality Eye Care. Preferred practice pattern: age\u2010related macular degeneration \u2013 Updated 2015. Available at:https:\/\/www.aao.org\/preferred-practice-pattern\/age-related-macular-degeneration-ppp-2015. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_3_1","unstructured":"AAO(2017):Retina\/Vitreous Panel Hoskins Center for Quality Eye Care. Preferred practice pattern: diabetic retinopathy \u2010 Updated 2017. Available at:https:\/\/www.aao.org\/preferred-practice-pattern\/diabetic-retinopathy-ppp-updated-2017. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_4_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaophthalmol.2013.1743"},{"key":"e_1_2_5_5_1","doi-asserted-by":"publisher","DOI":"10.1167\/iovs.16-19964"},{"key":"e_1_2_5_6_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0040-6"},{"key":"e_1_2_5_7_1","unstructured":"ADCIS.Messidor dataset. Available at:http:\/\/www.adcis.net\/en\/DownloadThirdParty\/Messidor.html. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_8_1","unstructured":"AntonyM&Br\u00fcggemannS(2015):Team o_O Solution. Available at:https:\/\/www.kaggle.com\/c\/diabetic-retinopathy-detection\/discussion\/15617. (Accessed on 30 June 2019)."},{"key":"e_1_2_5_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0002-9394(01)01218-1"},{"key":"e_1_2_5_10_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1464-5491.2003.t01-1-00899.x"},{"key":"e_1_2_5_11_1","doi-asserted-by":"crossref","unstructured":"BurlinaP FreundDE DupasB&BresslerN(2011):Automatic screening of age\u2010related macular degeneration and retinal abnormalities. In: Engineering in Medicine and Biology Society EMBC 2011 Annual International Conference of the IEEE:3962\u20133966.","DOI":"10.1109\/IEMBS.2011.6090984"},{"key":"e_1_2_5_12_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaophthalmol.2017.3782"},{"key":"e_1_2_5_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2015.06.050"},{"key":"e_1_2_5_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2015.08.007"},{"key":"e_1_2_5_15_1","doi-asserted-by":"crossref","unstructured":"CholletF(2017):Xception: Deep learning with depthwise separable convolutions. In Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit:1251\u20131258.","DOI":"10.1109\/CVPR.2017.195"},{"key":"e_1_2_5_16_1","doi-asserted-by":"publisher","DOI":"10.2165\/00002512-200421150-00002"},{"key":"e_1_2_5_17_1","doi-asserted-by":"publisher","DOI":"10.1177\/193229680900300315"},{"key":"e_1_2_5_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1096-9136(1998120)15:4 3.0.CO;2-S"},{"key":"e_1_2_5_19_1","doi-asserted-by":"publisher","DOI":"10.5566\/ias.1155"},{"key":"e_1_2_5_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-4541-9_15"},{"key":"e_1_2_5_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2017.02.008"},{"key":"e_1_2_5_22_1","doi-asserted-by":"publisher","DOI":"10.1001\/archophthalmol.2010.18"},{"key":"e_1_2_5_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2018.02.037"},{"key":"e_1_2_5_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.diabres.2013.11.002"},{"key":"e_1_2_5_25_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.17216"},{"key":"e_1_2_5_26_1","unstructured":"HarmonD&MerrittJ(2009):Demand for ophthalmic services and ophthalmologists\u2013a resource assessment. In: A Study Prepared by Market Scope. LLC 1\u20102."},{"key":"e_1_2_5_27_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0266462308080288"},{"key":"e_1_2_5_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1532-0464(02)00500-2"},{"key":"e_1_2_5_29_1","doi-asserted-by":"crossref","unstructured":"HuangG LiuZ Van Der MaatenL&WeinbergerKQ(2017):Densely connected convolutional networks. In Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit:4700\u20134708.","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_2_5_30_1","unstructured":"IAPB(2016):Diabetic retinopathy. Available at:https:\/\/www.iapb.org\/knowledge\/what-is-avoidable-blindness\/diabetic-retinopathy. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_31_1","unstructured":"IDF(2017):IDF Diabetes Atlas 8thEdition 2017. Available at:http:\/\/www.diabetesatlas.org. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_32_1","doi-asserted-by":"publisher","DOI":"10.1038\/sj.eye.6701916"},{"key":"e_1_2_5_33_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.320.7250.1627"},{"key":"e_1_2_5_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1464-5491.2009.02870.x"},{"key":"e_1_2_5_35_1","unstructured":"KarnonJ Czoski\u2010MurrayC SmithKet\u00a0al. (2008):A preliminary model\u2010based assessment of the cost\u2013utility of a screening programme for early age\u2010related macular degeneration. In: NIHR Health Technology Assessment programme: Executive Summaries. NIHR Journals Library."},{"key":"e_1_2_5_36_1","unstructured":"KrizhevskyA SutskeverI&HintonGE(2012):Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems:1097\u20131105."},{"key":"e_1_2_5_37_1","unstructured":"LaTIM(2011):Messidor\u20102 dataset. Available at:http:\/\/latim.univ-brest.fr\/indexfce0.html. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_5_39_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_2_5_40_1","doi-asserted-by":"publisher","DOI":"10.2337\/dc18-0147"},{"key":"e_1_2_5_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_2_5_42_1","unstructured":"NEI.AREDS dbGaP Data Tables: A User's Guide. Version 1.0. Available at:https:\/\/www.ncbi.nlm.nih.gov\/projects\/gap\/cgi-bin\/GetPdf.cgi?id=phd001552.1. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_43_1","unstructured":"NEI.AREDS dbGaP Study Accession. Available at:https:\/\/www.ncbi.nlm.nih.gov\/projects\/gap\/cgi-bin\/study.cgi?study_id=phs000001.v3.p1. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_44_1","unstructured":"NEI.AREDS Manual of Operations Study Design. Section 3.1.2.8. Available at:https:\/\/www.ncbi.nlm.nih.gov\/projects\/gap\/cgi-bin\/GetPdf.cgi?id=phd000003.2. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_45_1","unstructured":"NHS(2014):Diabetic Eye Screening Programme. Available at:https:\/\/www.gov.uk\/topic\/population-screening-programmes\/diabetic-eye. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_46_1","unstructured":"NOG(2017):Diabetic retinopathy multidisciplinary guideline authorized November 2017. Available at:https:\/\/www.oogheelkunde.org\/richtlijn\/diabetische-retinopathie-multidisciplinaire-richtlijn-geautoriseerd-November-2017. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_47_1","doi-asserted-by":"publisher","DOI":"10.1167\/iovs.12-10727"},{"key":"e_1_2_5_48_1","doi-asserted-by":"publisher","DOI":"10.1136\/bjophthalmol-2011-300539"},{"key":"e_1_2_5_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2018.11.015"},{"key":"e_1_2_5_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.optom.2013.03.004"},{"key":"e_1_2_5_51_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41433-018-0269-y"},{"key":"e_1_2_5_52_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.291.12.1487"},{"key":"e_1_2_5_53_1","doi-asserted-by":"publisher","DOI":"10.1167\/iovs.10-6633"},{"key":"e_1_2_5_54_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.preteyeres.2018.07.004"},{"key":"e_1_2_5_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.diabres.2009.10.007"},{"key":"e_1_2_5_56_1","article-title":"Very deep convolutional networks for large\u2010scale image recognition","author":"Simonyan K","year":"2014","journal-title":"arXiv"},{"key":"e_1_2_5_57_1","doi-asserted-by":"crossref","unstructured":"SzegedyC LiuW JiaYet\u00a0al. (2015):Going deeper with convolutions. In Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit:1\u20139.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_2_5_58_1","doi-asserted-by":"crossref","unstructured":"SzegedyC IoffeS VanhouckeV&AlemiAA(2017):Inception\u2010v4 inception\u2010resnet and the impact of residual connections on learning. In AAAI4:12.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"e_1_2_5_59_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.18152"},{"key":"e_1_2_5_60_1","unstructured":"United Nations Department of Economic and Social Affairs(2017):World Population Aging Report 2017. Available at:http:\/\/www.un.org\/en\/development\/desa\/population\/publications\/. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_61_1","unstructured":"University of Iowa Health Care Department of Ophthalmology and Visual Sciences Michael D. Abr\u00e0moff MD PhD(2017). Available at:https:\/\/medicine.uiowa.edu\/eye\/abramoff. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_62_1","unstructured":"WHO(2013):Global action plan for the prevention of avoidable blindness and visual impairment 2014\u20132019 \u2013 towards universal eye health. Available at:https:\/\/www.iapb.org\/advocacy\/global-action-plan-2014-2019. (Accessed on 4 March 2019)."},{"key":"e_1_2_5_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0161-6420(03)00475-5"},{"key":"e_1_2_5_64_1","doi-asserted-by":"publisher","DOI":"10.1016\/S2214-109X(13)70145-1"},{"key":"e_1_2_5_65_1","first-page":"29","article-title":"Epidemiology in practice: screening for eye disease","volume":"12","author":"Wormald R","year":"1999","journal-title":"Community Eye Health"},{"key":"e_1_2_5_66_1","doi-asserted-by":"publisher","DOI":"10.2337\/dc11-1909"},{"key":"e_1_2_5_67_1","doi-asserted-by":"publisher","DOI":"10.1097\/IAE.0b013e3181f57e30"},{"key":"e_1_2_5_68_1","doi-asserted-by":"publisher","DOI":"10.1093\/clinchem\/39.4.561"}],"container-title":["Acta Ophthalmologica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/aos.14306","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/aos.14306","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/aos.14306","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T00:05:10Z","timestamp":1693958710000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/aos.14306"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,26]]},"references-count":67,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["10.1111\/aos.14306"],"URL":"https:\/\/doi.org\/10.1111\/aos.14306","archive":["Portico"],"relation":{},"ISSN":["1755-375X","1755-3768"],"issn-type":[{"value":"1755-375X","type":"print"},{"value":"1755-3768","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,26]]},"assertion":[{"value":"2019-04-09","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-31","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-11-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}