{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,3]],"date-time":"2026-02-03T18:52:55Z","timestamp":1770144775752,"version":"3.49.0"},"reference-count":35,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,1,3]],"date-time":"2023-01-03T00:00:00Z","timestamp":1672704000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Singapore National Medical Research Council","award":["CIRG18Nov-0013"],"award-info":[{"award-number":["CIRG18Nov-0013"]}]},{"name":"Singapore National Medical Research Council","award":["05\/FY2019\/P2\/06-A60"],"award-info":[{"award-number":["05\/FY2019\/P2\/06-A60"]}]},{"name":"SingHealth Duke-NUS Medical School, Ophthalmology and Visual Sciences Academic Clinical Program","award":["CIRG18Nov-0013"],"award-info":[{"award-number":["CIRG18Nov-0013"]}]},{"name":"SingHealth Duke-NUS Medical School, Ophthalmology and Visual Sciences Academic Clinical Program","award":["05\/FY2019\/P2\/06-A60"],"award-info":[{"award-number":["05\/FY2019\/P2\/06-A60"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>The quality of ocular fundus photographs can affect the accuracy of the morphologic assessment of the optic nerve head (ONH), either by humans or by deep learning systems (DLS). In order to automatically identify ONH photographs of optimal quality, we have developed, trained, and tested a DLS, using an international, multicentre, multi-ethnic dataset of 5015 ocular fundus photographs from 31 centres in 20 countries participating to the Brain and Optic Nerve Study with Artificial Intelligence (BONSAI). The reference standard in image quality was established by three experts who independently classified photographs as of \u201cgood\u201d, \u201cborderline\u201d, or \u201cpoor\u201d quality. The DLS was trained on 4208 fundus photographs and tested on an independent external dataset of 807 photographs, using a multi-class model, evaluated with a one-vs-rest classification strategy. In the external-testing dataset, the DLS could identify with excellent performance \u201cgood\u201d quality photographs (AUC = 0.93 (95% CI, 0.91\u20130.95), accuracy = 91.4% (95% CI, 90.0\u201392.9%), sensitivity = 93.8% (95% CI, 92.5\u201395.2%), specificity = 75.9% (95% CI, 69.7\u201382.1%) and \u201cpoor\u201d quality photographs (AUC = 1.00 (95% CI, 0.99\u20131.00), accuracy = 99.1% (95% CI, 98.6\u201399.6%), sensitivity = 81.5% (95% CI, 70.6\u201393.8%), specificity = 99.7% (95% CI, 99.6\u2013100.0%). \u201cBorderline\u201d quality images were also accurately classified (AUC = 0.90 (95% CI, 0.88\u20130.93), accuracy = 90.6% (95% CI, 89.1\u201392.2%), sensitivity = 65.4% (95% CI, 56.6\u201372.9%), specificity = 93.4% (95% CI, 92.1\u201394.8%). The overall accuracy to distinguish among the three classes was 90.6% (95% CI, 89.1\u201392.1%), suggesting that this DLS could select optimal quality fundus photographs in patients with neuro-ophthalmic and neurological disorders affecting the ONH.<\/jats:p>","DOI":"10.3390\/diagnostics13010160","type":"journal-article","created":{"date-parts":[[2023,1,4]],"date-time":"2023-01-04T01:42:44Z","timestamp":1672796564000},"page":"160","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["A Deep Learning System for Automated Quality Evaluation of Optic Disc Photographs in Neuro-Ophthalmic Disorders"],"prefix":"10.3390","volume":"13","author":[{"given":"Ebenezer","family":"Chan","sequence":"first","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 169856, Singapore"},{"name":"Duke-NUS School of Medicine, Singapore 169857, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiqun","family":"Tang","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 169856, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3770-2300","authenticated-orcid":false,"given":"Raymond P.","family":"Najjar","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 169856, Singapore"},{"name":"Duke-NUS School of Medicine, Singapore 169857, Singapore"},{"name":"Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 117597, Singapore"},{"name":"Center for Innovation & Precision Eye Health, National University of Singapore, Singapore 119077, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arun","family":"Narayanaswamy","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 169856, Singapore"},{"name":"Glaucoma Department, Singapore National Eye Centre, Singapore 168751, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kanchalika","family":"Sathianvichitr","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 169856, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nancy J.","family":"Newman","sequence":"additional","affiliation":[{"name":"Departments of Ophthalmology and Neurology, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Val\u00e9rie","family":"Biousse","sequence":"additional","affiliation":[{"name":"Departments of Ophthalmology and Neurology, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6388-8639","authenticated-orcid":false,"given":"Dan","family":"Milea","sequence":"additional","affiliation":[{"name":"Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 169856, Singapore"},{"name":"Duke-NUS School of Medicine, Singapore 169857, Singapore"},{"name":"Department of Ophthalmology, Rigshospitalet, University of Copenhagen, 2600 Copenhagen, Denmark"},{"name":"Department of Ophthalmology, Angers University Hospital, 49100 Angers, France"},{"name":"Neuro-Ophthalmology Department, Singapore National Eye Centre, Singapore 168751, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"name":"for the BONSAI Group","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1701","DOI":"10.1016\/j.ophtha.2012.01.044","article-title":"Neuroimaging in patients referred to a neuro-ophthalmology service: The rates of appropriateness and concordance in interpretation","volume":"119","author":"McClelland","year":"2012","journal-title":"Ophthalmology"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1007\/s11910-022-01189-4","article-title":"Diagnostic Error in Neuro-ophthalmology: Avenues to Improve","volume":"22","author":"Stunkel","year":"2022","journal-title":"Curr. Neurol. Neurosci. Rep."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1097\/WNO.0000000000000846","article-title":"Referral patterns in neuro-ophthalmology","volume":"40","author":"Stunkel","year":"2020","journal-title":"J. Neuro-Ophthalmol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ahn, J.M., Kim, S., Ahn, K.-S., Cho, S.-H., and Kim, U.S. (2019). Accuracy of machine learning for differentiation between optic neuropathies and pseudopapilledema. BMC Ophthalmol., 19.","DOI":"10.1186\/s12886-019-1184-0"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"16685","DOI":"10.1038\/s41598-018-35044-9","article-title":"Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs","volume":"8","author":"Christopher","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1353","DOI":"10.1001\/jamaophthalmol.2019.3501","article-title":"Development and validation of a deep learning system to detect glaucomatous optic neuropathy using fundus photographs","volume":"137","author":"Liu","year":"2019","journal-title":"JAMA Ophthalmol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1687","DOI":"10.1056\/NEJMoa1917130","article-title":"Artificial intelligence to detect papilledema from ocular fundus photographs","volume":"382","author":"Milea","year":"2020","journal-title":"N. Engl. J. Med."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"785","DOI":"10.1002\/ana.25839","article-title":"Optic disc classification by deep learning versus expert neuro-ophthalmologists","volume":"88","author":"Biousse","year":"2020","journal-title":"Ann. Neurol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1120","DOI":"10.1167\/iovs.05-1155","article-title":"Automated assessment of diabetic retinal image quality based on clarity and field definition","volume":"47","author":"Fleming","year":"2006","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1046\/j.1464-5491.2003.00954.x","article-title":"The effectiveness of screening for diabetic retinopathy by digital imaging photography and technician ophthalmoscopy","volume":"20","author":"Scanlon","year":"2003","journal-title":"Diabet. Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1136\/bjo.2004.051334","article-title":"The impact of the Health Technology Board for Scotland\u2019s grading model on referrals to ophthalmology services","volume":"89","author":"Philip","year":"2005","journal-title":"Br. J. Ophthalmol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1089\/tmj.2006.12.89","article-title":"Results of implementation of the DigiScope for diabetic retinopathy assessment in the primary care environment","volume":"12","author":"Zeimer","year":"2006","journal-title":"Telemed. J. e-Health"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"668","DOI":"10.1089\/tmj.2005.11.668","article-title":"Web-based screening for diabetic retinopathy in a primary care population: The EyeCheck project","volume":"11","author":"Abramoff","year":"2005","journal-title":"Telemed. J. e-Health"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1211","DOI":"10.1049\/iet-ipr.2018.6212","article-title":"Fundus image quality assessment: Survey, challenges, and future scope","volume":"13","author":"Raj","year":"2019","journal-title":"IET Image Process."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1109\/TIP.2017.2760518","article-title":"Deep neural networks for no-reference and full-reference image quality assessment","volume":"27","author":"Bosse","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Mahapatra, D., Roy, P.K., Sedai, S., and Garnavi, R. (2016, January 16\u201320). A cnn based neurobiology inspired approach for retinal image quality assessment. Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7590946"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"869","DOI":"10.1007\/s10278-018-0084-9","article-title":"Automated quality assessment of colour fundus images for diabetic retinopathy screening in telemedicine","volume":"31","author":"Saha","year":"2018","journal-title":"J. Digit. Imaging"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.compbiomed.2018.10.004","article-title":"Retinal image quality assessment using deep learning","volume":"103","author":"Zago","year":"2018","journal-title":"Comput. Biol. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1016\/j.compbiomed.2019.03.019","article-title":"Quality and content analysis of fundus images using deep learning","volume":"108","author":"Chalakkal","year":"2019","journal-title":"Comput. Biol. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"101654","DOI":"10.1016\/j.media.2020.101654","article-title":"Domain-invariant interpretable fundus image quality assessment","volume":"61","author":"Shen","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"57810","DOI":"10.1109\/ACCESS.2020.2982588","article-title":"Multivariate regression-based convolutional neural network model for fundus image quality assessment","volume":"8","author":"Raj","year":"2020","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"419","DOI":"10.2147\/OPTH.S235751","article-title":"Artificial intelligence to identify retinal fundus images, quality validation, laterality evaluation, macular degeneration, and suspected glaucoma","volume":"14","author":"Zapata","year":"2020","journal-title":"Clin. Ophthalmol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1167\/tvst.10.11.16","article-title":"Deep-Learning\u2013Based Pre-Diagnosis Assessment Module for Retinal Photographs: A Multicenter Study","volume":"10","author":"Yuen","year":"2021","journal-title":"Transl. Vis. Sci. Technol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1199","DOI":"10.1016\/j.ophtha.2018.01.023","article-title":"Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs","volume":"125","author":"Li","year":"2018","journal-title":"Ophthalmology"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1097\/APO.0000000000000512","article-title":"Artificial intelligence meets neuro-ophthalmology","volume":"11","author":"Leong","year":"2022","journal-title":"Asia-Pac. J. Ophthalmol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1097\/APO.0000000000000404","article-title":"Deep learning for retinal image quality assessment of optic nerve head disorders","volume":"10","author":"Chan","year":"2021","journal-title":"Asia-Pac. J. Ophthalmol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e1415","DOI":"10.1111\/aos.14850","article-title":"Handheld fundus camera performance, image quality and outcomes of diabetic retinopathy grading in a pilot screening study","volume":"99","author":"Kubin","year":"2021","journal-title":"Acta Ophthalmol."},{"key":"ref_28","unstructured":"Milea, L., and Najjar, R. Classif-Eye: A semi-automated image classification application. 2020. GitHub repository., 2020."},{"key":"ref_29","unstructured":"Tan, M., and Le, Q. (2019, January 9\u201315). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Software, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"4522","DOI":"10.1093\/bioinformatics\/btz259","article-title":"Biomedical image augmentation using Augmentor","volume":"35","author":"Bloice","year":"2019","journal-title":"Bioinformatics"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1038\/nmeth.3968","article-title":"Points of significance: Model selection and overfitting","volume":"13","author":"Lever","year":"2016","journal-title":"Nat. Methods"},{"key":"ref_33","unstructured":"Fu, H., Wang, B., Shen, J., Cui, S., Xu, Y., Liu, J., and Shao, L. Evaluation of retinal image quality assessment networks in different color-spaces. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., and Vardoulakis, L.M. (2020, January 25\u201330). A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3313831.3376718"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1159\/000475773","article-title":"Predictors of photographic quality with a handheld nonmydriatic fundus camera used for screening of vision-threatening diabetic retinopathy","volume":"238","author":"Davila","year":"2017","journal-title":"Ophthalmologica"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/13\/1\/160\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T17:57:26Z","timestamp":1760119046000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/13\/1\/160"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,3]]},"references-count":35,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["diagnostics13010160"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics13010160","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,3]]}}}