{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T15:56:56Z","timestamp":1784390216240,"version":"3.55.0"},"reference-count":41,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,12,31]],"date-time":"2023-12-31T00:00:00Z","timestamp":1703980800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"University of La Laguna"},{"name":"Spanish Ministry of Science, Innovation and Universities"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Glaucoma, a leading cause of blindness, damages the optic nerve, making early diagnosis challenging due to no initial symptoms. Fundus eye images taken with a non-mydriatic retinograph help diagnose glaucoma by revealing structural changes, including the optic disc and cup. This research aims to thoroughly analyze saliency maps in interpreting convolutional neural network decisions for diagnosing glaucoma from fundus images. These maps highlight the most influential image regions guiding the network\u2019s decisions. Various network architectures were trained and tested on 739 optic nerve head images, with nine saliency methods used. Some other popular datasets were also used for further validation. The results reveal disparities among saliency maps, with some consensus between the folds corresponding to the same architecture. Concerning the significance of optic disc sectors, there is generally a lack of agreement with standard medical criteria. The background, nasal, and temporal sectors emerge as particularly influential for neural network decisions, showing a likelihood of being the most relevant ranging from 14.55% to 28.16% on average across all evaluated datasets. We can conclude that saliency maps are usually difficult to interpret and even the areas indicated as the most relevant can be very unintuitive. Therefore, its usefulness as an explanatory tool may be compromised, at least in problems such as the one addressed in this study, where the features defining the model prediction are generally not consistently reflected in relevant regions of the saliency maps, and they even cannot always be related to those used as medical standards.<\/jats:p>","DOI":"10.3390\/s24010239","type":"journal-article","created":{"date-parts":[[2023,12,31]],"date-time":"2023-12-31T06:00:21Z","timestamp":1704002421000},"page":"239","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["In-Depth Evaluation of Saliency Maps for Interpreting Convolutional Neural Network Decisions in the Diagnosis of Glaucoma Based on Fundus Imaging"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3309-5953","authenticated-orcid":false,"given":"Jose","family":"Sigut","sequence":"first","affiliation":[{"name":"Department of Computer Science and Systems Engineering, Universidad de La Laguna, Camino San Francisco de Paula, 19, La Laguna, 38203 Santa Cruz de Tenerife, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francisco","family":"Fumero","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Systems Engineering, Universidad de La Laguna, Camino San Francisco de Paula, 19, La Laguna, 38203 Santa Cruz de Tenerife, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jos\u00e9","family":"Est\u00e9vez","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Systems Engineering, Universidad de La Laguna, Camino San Francisco de Paula, 19, La Laguna, 38203 Santa Cruz de Tenerife, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8498-3275","authenticated-orcid":false,"given":"Silvia","family":"Alay\u00f3n","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Systems Engineering, Universidad de La Laguna, Camino San Francisco de Paula, 19, La Laguna, 38203 Santa Cruz de Tenerife, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4579-0555","authenticated-orcid":false,"given":"Tinguaro","family":"D\u00edaz-Alem\u00e1n","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Hospital Universitario de Canarias, Carretera Ofra S\/N, La Laguna, 38320 Santa Cruz de Tenerife, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,12,31]]},"reference":[{"key":"ref_1","first-page":"e11686","article-title":"Epidemiology of Glaucoma: The Past, Present, and Predictions for the Future","volume":"12","author":"Allison","year":"2020","journal-title":"Cureus"},{"key":"ref_2","unstructured":"Mahabadi, N., Foris, L.A., and Tripathy, K. (2023). StatPearls, StatPearls Publishing."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"102470","DOI":"10.1016\/j.media.2022.102470","article-title":"Explainable artificial intelligence (XAI) in deep learning-based medical image analysis","volume":"79","author":"Kuijf","year":"2022","journal-title":"Med. Image Anal."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","article-title":"A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI","volume":"32","author":"Tjoa","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2128","DOI":"10.1093\/jamia\/ocab127","article-title":"Trading off accuracy and explainability in AI decision-making: Findings from 2 citizens\u2019 juries","volume":"28","author":"Riste","year":"2021","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_7","unstructured":"Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"e157968","DOI":"10.1172\/JCI157968","article-title":"A deep-learning system predicts glaucoma incidence and progression using retinal photographs","volume":"132","author":"Li","year":"2022","journal-title":"J. Clin. Investig."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kim, M., Janssens, O., min Park, H., Zuallaert, J., van Hoecke, S., and Neve, W.D. (2018). Web Applicable Computer-aided Diagnosis of Glaucoma Using Deep Learning. arXiv.","DOI":"10.1109\/BIBM.2018.8621168"},{"key":"ref_10","first-page":"1624","article-title":"Glaucoma detection based on a deep learning approach","volume":"61","author":"Kuo","year":"2020","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3013","DOI":"10.1007\/s10792-020-01485-3","article-title":"Effect of color information on the diagnostic performance of glaucoma in deep learning using few fundus images","volume":"40","author":"Hirota","year":"2020","journal-title":"Int. Ophthalmol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"e200267","DOI":"10.1148\/ryai.2021200267","article-title":"Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging","volume":"3","author":"Arun","year":"2020","journal-title":"Radiol. Artif. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"867","DOI":"10.1038\/s42256-022-00536-x","article-title":"Benchmarking saliency methods for chest X-ray interpretation","volume":"4","author":"Saporta","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2573","DOI":"10.2147\/OPTH.S312236","article-title":"Evaluation of Explainable Deep Learning Methods for Ophthalmic Diagnosis","volume":"15","author":"Singh","year":"2021","journal-title":"Clin. Ophthalmol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1167\/tvst.9.2.64","article-title":"Systematic Comparison of Heatmapping Techniques in Deep Learning in the Context of Diabetic Retinopathy Lesion Detection","volume":"9","author":"Elen","year":"2020","journal-title":"Transl. Vis. Sci. Technol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"102364","DOI":"10.1016\/j.media.2022.102364","article-title":"Clinical validation of saliency maps for understanding deep neural networks in ophthalmology","volume":"77","author":"Ayhan","year":"2022","journal-title":"Med. Image Anal."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"161","DOI":"10.5566\/ias.2346","article-title":"RIM-ONE DL: A Unified Retinal Image Database for Assessing Glaucoma Using Deep Learning","volume":"39","author":"Fumero","year":"2020","journal-title":"Image Anal. Stereol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"101570","DOI":"10.1016\/j.media.2019.101570","article-title":"REFUGE Challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs","volume":"59","author":"Orlando","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_19","unstructured":"Sivaswamy, J., Chakravarty, A., Joshi, G.D., and Syed, T.A. (2015). A Comprehensive Retinal Image Dataset for the Assessment of Glaucoma from the Optic Nerve Head Analysis. JSM Biomed. Imaging Data Pap., 2."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Bajwa, M.N., Singh, G.A.P., Neumeier, W., Malik, M.I., Dengel, A., and Ahmed, S. (2020, January 19\u201324). G1020: A Benchmark Retinal Fundus Image Dataset for Computer-Aided Glaucoma Detection. Proceedings of the 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207664"},{"key":"ref_21","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA."},{"key":"ref_22","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_23","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (July, January 26). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_24","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2023, September 18). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015. Software. Available online: www.tensorflow.org."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Brodersen, K.H., Ong, C.S., Stephan, K.E., and Buhmann, J.M. (2010, January 23\u201326). The Balanced Accuracy and Its Posterior Distribution. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.764"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1145\/3359786","article-title":"Techniques for interpretable machine learning","volume":"63","author":"Du","year":"2018","journal-title":"Commun. ACM"},{"key":"ref_27","first-page":"1803","article-title":"How to Explain Individual Classification Decisions","volume":"11","author":"Baehrens","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_28","unstructured":"Shrikumar, A., Greenside, P., and Kundaje, A. (2017, January 6\u201311). Learning Important Features Through Propagating Activation Differences. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_29","unstructured":"Springenberg, J., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015, January 7\u20139). Striving for Simplicity: The All Convolutional Net. Proceedings of the 3rd International Conference on Learning Representations (Workshop Track), San Diego, CA, USA."},{"key":"ref_30","unstructured":"Smilkov, D., Thorat, N., Kim, B., Viegas, F., and Wattenberg, M. (2017). SmoothGrad: Removing noise by adding noise. arXiv."},{"key":"ref_31","unstructured":"Adebayo, J., Gilmer, J., Goodfellow, I., and Kim, B. (2018). Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values. arXiv."},{"key":"ref_32","unstructured":"Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6\u201311). Axiomatic Attribution for Deep Networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. Visualizing and Understanding Convolutional Networks. Proceedings of the Computer Vision\u2014ECCV 2014.","DOI":"10.1007\/978-3-319-10599-4"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., and Hu, X. (2020, January 13\u201319). Score-CAM: Score-weighted visual explanations for convolutional neural networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00020"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1809","DOI":"10.1016\/S0161-6420(00)00284-0","article-title":"Mapping the visual field to the optic disc in normal tension glaucoma eyes","volume":"107","author":"Poinoosawmy","year":"2000","journal-title":"Ophthalmology"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"578","DOI":"10.1080\/01621459.1972.10481251","article-title":"Significance Testing of the Spearman Rank Correlation Coefficient","volume":"67","author":"Zar","year":"1972","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_38","unstructured":"Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (2018, January 3\u20138). Sanity Checks for Saliency Maps. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"20313","DOI":"10.1038\/s41598-021-99605-1","article-title":"Deep learning on fundus images detects glaucoma beyond the optic disc","volume":"11","author":"Hemelings","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_40","unstructured":"Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., and Brendel, W. (2019, January 6\u20139). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"045024","DOI":"10.1088\/2632-2153\/ad0798","article-title":"Interpretable surrogate models to approximate the predictions of convolutional neural networks in glaucoma diagnosis","volume":"4","author":"Sigut","year":"2023","journal-title":"Mach. Learn. Sci. Technol."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/1\/239\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:45:14Z","timestamp":1760132714000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/1\/239"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,31]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["s24010239"],"URL":"https:\/\/doi.org\/10.3390\/s24010239","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,31]]}}}