{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:32:38Z","timestamp":1760059958812,"version":"build-2065373602"},"reference-count":47,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:00:00Z","timestamp":1753401600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Canary Islands Agency for Research, Innovation and the Information Society (ACIISI)","award":["PROID2024010027"],"award-info":[{"award-number":["PROID2024010027"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>This work aims to leverage Shapley values to explain the decisions of convolutional neural networks trained to predict glaucoma. Although Shapley values offer a mathematically sound approach rooted in game theory, they require evaluating all possible combinations of features, which can be computationally intensive. To address this challenge, we introduce a novel strategy that discretizes the input by dividing the image into standard regions or sectors of interest, significantly reducing the number of features while maintaining clinical relevance. Moreover, applying Shapley values in a machine learning context necessitates the ability to selectively exclude features to evaluate their combinations. To achieve this, we propose a method involving the occlusion of specific sectors and re-training only the non-convolutional portion of the models. Despite achieving strong predictive performance, our findings reveal limited alignment with medical expectations, particularly the unexpected dominance of the background sector in the model\u2019s decision-making process. This highlights potential concerns regarding the interpretability of convolutional neural network-based glaucoma diagnostics.<\/jats:p>","DOI":"10.3390\/a18080464","type":"journal-article","created":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T08:13:31Z","timestamp":1753431211000},"page":"464","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Using Shapley Values to Explain the Decisions of Convolutional Neural Networks in Glaucoma Diagnosis"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3309-5953","authenticated-orcid":false,"given":"Jose","family":"Sigut","sequence":"first","affiliation":[{"name":"Department of Computer Science and Systems Engineering, Universidad de La Laguna, Camino San Francisco de Paula, 19, La Laguna, 38203 Santa Cruz de Tenerife, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francisco","family":"Fumero","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Systems Engineering, Universidad de La Laguna, Camino San Francisco de Paula, 19, La Laguna, 38203 Santa Cruz de Tenerife, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tinguaro","family":"D\u00edaz-Alem\u00e1n","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, Hospital Universitario de Canarias, Carretera Ofra S\/N, La Laguna, 38320 Santa Cruz de Tenerife, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,7,25]]},"reference":[{"key":"ref_1","unstructured":"Molnar, C. (2022). Interpretable Machine Learning, Lulu Press, Inc.. [2nd ed.]."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Hemelings, R., Elen, B., Barbosa-Breda, J., Blaschko, M.B., De Boever, P., and Stalmans, I. (2021). Deep learning on fundus images detects glaucoma beyond the optic disc. Sci. Rep., 11.","DOI":"10.1038\/s41598-021-99605-1"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Loh, H.W., Ooi, C.P., Seoni, S., Barua, P.D., Molinari, F., and Acharya, U.R. (2022). Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011\u20132022). Comput. Methods Programs Biomed., 226.","DOI":"10.1016\/j.cmpb.2022.107161"},{"key":"ref_4","unstructured":"Shapley, L.S. (1952). A Value for N-Person Games, RAND Corporation."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"103502","DOI":"10.1016\/j.artint.2021.103502","article-title":"Explaining individual predictions when features are dependent: More accurate approximations to Shapley values","volume":"298","author":"Aas","year":"2021","journal-title":"Artif. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Sigut, J., Fumero, F., Est\u00e9vez, J., Alay\u00f3n, S., and D\u00edaz-Alem\u00e1n, T. (2024). In-Depth Evaluation of Saliency Maps for Interpreting Convolutional Neural Network Decisions in the Diagnosis of Glaucoma Based on Fundus Imaging. Sensors, 24.","DOI":"10.3390\/s24010239"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Roth, A.E. (1988). The Shapley Value: Essays in Honor of Lloyd S. Shapley, Cambridge University Press.","DOI":"10.1017\/CBO9780511528446"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1007\/s10115-013-0679-x","article-title":"Explaining prediction models and individual predictions with feature contributions","volume":"41","author":"Kononenko","year":"2014","journal-title":"Knowl. Inf. Syst."},{"key":"ref_9","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). A Unified Approach to Interpreting Model Predictions. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chen, H., Lundberg, S., and Lee, S.I. (2019). Explaining Models by Propagating Shapley Values of Local Components. arXiv.","DOI":"10.1007\/978-3-030-53352-6_24"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2081","DOI":"10.1016\/j.ophtha.2014.05.013","article-title":"Global Prevalence of Glaucoma and Projections of Glaucoma Burden through 2040: A Systematic Review and Meta-Analysis","volume":"121","author":"Tham","year":"2014","journal-title":"Ophthalmology"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"European Glaucoma Society (2021). European Glaucoma Society Terminology and Guidelines for Glaucoma, 5th Edition. Br. J. Ophthalmol., 105, 1\u2013169.","DOI":"10.1136\/bjophthalmol-2021-egsguidelines"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2573","DOI":"10.2147\/OPTH.S312236","article-title":"Evaluation of explainable deep learning methods for ophthalmic diagnosis","volume":"15","author":"Singh","year":"2021","journal-title":"Clin. Ophthalmol."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Shorfuzzaman, M., Hossain, M.S., and El Saddik, A. (2021). An Explainable Deep Learning Ensemble Model for Robust Diagnosis of Diabetic Retinopathy Grading. ACM Trans. Multimed. Comput. Commun. Appl., 17.","DOI":"10.1145\/3469841"},{"key":"ref_15","unstructured":"Shrikumar, A., Greenside, P., and Kundaje, A. (2017, January 6\u201311). Learning important features through propagating activation differences. Proceedings of the 34th International Conference on Machine Learning-Volume 70, ICML\u201917, Sydney, Australia."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.ajo.2021.04.021","article-title":"Automated Detection of Glaucoma with Interpretable Machine Learning Using Clinical Data and Multimodal Retinal Images","volume":"231","author":"Mehta","year":"2021","journal-title":"Am. J. Ophthalmol."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Hasan, M.M., Phu, J., Wang, H., Sowmya, A., Kalloniatis, M., and Meijering, E. (2025). OCT-based diagnosis of glaucoma and glaucoma stages using explainable machine learning. Sci. Rep., 15.","DOI":"10.1038\/s41598-025-87219-w"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Oh, S., Park, Y., Cho, K.J., and Kim, S.J. (2021). Explainable Machine Learning Model for Glaucoma Diagnosis and Its Interpretation. Diagnostics, 11.","DOI":"10.3390\/diagnostics11030510"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"100336","DOI":"10.1016\/j.xops.2023.100336","article-title":"Predicting Glaucoma Progression to Surgery with Artificial Intelligence Survival Models","volume":"3","author":"Tao","year":"2023","journal-title":"Ophthalmol. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"100685","DOI":"10.1016\/j.xops.2024.100685","article-title":"Artificial Intelligence Models to Identify Patients at High Risk for Glaucoma Using Self-reported Health Data in a United States National Cohort","volume":"5","author":"Ravindranath","year":"2025","journal-title":"Ophthalmol. Sci."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Christopher, M., Gonzalez, R., Huynh, J., Walker, E., Radha Saseendrakumar, B., Bowd, C., Belghith, A., Goldbaum, M.H., Fazio, M.A., and Girkin, C.A. (2024). Proactive Decision Support for Glaucoma Treatment: Predicting Surgical Interventions with Clinically Available Data. Bioengineering, 11.","DOI":"10.3390\/bioengineering11020140"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"100445","DOI":"10.1016\/j.xops.2023.100445","article-title":"Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative","volume":"4","author":"Wang","year":"2024","journal-title":"Ophthalmol. Sci."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wang, R., Bradley, C., Herbert, P., Hou, K., Ramulu, P., Breininger, K., Unberath, M., and Yohannan, J. (2024). Deep learning-based identification of eyes at risk for glaucoma surgery. Sci. Rep., 14.","DOI":"10.1038\/s41598-023-50597-0"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Yoon, J.S., Kim, Y.e., Lee, E.J., Kim, H., and Kim, T.W. (2023). Systemic factors associated with 10-year glaucoma progression in South Korean population: A single center study based on electronic medical records. Sci. Rep., 13.","DOI":"10.1038\/s41598-023-27858-z"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"161","DOI":"10.5566\/ias.2346","article-title":"RIM-ONE DL: A Unified Retinal Image Database for Assessing Glaucoma Using Deep Learning","volume":"39","author":"Fumero","year":"2020","journal-title":"Image Anal. Stereol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Fumero, F., Alayon, S., Sanchez, J.L., Sigut, J., and Gonzalez-Hernandez, M. (2011, January 27\u201330). RIM-ONE: An open retinal image database for optic nerve evaluation. Proceedings of the 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), Bristol, UK.","DOI":"10.1109\/CBMS.2011.5999143"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"101570","DOI":"10.1016\/j.media.2019.101570","article-title":"REFUGE Challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs","volume":"59","author":"Orlando","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_28","unstructured":"Sivaswamy, J., Chakravarty, A., Joshi, G.D., and Syed, T.A. (2015). A Comprehensive Retinal Image Dataset for the Assessment of Glaucoma from the Optic Nerve Head Analysis. JSM Biomed. Imaging Data Pap., 2."},{"key":"ref_29","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_33","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2025, May 16). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Available online: www.tensorflow.org."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chalkiadakis, G., Elkind, E., and Wooldridge, M. (2012). Basic Concepts. Computational Aspects of Cooperative Game Theory, Springer International Publishing. Synthesis Lectures on Artificial Intelligence and Machine Learning.","DOI":"10.1007\/978-3-031-01558-8"},{"key":"ref_35","first-page":"100345","article-title":"Exact Shapley values for local and model-true explanations of decision tree ensembles","volume":"9","author":"Campbell","year":"2022","journal-title":"Mach. Learn. Appl."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"104594","DOI":"10.1016\/j.ijmedinf.2021.104594","article-title":"Predicting prognosis in COVID-19 patients using machine learning and readily available clinical data","volume":"155","author":"Campbell","year":"2021","journal-title":"Int. J. Med. Inform."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"886","DOI":"10.1016\/j.datak.2009.01.004","article-title":"Explaining instance classifications with interactions of subsets of feature values","volume":"68","author":"Kononenko","year":"2009","journal-title":"Data Knowl. Eng."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1809","DOI":"10.1016\/S0161-6420(00)00284-0","article-title":"Mapping the visual field to the optic disc in normal tension glaucoma eyes11The authors have no proprietary interest in the development or marketing of any product or instrument mentioned in this article","volume":"107","author":"Poinoosawmy","year":"2000","journal-title":"Ophthalmology"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1167\/tvst.9.2.64","article-title":"Systematic Comparison of Heatmapping Techniques in Deep Learning in the Context of Diabetic Retinopathy Lesion Detection","volume":"9","author":"Elen","year":"2020","journal-title":"Transl. Vis. Sci. Technol."},{"key":"ref_40","unstructured":"Covert, I., Lundberg, S., and Lee, S.I. (2022). Explaining by Removing: A Unified Framework for Model Explanation. arXiv."},{"key":"ref_41","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). A Benchmark for Interpretability Methods in Deep Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_42","unstructured":"Chen, H., Janizek, J.D., Lundberg, S., and Lee, S.I. (2020). True to the Model or True to the Data?. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From Local Explanations to Global Understanding with Explainable AI for Trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"578","DOI":"10.1080\/01621459.1972.10481251","article-title":"Significance Testing of the Spearman Rank Correlation Coefficient","volume":"67","author":"Zar","year":"1972","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_45","unstructured":"Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (2018). Sanity Checks for Saliency Maps. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Brodersen, K.H., Ong, C.S., Stephan, K.E., and Buhmann, J.M. (2010, January 23\u201326). The Balanced Accuracy and Its Posterior Distribution. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.764"},{"key":"ref_47","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy. Chia Laguna Resort."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/8\/464\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:15:48Z","timestamp":1760033748000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/8\/464"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,25]]},"references-count":47,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["a18080464"],"URL":"https:\/\/doi.org\/10.3390\/a18080464","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2025,7,25]]}}}