{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T08:15:39Z","timestamp":1777536939506,"version":"3.51.4"},"reference-count":57,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2023,11,6]],"date-time":"2023-11-06T00:00:00Z","timestamp":1699228800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,6]],"date-time":"2023-11-06T00:00:00Z","timestamp":1699228800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100015528","name":"Universidad de La Laguna","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100015528","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Spanish Ministry of Science, Innovation and Universities"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2023,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Deep learning systems, especially in critical fields like medicine, suffer from a significant drawback, their black box nature, which lacks mechanisms for explaining or interpreting their decisions. In this regard, our research aims to evaluate the use of surrogate models for interpreting convolutional neural network (CNN) decisions in glaucoma diagnosis. Our approach is novel in that we approximate the original model with an interpretable one and also change the input features, replacing pixels with tabular geometric features of the optic disc, cup, and neuroretinal rim. We trained CNNs with two types of images: original images of the optic nerve head and simplified images showing only the disc and cup contours on a uniform background. Decision trees were used as surrogate models due to their simplicity and visualization properties, while saliency maps were calculated for some images for comparison. The experiments carried out with 1271 images of healthy subjects and 721 images of glaucomatous eyes demonstrate that decision trees can closely approximate the predictions of neural networks trained on simplified contour images, with R-squared values near 0.9 for VGG19, Resnet50, InceptionV3 and Xception architectures. Saliency maps proved difficult to interpret and showed inconsistent results across architectures, in contrast to the decision trees. Additionally, some decision trees trained as surrogate models outperformed a decision tree trained on the actual outcomes without surrogation. Decision trees may be a more interpretable alternative to saliency methods. Moreover, the fact that we matched the performance of a decision tree without surrogation to that obtained by decision trees using knowledge distillation from neural networks is a great advantage since decision trees are inherently interpretable. Therefore, based on our findings, we think this approach would be the most recommendable choice for specialists as a diagnostic tool.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad0798","type":"journal-article","created":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T22:25:26Z","timestamp":1698445526000},"page":"045024","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Interpretable surrogate models to approximate the predictions of convolutional neural networks in glaucoma diagnosis"],"prefix":"10.1088","volume":"4","author":[{"given":"Jose","family":"Sigut","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9806-2477","authenticated-orcid":true,"given":"Francisco","family":"Fumero","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rafael","family":"Arnay","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7452-2958","authenticated-orcid":true,"given":"Jos\u00e9","family":"Est\u00e9vez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tinguaro","family":"D\u00edaz-Alem\u00e1n","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2023,11,6]]},"reference":[{"key":"mlstad0798bib1","doi-asserted-by":"publisher","first-page":"e339","DOI":"10.1016\/S2214-109X(13)70113-X","article-title":"Causes of vision loss worldwide, 1990\u20132010: a systematic analysis","volume":"1","author":"Bourne","year":"2013","journal-title":"Lancet Glob. Health"},{"key":"mlstad0798bib2","doi-asserted-by":"publisher","first-page":"2081","DOI":"10.1016\/j.ophtha.2014.05.013","article-title":"Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis","volume":"121","author":"Tham","year":"2014","journal-title":"Ophthalmology"},{"key":"mlstad0798bib3","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2022.102470","article-title":"Explainable artificial intelligence (XAI) in deep learning-based medical image analysis","volume":"79","author":"van der Velden","year":"2022","journal-title":"Med. Image Anal."},{"key":"mlstad0798bib4","doi-asserted-by":"publisher","first-page":"4793\u20144813","DOI":"10.1109\/TNNLS.2020.3027314","article-title":"A survey on explainable artificial intelligence (XAI): toward medical XAI","volume":"32","author":"Tjoa","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Sys."},{"key":"mlstad0798bib5","doi-asserted-by":"publisher","first-page":"2128","DOI":"10.1093\/jamia\/ocab127","article-title":"Trading off accuracy and explainability in AI decision-making: findings from 2 citizens\u2019 juries","volume":"28","author":"van der Veer","year":"2021","journal-title":"J. Am. Med. Inf. Assoc."},{"key":"mlstad0798bib6","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"mlstad0798bib7","author":"Ordish","year":"2020"},{"key":"mlstad0798bib8","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1097\/ICU.0000000000000780","article-title":"Applications of interpretability in deep learning models for ophthalmology","volume":"32","author":"Hanif","year":"2021","journal-title":"Curr. Opin. Ophthalmol."},{"key":"mlstad0798bib9","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1016\/j.cjca.2021.09.004","article-title":"Opening the black box: the promise and limitations of explainable machine learning in cardiology","volume":"38","author":"Petch","year":"2022","journal-title":"Can. J. Cardiol."},{"key":"mlstad0798bib10","article-title":"Deep inside convolutional networks: visualising image classification models and saliency maps","author":"Simonyan","year":"2014"},{"key":"mlstad0798bib11","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2021200267","article-title":"Assessing the (un)trustworthiness of saliency maps for localizing abnormalities in medical imaging","volume":"3","author":"Arun","year":"2020","journal-title":"Radiol.: Artif. Intell."},{"key":"mlstad0798bib12","doi-asserted-by":"publisher","first-page":"867","DOI":"10.1038\/s42256-022-00536-x","article-title":"Benchmarking saliency methods for chest x-ray interpretation","volume":"4","author":"Saporta","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"mlstad0798bib13","doi-asserted-by":"publisher","first-page":"2573","DOI":"10.2147\/OPTH.S312236","article-title":"Evaluation of explainable deep learning methods for ophthalmic diagnosis","volume":"15","author":"Singh","year":"2021","journal-title":"Clin. Ophthalmol."},{"key":"mlstad0798bib14","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1167\/tvst.9.2.64","article-title":"Systematic comparison of heatmapping techniques in deep learning in the context of diabetic retinopathy lesion detection","volume":"9","author":"Van Craenendonck","year":"2020","journal-title":"Transl. Vis. Sci. Tech."},{"key":"mlstad0798bib15","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2022.102364","article-title":"Clinical validation of saliency maps for understanding deep neural networks in ophthalmology","volume":"77","author":"Ayhan","year":"2022","journal-title":"Med. Image Anal."},{"key":"mlstad0798bib16","author":"Molnar","year":"2022","edition":"2nd edn"},{"key":"mlstad0798bib17","first-page":"pp 1135","article-title":"Why should i trust you?\u2019: explaining the predictions of any classifier","author":"Ribeiro","year":"2016"},{"key":"mlstad0798bib18","doi-asserted-by":"publisher","DOI":"10.5089\/9798400235832.001","article-title":"Surrogate data models: interpreting large-scale machine learning crisis prediction models","volume":"2023","author":"Chan-Lau","year":"2023","journal-title":"IMF Working Papers"},{"key":"mlstad0798bib19","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1177\/0963721409359277","article-title":"The magical mystery four: how is working memory capacity limited and why?","volume":"19","author":"Cowan","year":"2010","journal-title":"Curr. Dir. Psychol. Sci."},{"key":"mlstad0798bib20","doi-asserted-by":"publisher","first-page":"393","DOI":"10.18240\/ijo.2021.03.10","article-title":"High interpretable machine learning classifier for early glaucoma diagnosis","volume":"14","author":"Escamez","year":"2021","journal-title":"Int. J. Ophthalmol."},{"key":"mlstad0798bib21","doi-asserted-by":"publisher","first-page":"2685","DOI":"10.2147\/OPTH.S367722","article-title":"Glaucoma diagnosis through the integration of optical coherence tomography\/angiography and machine learning diagnostic models","volume":"16","author":"Kooner","year":"2022","journal-title":"Clin. Ophthalmol."},{"key":"mlstad0798bib22","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/j.knosys.2018.07.043","article-title":"Glaucoma diagnosis based on both hidden features and domain knowledge through deep learning models","volume":"161","author":"Chai","year":"2018","journal-title":"Knowl. Based Syst."},{"key":"mlstad0798bib23","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1016\/j.ajo.2021.04.021","article-title":"Automated detection of glaucoma with interpretable machine learning using clinical data and multimodal retinal images","volume":"231","author":"Mehta","year":"2021","journal-title":"Am. J. Ophthalmol."},{"key":"mlstad0798bib24","doi-asserted-by":"publisher","first-page":"510","DOI":"10.3390\/diagnostics11030510","article-title":"Explainable machine learning model for glaucoma diagnosis and its interpretation","volume":"11","author":"Oh","year":"2021","journal-title":"Diagnostics"},{"key":"mlstad0798bib25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00417-4","article-title":"A hierarchical deep learning approach with transparency and interpretability based on small samples for glaucoma diagnosis","volume":"4","author":"Xu","year":"2021","journal-title":"npj Digit. Med."},{"key":"mlstad0798bib26","doi-asserted-by":"publisher","first-page":"92","DOI":"10.3390\/jimaging7060092","article-title":"EffUnet-SpaGen: an efficient and spatial generative approach to glaucoma detection","volume":"7","author":"Krishna Adithya","year":"2021","journal-title":"J. Imaging"},{"key":"mlstad0798bib27","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0209409","article-title":"Accurate, fast, data efficient and interpretable glaucoma diagnosis with automated spatial analysis of the whole cup to disc profile","volume":"14","author":"MacCormick","year":"2019","journal-title":"PLoS One"},{"key":"mlstad0798bib28","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1007\/s11517-020-02307-5","article-title":"An enhanced deep image model for glaucoma diagnosis using feature-based detection in retinal fundus","volume":"59","author":"Singh","year":"2021","journal-title":"Med. Biol. Eng. Comput."},{"key":"mlstad0798bib29","first-page":"pp 619","article-title":"Explainability of deep learning\u2013based system in health care","author":"Kinger","year":"2022"},{"key":"mlstad0798bib30","doi-asserted-by":"publisher","first-page":"1945","DOI":"10.1038\/s41598-021-81554-4","article-title":"A combined convolutional and recurrent neural network for enhanced glaucoma detection","volume":"11","author":"Gheisari","year":"2021","journal-title":"Sci. Rep."},{"key":"mlstad0798bib31","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2022.103823","article-title":"G- LIME: statistical learning for local interpretations of deep neural networks using global priors","volume":"314","author":"Li","year":"2023","journal-title":"Artif. Intell."},{"key":"mlstad0798bib32","article-title":"OptiLIME: optimized lime explanations for diagnostic computer algorithms","author":"Visani","year":"2022"},{"key":"mlstad0798bib33","doi-asserted-by":"publisher","first-page":"2969","DOI":"10.3390\/s19132969","article-title":"Local interpretable model-agnostic explanations for classification of lymph node metastases","volume":"19","author":"Palatnik de Sousa","year":"2019","journal-title":"Sensors"},{"key":"mlstad0798bib34","article-title":"Global aggregations of local explanations for black box models","author":"van der Linden","year":"2019"},{"key":"mlstad0798bib35","article-title":"NormLime: a new feature importance metric for explaining deep neural networks","author":"Ahern","year":"2019"},{"key":"mlstad0798bib36","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1186\/s12911-019-0874-0","article-title":"On the interpretability of machine learning-based model for predicting hypertension","volume":"19","author":"Elshawi","year":"2019","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"mlstad0798bib37","first-page":"pp 2310","article-title":"Interpretability methods of machine learning algorithms with applications in breast cancer diagnosis","author":"Karatza","year":"2021"},{"key":"mlstad0798bib38","article-title":"Machine learning repository","author":"Dua","year":"2019"},{"key":"mlstad0798bib39","doi-asserted-by":"publisher","first-page":"161","DOI":"10.5566\/ias.2346","article-title":"Rim\u2013one dl: a unified retinal image database for assessing glaucoma using deep learning","volume":"39","author":"Batista","year":"2020","journal-title":"Image Anal. Stereol."},{"key":"mlstad0798bib40","first-page":"pp 1","article-title":"Rim\u2013one: an open retinal image Stereology","author":"Fumero","year":"2011"},{"key":"mlstad0798bib41","first-page":"pp 91","article-title":"Interactive tool and database for optic disc and cup segmentation of stereo and monocular retinal fundus images","author":"Fumero","year":"2015","edition":"ed"},{"key":"mlstad0798bib42","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2019.101570","article-title":"Refuge challenge: a unified framework for evaluating automated methods for glaucoma assessment from fundus photographs","volume":"59","author":"Orlando","year":"2020","journal-title":"Med. Image Anal."},{"key":"mlstad0798bib43","first-page":"1004","article-title":"A comprehensive retinal image dataset for the assessment of glaucoma from the optic nerve head analysis","volume":"2","author":"Sivaswamy","year":"2015","journal-title":"JSM Biomed. Imaging Data Pap."},{"key":"mlstad0798bib44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1136\/bjophthalmol-2021-egsguidelines","article-title":"European glaucoma society terminology and guidelines for glaucoma, 5th edition","volume":"105","author":"","year":"2021","journal-title":"Br. J. Ophthalmol."},{"key":"mlstad0798bib45","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1016\/S0039-6257(98)00049-6","article-title":"Ophthalmoscopic evaluation of the optic nerve head","volume":"43","author":"Jonas","year":"1999","journal-title":"Surv. Ophthalmol."},{"key":"mlstad0798bib46","first-page":"1151","article-title":"Optic disc, cup and neuroretinal rim size, configuration and correlations in normal eyes","volume":"297","author":"Jonas","year":"1988","journal-title":"Investigative Ophthalmol. Vis. Sci."},{"key":"mlstad0798bib47","first-page":"181","article-title":"The disc damage likelihood scale: reproducibility of a new method of estimating the amount of optic nerve damage caused by glaucoma","volume":"100","author":"Spaeth","year":"2002","journal-title":"Trans. Am. Ophthalmol. Soc."},{"key":"mlstad0798bib48","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1001\/archopht.1969.00990020193008","article-title":"The cup\/disc ratio: the findings of tonometry and tonography in the normal eye","volume":"82","author":"Armaly","year":"1969","journal-title":"Arch. Ophthalmol."},{"key":"mlstad0798bib49","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2015"},{"key":"mlstad0798bib50","first-page":"pp 770","article-title":"Deep residual learning for image recognition","author":"He","year":"2016"},{"key":"mlstad0798bib51","first-page":"pp 2818","article-title":"Rethinking the inception architecture for computer vision","author":"Szegedy","year":"2016"},{"key":"mlstad0798bib52","first-page":"pp 1800","article-title":"Xception: Deep learning with depthwise separable convolutions","author":"Chollet","year":"2017"},{"key":"mlstad0798bib53","first-page":"pp 3121","article-title":"The balanced accuracy and its posterior distribution","author":"Brodersen","year":"2010"},{"key":"mlstad0798bib54","doi-asserted-by":"publisher","first-page":"7099","DOI":"10.1038\/s41598-019-43385-2","article-title":"Rim-to-disc ratio outperforms cup-to-disc ratio for glaucoma prescreening","volume":"9","author":"Kumar","year":"2019","journal-title":"Sci. Rep."},{"key":"mlstad0798bib55","doi-asserted-by":"crossref","DOI":"10.2139\/ssrn.4327677","article-title":"Systematic application of saliency maps to explain the decisions of convolutional neural networks for glaucoma diagnosis based on disc and cup geometry","author":"Fumero","year":"2023"},{"key":"mlstad0798bib56","doi-asserted-by":"publisher","first-page":"2335","DOI":"10.1007\/s00530-022-00960-4","article-title":"A survey on the interpretability of deep learning in medical diagnosis","volume":"28","author":"Teng","year":"2022","journal-title":"Multimedia Syst."},{"key":"mlstad0798bib57","first-page":"pp 818","article-title":"Visualizing and understanding convolutional networks","author":"Zeiler","year":"2014","edition":"ed"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,6]],"date-time":"2023-11-06T10:02:34Z","timestamp":1699264954000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad0798"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,6]]},"references-count":57,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,11,6]]},"published-print":{"date-parts":[[2023,12,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad0798","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,6]]},"assertion":[{"value":"Interpretable surrogate models to approximate the predictions of convolutional neural networks in glaucoma diagnosis","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2023 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2023-06-08","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2023-10-27","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2023-11-06","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}