{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T17:11:22Z","timestamp":1781716282517,"version":"3.54.5"},"reference-count":22,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2019,7,5]],"date-time":"2019-07-05T00:00:00Z","timestamp":1562284800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003593","name":"Conselho Nacional de Desenvolvimento Cient\u00edfico e Tecnol\u00f3gico","doi-asserted-by":"publisher","award":["165019\/2018-2"],"award-info":[{"award-number":["165019\/2018-2"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>An application of explainable artificial intelligence on medical data is presented. There is an increasing demand in machine learning literature for such explainable models in health-related applications. This work aims to generate explanations on how a Convolutional Neural Network (CNN) detects tumor tissue in patches extracted from histology whole slide images. This is achieved using the \u201clocally-interpretable model-agnostic explanations\u201d methodology. Two publicly-available convolutional neural networks trained on the Patch Camelyon Benchmark are analyzed. Three common segmentation algorithms are compared for superpixel generation, and a fourth simpler parameter-free segmentation algorithm is proposed. The main characteristics of the explanations are discussed, as well as the key patterns identified in true positive predictions. The results are compared to medical annotations and literature and suggest that the CNN predictions follow at least some aspects of human expert knowledge.<\/jats:p>","DOI":"10.3390\/s19132969","type":"journal-article","created":{"date-parts":[[2019,7,5]],"date-time":"2019-07-05T11:44:16Z","timestamp":1562327056000},"page":"2969","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":132,"title":["Local Interpretable Model-Agnostic Explanations for Classification of Lymph Node Metastases"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7536-0081","authenticated-orcid":false,"given":"Iam","family":"Palatnik de Sousa","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro 22451-900, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9790-1328","authenticated-orcid":false,"given":"Marley","family":"Maria Bernardes Rebuzzi Vellasco","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro 22451-900, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4513-0004","authenticated-orcid":false,"given":"Eduardo","family":"Costa da Silva","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro 22451-900, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2019,7,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1016\/j.cell.2018.02.010","article-title":"Identifying medical diagnoses and treatable diseases by image-based deep learning","volume":"172","author":"Kermany","year":"2018","journal-title":"Cell"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1236","DOI":"10.1093\/bib\/bbx044","article-title":"Deep learning for healthcare: Review, opportunities and challenges","volume":"19","author":"Miotto","year":"2017","journal-title":"Brief. Bioinform."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Veeling, B.S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. (2018, January 16\u201320). Rotation equivariant CNNs for digital pathology. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain.","DOI":"10.1007\/978-3-030-00934-2_24"},{"key":"ref_5","unstructured":"Liu, Y., Gadepalli, K., Norouzi, M., Dahl, G.E., Kohlberger, T., Boyko, A., Venugopalan, S., Timofeev, A., Nelson, P.Q., and Corrado, G.S. (2017). Detecting cancer metastases on gigapixel pathology images. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2199","DOI":"10.1001\/jama.2017.14585","article-title":"Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer","volume":"318","author":"Bejnordi","year":"2017","journal-title":"JAMA"},{"key":"ref_7","unstructured":"Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). Why should i trust you?: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_10","unstructured":"Alber, M., Lapuschkin, S., Seegerer, P., H\u00e4gele, M., Sch\u00fctt, K.T., Montavon, G., Samek, W., M\u00fcller, K.R., D\u00e4hne, S., and Kindermans, P.J. (2018). iNNvestigate neural networks!. arXiv."},{"key":"ref_11","unstructured":"Kindermans, P.J., Hooker, S., Adebayo, J., Alber, M., Sch\u00fctt, K.T., D\u00e4hne, S., Erhan, D., and Kim, B. (2017). The (un) reliability of saliency methods. arXiv."},{"key":"ref_12","unstructured":"Veeling, B. (2019, March 13). The PatchCamelyon (PCam) Deep Learning Classification Benchmark. Available online: https:\/\/github.com\/basveeling\/pcam."},{"key":"ref_13","unstructured":"Veeling, B. (2019, March 13). Histopathologic Cancer Detection. Available online: https:\/\/www.kaggle.com\/c\/histopathologic-cancer-detection\/data."},{"key":"ref_14","unstructured":"(2019, July 05). Skimage Segmentation Module. Available online: https:\/\/scikit-image.org\/docs\/dev\/api\/skimage.segmentation.html."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1023\/B:VISI.0000022288.19776.77","article-title":"Efficient graph-based image segmentation","volume":"59","author":"Felzenszwalb","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2274","DOI":"10.1109\/TPAMI.2012.120","article-title":"SLIC superpixels compared to state-of-the-art superpixel methods","volume":"34","author":"Achanta","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Vedaldi, A., and Soatto, S. (2008, January 12\u201318). Quick shift and kernel methods for mode seeking. Proceedings of the European Conference on Computer Vision, Marseille, France.","DOI":"10.1007\/978-3-540-88693-8_52"},{"key":"ref_18","unstructured":"(2019, July 05). Keras. Available online: https:\/\/keras.io."},{"key":"ref_19","unstructured":"(2019, July 05). CNN\u2014How to Use 160,000 Images without Crashing. Available online: https:\/\/www.kaggle.com\/vbookshelf\/cnn-how-to-use-160-000-images-without-crashing\/data."},{"key":"ref_20","unstructured":"(2019, July 05). 180k img (VGG19) callback. Available online: https:\/\/www.kaggle.com\/maxlenormand\/vgg19-with-180k-images-public-lb-0-968\/data."},{"key":"ref_21","unstructured":"(2019, March 13). Camelyon16. Available online: https:\/\/camelyon16.grand-challenge.org\/Data\/."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1111\/j.1365-2559.1991.tb00229.x","article-title":"Ppathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: Experience from a large study with long-term follow-up","volume":"19","author":"Elston","year":"1991","journal-title":"Histopathology"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/13\/2969\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:02:53Z","timestamp":1760187773000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/13\/2969"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,5]]},"references-count":22,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2019,7]]}},"alternative-id":["s19132969"],"URL":"https:\/\/doi.org\/10.3390\/s19132969","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,5]]}}}