{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,23]],"date-time":"2026-02-23T13:42:38Z","timestamp":1771854158012,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2025,2,15]],"date-time":"2025-02-15T00:00:00Z","timestamp":1739577600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Explainability is essential for AI models, especially in clinical settings where understanding the model\u2019s decisions is crucial. Despite their impressive performance, black-box AI models are unsuitable for clinical use if their operations cannot be explained to clinicians. While deep neural networks (DNNs) represent the forefront of model performance, their explanations are often not easily interpreted by humans. On the other hand, hand-crafted features extracted to represent different aspects of the input data and traditional machine learning models are generally more understandable. However, they often lack the effectiveness of advanced models due to human limitations in feature design. To address this, we propose ExShall-CNN, a novel explainable shallow convolutional neural network for medical image processing. This model improves upon hand-crafted features to maintain human interpretability, ensuring that its decisions are transparent and understandable. We introduce the explainable shallow convolutional neural network (ExShall-CNN), which combines the interpretability of hand-crafted features with the performance of advanced deep convolutional networks like U-Net for medical image segmentation. Built on recent advancements in machine learning, ExShall-CNN incorporates widely used kernels while ensuring transparency, making its decisions visually interpretable by physicians and clinicians. This balanced approach offers both the accuracy of deep learning models and the explainability needed for clinical applications.<\/jats:p>","DOI":"10.3390\/make7010019","type":"journal-article","created":{"date-parts":[[2025,2,17]],"date-time":"2025-02-17T04:40:25Z","timestamp":1739767225000},"page":"19","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["ExShall-CNN: An Explainable Shallow Convolutional Neural Network for Medical Image Segmentation"],"prefix":"10.3390","volume":"7","author":[{"given":"Vahid","family":"Khalkhali","sequence":"first","affiliation":[{"name":"Center for Computational and Integrative Biology, Rutgers University, Camden, NJ 08102, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9080-7307","authenticated-orcid":false,"given":"Sayed Mehedi","family":"Azim","sequence":"additional","affiliation":[{"name":"Center for Computational and Integrative Biology, Rutgers University, Camden, NJ 08102, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Iman","family":"Dehzangi","sequence":"additional","affiliation":[{"name":"Center for Computational and Integrative Biology, Rutgers University, Camden, NJ 08102, USA"},{"name":"Department of Computer Science, Rutgers University, Camden, NJ 08102, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,2,15]]},"reference":[{"key":"ref_1","unstructured":"Murphy, K.P. (2012). Machine Learning: A Probabilistic Perspective, The MIT Press."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/0893-6080(89)90020-8","article-title":"Multilayer feedforward networks are universal approximators","volume":"2","author":"Hornik","year":"1989","journal-title":"Neural Netw."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Mhaskar, H., Liao, Q., and Poggio, T. When and why are deep networks better than shallow ones? In Proceedings of the AAAI Conference On Artificial Intelligence, San Francisco, CA, USA, 4\u20139 February 2017; Volume 31.","DOI":"10.1609\/aaai.v31i1.10913"},{"key":"ref_5","unstructured":"Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1986). Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. Biometrika, 71."},{"key":"ref_6","unstructured":"Vaswani, A. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e220012","DOI":"10.1148\/ryai.220012","article-title":"Visual transformers and convolutional neural networks for disease classification on radiographs: A comparison of performance, sample efficiency, and hidden stratification","volume":"4","author":"Murphy","year":"2022","journal-title":"Radiol. Artif. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shamshirband, S., Fathi, M., Dehzangi, A., Chronopoulos, A.T., and Alinejad-Rokny, H. (2021). A review on deep learning approaches in healthcare systems: Taxonomies, challenges, and open issues. J. Biomed. Inform., 113.","DOI":"10.1016\/j.jbi.2020.103627"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"110424","DOI":"10.1016\/j.patcog.2024.110424","article-title":"Federated learning for medical image analysis: A survey","volume":"151","author":"Guan","year":"2024","journal-title":"Pattern Recognit."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4733","DOI":"10.1016\/j.csbj.2022.08.039","article-title":"Accurate brain tumor detection using deep convolutional neural network","volume":"20","author":"Khan","year":"2022","journal-title":"Comput. Struct. Biotechnol. J."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_12","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention\u2013MICCAI 2015: 18th International Conference, Munich, Germany. Proceedings, Part III 18."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3563691","article-title":"Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions","volume":"55","author":"Ibrahim","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Bennetot","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101286","DOI":"10.1016\/j.imu.2023.101286","article-title":"Application of explainable artificial intelligence in medical health: A systematic review of interpretability methods","volume":"40","author":"Band","year":"2023","journal-title":"Inform. Med. Unlocked"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"102470","DOI":"10.1016\/j.media.2022.102470","article-title":"Explainable artificial intelligence (XAI) in deep learning-based medical image analysis","volume":"79","author":"Kuijf","year":"2022","journal-title":"Med. Image Anal."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Wu, Y.N., and Zhu, S.-C. (2018, January 18\u201322). Interpretable Convolutional Neural Networks. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00920"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Huang, S., Xu, Z., Tao, D., and Zhang, Y. (2016, January 27\u201330). Part-Stacked CNN for Fine-Grained Visual Categorization. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.132"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1109\/TETCI.2021.3100641","article-title":"A Survey on Neural Network Interpretability","volume":"5","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Salahuddin, Z., Woodruff, H.C., Chatterjee, A., and Lambin, P. (2022). Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Comput. Biol. Med., 140.","DOI":"10.1016\/j.compbiomed.2021.105111"},{"key":"ref_21","first-page":"23","article-title":"A threshold selection method from gray-level histograms","volume":"11","author":"Otsu","year":"1975","journal-title":"Automatica"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1016\/S0031-3203(99)00055-2","article-title":"Adaptive document image binarization","volume":"33","author":"Sauvola","year":"2000","journal-title":"Pattern Recognit."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Scholkopf, B., and Smola, A.J. (2001). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press.","DOI":"10.7551\/mitpress\/4175.001.0001"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/s11263-006-9794-4","article-title":"Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study","volume":"73","author":"Zhang","year":"2006","journal-title":"Int. J. Comput. Vis."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e21","DOI":"10.23915\/distill.00021","article-title":"Computing Receptive Fields of Convolutional Neural Networks","volume":"4","author":"Araujo","year":"2019","journal-title":"Distill"},{"key":"ref_27","unstructured":"Luo, W., Li, Y., Urtasun, R., and Zemel, R. (2016, January 5\u201310). Understanding the effective receptive field in deep convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_28","unstructured":"(2024, July 01). Retina Blood Vessel. Available online: https:\/\/www.kaggle.com\/datasets\/abdallahwagih\/retina-blood-vessel."},{"key":"ref_29","unstructured":"Gutman, D., Codella, N.C.F., Celebi, E., Helba, B., Marchetti, M., Mishra, N., and Halpern, A. (2016). Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC). arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Codella, N.C.F., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., and Kittler, H. (2018, January 4\u20137). Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA.","DOI":"10.1109\/ISBI.2018.8363547"},{"key":"ref_31","unstructured":"Codella, N., Rotemberg, V., Tschandl, P., Emre Celebi, M., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., and Marchetti, M. (2019). Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC). arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"126626","DOI":"10.1016\/j.neucom.2023.126626","article-title":"A comprehensive survey on segmentation techniques for retinal vessel segmentation","volume":"556","author":"Cervantes","year":"2023","journal-title":"Neurocomputing"},{"key":"ref_33","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library, Curran Associates Inc."},{"key":"ref_34","unstructured":"Kingma, D.P. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_35","unstructured":"Wang, K.Y.W., Hugonot, J., Fua, P., and Salzmann, M. (2025, January 08). Recurrent U-Net for Resource-Constrained Segmentation. Available online: https:\/\/ieeexplore.ieee.org\/document\/9010910\/."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Marques dos Santos, J.D., and Marques dos Santos, J.P. (2022). Towards XAI: Interpretable Shallow Neural Network Used to Model HCP\u2019s fMRI Motor Paradigm Data. Bioinformatics and Bio-Medical Engineering, Springer International Publishing.","DOI":"10.1007\/978-3-031-07802-6_22"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27\u201330). Learning Deep Features for Discriminative Localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.319"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Liu, S., and Deng, W. (2015, January 3\u20136). Very deep convolutional neural network based image classification using small training sample size. Proceedings of the 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ACPR.2015.7486599"},{"key":"ref_39","first-page":"1","article-title":"All models are wrong, but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously","volume":"20","author":"Fisher","year":"2019","journal-title":"J. Mach. Learn. Res."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/1\/19\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:35:17Z","timestamp":1760027717000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/1\/19"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,15]]},"references-count":39,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["make7010019"],"URL":"https:\/\/doi.org\/10.3390\/make7010019","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,15]]}}}