{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T19:16:00Z","timestamp":1790709360975,"version":"4.1.0"},"reference-count":42,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,8,17]],"date-time":"2021-08-17T00:00:00Z","timestamp":1629158400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100007514","name":"Universit\u00e0 di Pisa","doi-asserted-by":"publisher","award":["PRA_2018_43"],"award-info":[{"award-number":["PRA_2018_43"]}],"id":[{"id":"10.13039\/501100007514","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010666","name":"H2020 Research Infrastructures","doi-asserted-by":"publisher","award":["G.A. 871042"],"award-info":[{"award-number":["G.A. 871042"]}],"id":[{"id":"10.13039\/100010666","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010663","name":"H2020 European Research Council","doi-asserted-by":"publisher","award":["G.A. 834756"],"award-info":[{"award-number":["G.A. 834756"]}],"id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010669","name":"H2020 LEIT Information and Communication Technologies","doi-asserted-by":"publisher","award":["G.A. 952215"],"award-info":[{"award-number":["G.A. 952215"]}],"id":[{"id":"10.13039\/100010669","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010669","name":"H2020 LEIT Information and Communication Technologies","doi-asserted-by":"publisher","award":["G.A. 952026"],"award-info":[{"award-number":["G.A. 952026"]}],"id":[{"id":"10.13039\/100010669","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The biomedical field is characterized by an ever-increasing production of sequential data, which often come in the form of biosignals capturing the time-evolution of physiological processes, such as blood pressure and brain activity. This has motivated a large body of research dealing with the development of machine learning techniques for the predictive analysis of such biosignals. Unfortunately, in high-stakes decision making, such as clinical diagnosis, the opacity of machine learning models becomes a crucial aspect to be addressed in order to increase the trust and adoption of AI technology. In this paper, we propose a model agnostic explanation method, based on occlusion, that enables the learning of the input\u2019s influence on the model predictions. We specifically target problems involving the predictive analysis of time-series data and the models that are typically used to deal with data of such nature, i.e., recurrent neural networks. Our approach is able to provide two different kinds of explanations: one suitable for technical experts, who need to verify the quality and correctness of machine learning models, and one suited to physicians, who need to understand the rationale underlying the prediction to make aware decisions. A wide experimentation on different physiological data demonstrates the effectiveness of our approach both in classification and regression tasks.<\/jats:p>","DOI":"10.3390\/e23081064","type":"journal-article","created":{"date-parts":[[2021,8,17]],"date-time":"2021-08-17T21:27:40Z","timestamp":1629235660000},"page":"1064","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7811-3895","authenticated-orcid":false,"given":"Michele","family":"Resta","sequence":"first","affiliation":[{"name":"Computer Science Department, University of Pisa, 56127 Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8541-0284","authenticated-orcid":false,"given":"Anna","family":"Monreale","sequence":"additional","affiliation":[{"name":"KDDLab, Computer Science Department, University of Pisa, 56127 Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5213-2468","authenticated-orcid":false,"given":"Davide","family":"Bacciu","sequence":"additional","affiliation":[{"name":"Computer Science Department, University of Pisa, 56127 Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,17]]},"reference":[{"key":"ref_1","unstructured":"Bacciu, D., Lisboa, P.J., Mart\u00edn, J.D., Stoean, R., and Vellido Alcacena, A. (2018, January 25\u201327). Bioinformatics and medicine in the era of deep learning. Proceedings fo the ESANN 2018: 26th European Symposium on Artificial Neural Networks, Bruges, Belgium."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1055\/s-0038-1667083","article-title":"Deep Learning on 1-D Biosignals: A Taxonomy-Based Survey","volume":"27","author":"Ganapathy","year":"2018","journal-title":"Yearb. Med Inform."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhang, Y., and Feng, Z. (2017, January 24\u201326). A SVM Method for Continuous Blood Pressure Estimation from a PPG Signal. Proceedings of the 9th International Conference on Machine Learning and Computing, Singapore.","DOI":"10.1145\/3055635.3056634"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wang, R., Jia, W., Mao, Z.H., Sclabassi, R.J., and Sun, M. (2014, January 19\u201323). Cuff-Free Blood Pressure Estimation Using Pulse Transit Time and Heart Rate. Proceedings of the 2014 12th international conference on signal processing (ICSP), Hangzhou, China.","DOI":"10.1109\/ICOSP.2014.7014980"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","article-title":"Explanation in Artificial Intelligence: Insights from the Social Sciences","volume":"267","author":"Miller","year":"2019","journal-title":"Artif. Intell."},{"key":"ref_7","unstructured":"Lundberg, S.M., and Lee, S. A Unified Approach to Interpreting Model Predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_8","first-page":"93","article-title":"A Survey of Methods for Explaining Black Box Models","volume":"51","author":"Guidotti","year":"2018","journal-title":"Acm Comput. Surv. (CSUR)"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bai, T., Zhang, S., Egleston, B.L., and Vucetic, S. (2018, January 19\u201323). Interpretable Representation Learning for Healthcare via Capturing Disease Progression through Time. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3219904"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The Mythos of Model Interpretability: In Machine Learning, the Concept of Interpretability is Both Important and Slippery","volume":"16","author":"Lipton","year":"2018","journal-title":"Queue"},{"key":"ref_11","unstructured":"Springenberg, J.T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015). Striving for Simplicity: The All Convolutional Net. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014, January 6\u201312). Visualizing and Understanding Convolutional Networks. Proceedings of the Computer Vision\u2014ECCV 2014, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_13","unstructured":"Shrikumar, A., Greenside, P., and Kundaje, A. (2017, January 6\u201311). Learning Important Features Through Propagating Activation Differences. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.R., and Samek, W. (2015). On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130140"},{"key":"ref_15","unstructured":"Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6\u201311). Axiomatic Attribution for Deep Networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_17","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7\u20139). Neural Machine Translation by Jointly Learning to Align and Translate. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA."},{"key":"ref_18","unstructured":"Choi, E., Bahadori, M.T., Kulas, J.A., Schuetz, A., Stewart, W.F., and Sun, J. (2017). RETAIN: An Interpretable Predictive Model for Healthcare Using Reverse Time Attention Mechanism. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Girkar, U.M., Uchimido, R., Lehman, L.H., Szolovits, P., Celi, L.A., and Weng, W. (2018). Predicting Blood Pressure Response to Fluid Bolus Therapy Using Attention-Based Neural Networks for Clinical Interpretability. arXiv.","DOI":"10.1161\/res.125.suppl_1.448"},{"key":"ref_20","unstructured":"Beer, T., Eini-Porat, B., Goodfellow, S., Eytan, D., and Shalit, U. (2020, January 7\u20138). Using deep networks for scientific discovery in physiological signals. Proceedings of the Machine Learning for Healthcare Conference (MLHC 2020), Virtual Event, Durham, NC, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Mercier, D., Dengel, A., and Ahmed, S. (2021). PatchX: Explaining Deep Models by Intelligible Pattern Patches for Time-series Classification. arXiv.","DOI":"10.1109\/IJCNN52387.2021.9533293"},{"key":"ref_22","unstructured":"Gee, A.H., Garc\u00eda-Olano, D., Ghosh, J., and Paydarfar, D. (2019, January 10). Explaining Deep Classification of Time-Series Data with Learned Prototypes. Proceedings of the 4th International Workshop on Knowledge Discovery in Healthcare Data Co-Located with the 28th International Joint Conference on Artificial Intelligence, KDH@IJCAI 2019, Macao, China."},{"key":"ref_23","unstructured":"Li, O., Liu, H., Chen, C., and Rudin, C. (2018, January 2\u20137). Deep Learning for Case-Based Reasoning Through Prototypes: A Neural Network That Explains Its Predictions. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, LA, USA."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"825","DOI":"10.1109\/LSP.2020.2993176","article-title":"An Information-Theoretic Approach to Personalized Explainable Machine Learning","volume":"27","author":"Jung","year":"2020","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zhou, C., and Yuan, J. (2018, January 8\u201314). Bi-Box Regression for Pedestrian Detection and Occlusion Estimation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01246-5_9"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kohonen, T. (2001). Self-Organizing Maps, Springer. [3rd ed.].","DOI":"10.1007\/978-3-642-56927-2"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Bacciu, D., Bertoncini, G., and Morelli, D. (2021). Topographic Mapping for Quality Inspection and Intelligent Filtering of Smart-Bracelet Data. Neural Comput. Appl.","DOI":"10.1007\/s00521-020-05600-4"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kachuee, M., Kiani, M.M., Mohammadzade, H., and Shabany, M. (2015, January 24\u201327). Cuff-Less High-Accuracy Calibration-Free Blood Pressure Estimation Using Pulse Transit Time. Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), Lisbon, Portugal.","DOI":"10.1109\/ISCAS.2015.7168806"},{"key":"ref_29","unstructured":"Dua, D., and Graff, C. (2021, May 17). UCI Machine Learning Repository. Available online: https:\/\/ergodicity.net\/2013\/07\/."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1097\/CCM.0b013e31820a92c6","article-title":"Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC-II): A Public-Access Intensive Care Unit Database","volume":"39","author":"Saeed","year":"2011","journal-title":"Crit. Care Med."},{"key":"ref_31","first-page":"e215","article-title":"PhysioBank, PhysioToolkit, and PhysioNet","volume":"101","author":"Glass","year":"2000","journal-title":"Circulation"},{"key":"ref_32","unstructured":"Garc\u00eda-Gonz\u00e1lez, M.A., Argelag\u00f3s-Palau, A., Fern\u00e1ndez-Chimeno, M., and Ramos-Castro, J. (2013, January 22\u201325). A Comparison of Heartbeat Detectors for the Seismocardiogram. Proceedings of the Computing in Cardiology, Zaragoza, Spain."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Bousseljot, R., Kreiseler, D., and Schnabel, A. (2009). Nutzung Der EKG-Signaldatenbank CARDIODAT Der PTB \u00dcber Das Internet. Biomed. Tech. Eng., 317\u2013318.","DOI":"10.1515\/bmte.1995.40.s1.317"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kachuee, M., Fazeli, S., and Sarrafzadeh, M. (2018, January 4\u20137). ECG Heartbeat Classification: A Deep Transferable Representation. Proceedings of the 2018 IEEE International Conference on Healthcare Informatics (ICHI), New York, NY, USA.","DOI":"10.1109\/ICHI.2018.00092"},{"key":"ref_35","unstructured":"Chollet, F. (2015, March 15). Keras. Available online: https:\/\/github.com\/keras-team\/keras."},{"key":"ref_36","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2016). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. arXiv."},{"key":"ref_37","unstructured":"Hinton, G.E., and Zemel, R.S. (December, January 29). Autoencoders, Minimum Description Length and Helmholtz Free Energy. Proceedings of the 7th Neural Information Processing Systems (NIPS), Denver, CO, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Cho, K., van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014, January 25\u201329). Learning Phrase Representations Using RNN Encoder\u2014Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/35.41400","article-title":"Handwritten Digit Recognition: Applications of Neural Network Chips and Automatic Learning","volume":"27","author":"Cun","year":"1989","journal-title":"IEEE Commun. Mag."},{"key":"ref_40","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"18069","DOI":"10.1007\/s00521-019-04051-w","article-title":"The Importance of Interpretability and Visualization in Machine Learning for Applications in Medicine and Health Care","volume":"32","author":"Vellido","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_42","unstructured":"Vettigli, G. (2018, May 08). MiniSom: Minimalistic and NumPy-Based Implementation of the Self Organizing Map. Available online: https:\/\/github.com\/JustGlowing\/minisom."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/8\/1064\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:45:45Z","timestamp":1760165145000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/8\/1064"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,17]]},"references-count":42,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["e23081064"],"URL":"https:\/\/doi.org\/10.3390\/e23081064","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,17]]}}}