{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,12]],"date-time":"2026-11-12T12:44:54Z","timestamp":1794487494171,"version":"4.9.1"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,3,13]],"date-time":"2023-03-13T00:00:00Z","timestamp":1678665600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,13]],"date-time":"2023-03-13T00:00:00Z","timestamp":1678665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Post-hoc interpretability methods are critical tools to explain neural-network results. Several post-hoc methods have emerged in recent years but they produce different results when applied to a given task, raising the question of which method is the most suitable to provide accurate post-hoc interpretability. To understand the performance of each method, quantitative evaluation of interpretability methods is essential; however, currently available frameworks have several drawbacks that hinder the adoption of post-hoc interpretability methods, especially in high-risk sectors. In this work we propose a framework with quantitative metrics to assess the performance of existing post-hoc interpretability methods, particularly in time-series classification. We show that several drawbacks identified in the literature are addressed, namely, the dependence on human judgement, retraining and the shift in the data distribution when occluding samples. We also design a synthetic dataset with known discriminative features and tunable complexity. The proposed methodology and quantitative metrics can be used to understand the reliability of interpretability methods results obtained in practical applications. In turn, they can be embedded within operational workflows in critical fields that require accurate interpretability results for, example, regulatory policies.<\/jats:p>","DOI":"10.1038\/s42256-023-00620-w","type":"journal-article","created":{"date-parts":[[2023,3,13]],"date-time":"2023-03-13T17:03:03Z","timestamp":1678726983000},"page":"250-260","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":140,"title":["Evaluation of post-hoc interpretability methods in time-series classification"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6063-9039","authenticated-orcid":false,"given":"Hugues","family":"Turb\u00e9","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6922-3283","authenticated-orcid":false,"given":"Mina","family":"Bjelogrlic","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2681-8076","authenticated-orcid":false,"given":"Christian","family":"Lovis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0157-5477","authenticated-orcid":false,"given":"Gianmarco","family":"Mengaldo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,3,13]]},"reference":[{"key":"620_CR1","doi-asserted-by":"crossref","unstructured":"Weyn, J. A., Durran, D. R. & Caruana, R. Improving data driven global weather prediction using deep convolutional neural networks on a cubed sphere. J. Adv. Modell. Earth Syst. Sep 12, e2020MS002109 (2020).","DOI":"10.1029\/2020MS002109"},{"key":"620_CR2","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1016\/j.ijinfomgt.2019.05.027","volume":"50","author":"R Yang","year":"2020","unstructured":"Yang, R. et al. Big data analytics for financial Market volatility forecast based on support vector machine. Int. J. Inf. Manage. 50, 452\u2013462 (2020).","journal-title":"Int. J. Inf. Manage."},{"key":"620_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-018-0029-1","volume":"1","author":"A Rajkomar","year":"2018","unstructured":"Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. NPJ Digit. Med. 1, 1\u201310 (2018).","journal-title":"NPJ Digit. Med."},{"key":"620_CR4","doi-asserted-by":"publisher","first-page":"1293","DOI":"10.1109\/JAS.2019.1911747","volume":"6","author":"HA Dau","year":"2019","unstructured":"Dau, H. A. et al. The UCR time series archive. IEEE\/CAA J. Autom. Sin. 6, 1293\u20131305 (2019).","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"620_CR5","doi-asserted-by":"crossref","unstructured":"Manibardo, E. L., La\u00f1a, I. & Del Ser, J. Deep learning for road traffic forecasting: does it make a difference? IEEE Trans. Intell. Transp. Syst. 23, 6164\u20136188 (2021).","DOI":"10.1109\/TITS.2021.3083957"},{"key":"620_CR6","doi-asserted-by":"crossref","unstructured":"Ye, L & Keogh, E. Time series shapelets: a new primitive for data mining. In Proc. 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 947\u2013956 (ACM, 2009).","DOI":"10.1145\/1557019.1557122"},{"key":"620_CR7","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1016\/j.ijforecast.2020.06.008","volume":"37","author":"H Hewamalage","year":"2021","unstructured":"Hewamalage, H., Bergmeir, C. & Bandara, K. Recurrent neural networks for time series forecasting: current status and future directions. Int. J. Forecast. 37, 388\u2013427 (2021).","journal-title":"Int. J. Forecast."},{"key":"620_CR8","doi-asserted-by":"publisher","first-page":"1748","DOI":"10.1016\/j.ijforecast.2021.03.012","volume":"37","author":"B Lim","year":"2021","unstructured":"Lim, B., Ar\u0131k, S. \u00d6., Loeff, N. & Pfister, T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. Int. J. Forecast. 37, 1748\u20131764 (2021).","journal-title":"Int. J. Forecast."},{"key":"620_CR9","unstructured":"Tang, B. & Matteson, D. S. Probabilistic transformer for time series analysis. In Advances in Neural Information Processing Systems Vol. 34, 23592\u201324608 (NeurIPS, 2021)."},{"key":"620_CR10","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1007\/s10618-019-00619-1","volume":"33","author":"HI Fawaz","year":"2019","unstructured":"Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L. & Muller, P. A. Deep learning for time series classification: a review. Data Min. Knowl. Discov. 33, 917\u2013963 (2019).","journal-title":"Data Min. Knowl. Discov."},{"key":"620_CR11","doi-asserted-by":"publisher","unstructured":"Hong, S., Zhang, W., Sun, C., Zhou, Y. & Li, H. Practical lessons on 12-lead ECG classification: meta-analysis of methods from PhysioNet\/computing in cardiology challenge 2020. Front. Physiol. https:\/\/doi.org\/10.3389\/fphys.2021.811661 (2022).","DOI":"10.3389\/fphys.2021.811661"},{"key":"620_CR12","unstructured":"Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts COM\/2021\/206 final (European Commission, Directorate-General for Communications Networks, Content and Technology, 2021); https:\/\/eur-lex.europa.eu\/legal-content\/EN\/ALL\/?uri=CELEX:52021PC0206"},{"key":"620_CR13","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton, Z. C. The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 16, 31\u201357 (2018).","journal-title":"Queue"},{"key":"620_CR14","doi-asserted-by":"crossref","unstructured":"Shad, R., Cunningham, J. P., Ashley, E. A., Langlotz, C. P. & Hiesinger, W. Designing clinically translatable artificial intelligence systems for high-dimensional medical imaging. Nat. Mach. Intell. 3, 929\u2013935 (2021).","DOI":"10.1038\/s42256-021-00399-8"},{"key":"620_CR15","unstructured":"Kokhlikyan, N. et al. Captum: a unified and generic model interpretability library for PyTorch. Preprint at https:\/\/arxiv.org\/abs\/2009.07896 (2020)."},{"key":"620_CR16","doi-asserted-by":"crossref","unstructured":"Montavon, G., Bach, S., Binder, A., Samek, W. & M\u00fcller, K. R. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recogn. 65, 211\u2013222 (2017).","DOI":"10.1016\/j.patcog.2016.11.008"},{"key":"620_CR17","unstructured":"Lundberg, S. & Lee, S.-I. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) Vol. 30 (NeurIPS, 2017)."},{"key":"620_CR18","doi-asserted-by":"publisher","first-page":"104393","DOI":"10.1016\/j.compbiomed.2021.104393","volume":"133","author":"I Neves","year":"2021","unstructured":"Neves, I. et al. Interpretable heartbeat classification using local model-agnostic explanations on ECGs. Comput. Biol. Med. 133, 104393 (2021).","journal-title":"Comput. Biol. Med."},{"key":"620_CR19","doi-asserted-by":"crossref","unstructured":"Jacovi, A. & Goldberg, Y. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proc. of the 58th Annual Meeting of the Association for Computational Linguistics 4198\u20134205 (Association for Computational Linguistics, 2020).","DOI":"10.18653\/v1\/2020.acl-main.386"},{"key":"620_CR20","unstructured":"Adebayo, J. et al. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems Vol. 31 (2018)."},{"key":"620_CR21","doi-asserted-by":"publisher","first-page":"2660","DOI":"10.1109\/TNNLS.2016.2599820","volume":"28","author":"W Samek","year":"2016","unstructured":"Samek, W., Binder, A., Montavon, G., Lapuschkin, S. & M\u00fcller, K. R. Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Netw. Learn. Syst. 28, 2660\u20132673 (2016).","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"620_CR22","unstructured":"Hooker, S., Erhan, D., Kindermans, P. J. & Kim, B. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems Vol. 32 (NeurIPS, 2019)."},{"key":"620_CR23","unstructured":"Hacohen, G., Choshen, L. & Weinshall, D. Let\u2019s agree to agree: neural networks share classification order on real datasets. In International Conference on Machine Learning 3950\u20133960 (PMLR, 2020)."},{"key":"620_CR24","doi-asserted-by":"publisher","unstructured":"Schlegel, U., Arnout, H., El-Assady, M., Oelke, D. & Keim, D. A. Towards a rigorous evaluation of XAI methods on time series. In 2019 IEEE\/CVF International Conference on Computer Vision Workshop (ICCVW) 4197\u20134201 (IEEE, 2019); https:\/\/doi.org\/10.1109\/ICCVW.2019.00516","DOI":"10.1109\/ICCVW.2019.00516"},{"key":"620_CR25","unstructured":"Ismail, A. A., Gunady, M., Corrada Bravo, H. & Feizi, S. Benchmarking deep learning interpretability in time series predictions. In Advances in Neural Information Processing Systems Vol. 33, 6441\u20136452 (2020)."},{"key":"620_CR26","doi-asserted-by":"crossref","unstructured":"Liu, B., Wang, X., Dixit, M., Kwitt, R. & Vasconcelos, N. Feature space transfer for data augmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 9090\u20139098 (IEEE, 2018).","DOI":"10.1109\/CVPR.2018.00947"},{"key":"620_CR27","unstructured":"Shrikumar, A., Greenside, P. & Kundaje, A. PMLR. Learning important features through propagating activation differences. In International Conference on Machine Learning 3145\u20133153 (ICML, 2017)."},{"key":"620_CR28","unstructured":"Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. In International Conference on Machine Learning 3319\u20133328 (PMLR, 2017)."},{"key":"620_CR29","doi-asserted-by":"crossref","unstructured":"Castro, J., G\u00f3mez, D. & Tejada, J. Polynomial calculation of the Shapley value based on sampling. Comput. Oper. Res. 36, 1726\u20131730 (2009).","DOI":"10.1016\/j.cor.2008.04.004"},{"key":"620_CR30","unstructured":"Shah, H., Jain, P. & Netrapalli, P. Do input gradients highlight discriminative features? In Advances in Neural Information Processing Systems Vol. 34, 2046\u20132059 (NeurIPS, 2021)."},{"key":"620_CR31","doi-asserted-by":"publisher","first-page":"976","DOI":"10.1016\/j.jacc.2008.12.013","volume":"53","author":"B Surawicz","year":"2009","unstructured":"Surawicz, B., Childers, R., Deal, B. J. & Gettes, L. S. AHA\/ACCF\/HRS recommendations for the standardization and interpretation of the electrocardiogram: part III: intraventricular conduction disturbances: a scientific statement from the American Heart Association Electrocardiography and Arrhythmias Committee, Council on Clinical Cardiology; the American College of Cardiology Foundation; and the Heart Rhythm Society Endorsed by the International Society for Computerized Electrocardiology. J. Am. College Cardiol. 53, 976\u2013981 (2009).","journal-title":"J. Am. College Cardiol."},{"key":"620_CR32","doi-asserted-by":"crossref","unstructured":"Cubuk, E. D., Zoph, B., Shlens, J. & Le, Q. RandAugment: practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems (eds Larochelle H. et al.) Vol. 33, 18613\u201318624 (NeurIPS, 2020).","DOI":"10.1109\/CVPRW50498.2020.00359"},{"key":"620_CR33","unstructured":"Ghiasi, G., Lin, T. Y. & Le, Q. V. Dropblock: a regularization method for convolutional networks. In Advances in Neural Information Processing Systems Vol. 31 (NeurIPS, 2018)."},{"key":"620_CR34","unstructured":"Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proc. of the 2nd International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (ICLR, 2014)."},{"key":"620_CR35","doi-asserted-by":"crossref","unstructured":"Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV) 618\u2013626 (IEEE, 2017).","DOI":"10.1109\/ICCV.2017.74"},{"key":"620_CR36","unstructured":"Srinivas, S. & Fleuret, F. Rethinking the role of gradient-based attribution methods for model interpretability. In 2021 International Conference on Learning Representations (ICLR, 2021)."},{"key":"620_CR37","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1007\/s10618-016-0483-9","volume":"31","author":"A Bagnall","year":"2017","unstructured":"Bagnall, A., Lines, J., Bostrom, A., Large, J. & Keogh, E. The Great Time Series Classification Bake Off: a review and experimental evaluation of recent algorithmic advances. Data Min. Knowl. Discov. 31, 606\u2013660 (2017).","journal-title":"Data Min. Knowl. Discov."},{"key":"620_CR38","unstructured":"Yang, C. H. H., Tsai, Y. Y. & Chen, P Y. Voice2Series: Reprogramming acoustic models for time series classification. In Proc. 38th International Conference on Machine Learning (eds Meila M. & Zhang, T.) Vol. 139, 11808\u201311819 (PMLR, 2021); https:\/\/proceedings.mlr.press\/v139\/yang21j.html"},{"key":"620_CR39","doi-asserted-by":"crossref","unstructured":"Perez Alday, E. A. et al. Classification of 12-lead ECGs: The PhysioNet\/Computing in Cardiology Challenge 2020 (PhysioNet, 2022); https:\/\/physionet.org\/content\/challenge-2020\/1.0.2\/","DOI":"10.1101\/2020.08.11.20172601"},{"key":"620_CR40","doi-asserted-by":"publisher","first-page":"1368","DOI":"10.1166\/jmihi.2018.2442","volume":"8","author":"F Liu","year":"2018","unstructured":"Liu, F. et al. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. J. Med. Imaging Health Inform. 8, 1368\u20131373 (2018).","journal-title":"J. Med. Imaging Health Inform."},{"key":"620_CR41","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1093\/eurheartj\/ehs291","volume":"34","author":"BE Bussink","year":"2012","unstructured":"Bussink, B. E. et al. Right bundle branch block: prevalence, risk factors, and outcome in the general population: results from the Copenhagen City Heart Study. European Heart J. 34, 138\u2013146 (2012).","journal-title":"European Heart J."},{"key":"620_CR42","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1109\/10.83591","volume":"38","author":"NV Thakor","year":"1991","unstructured":"Thakor, N. V. & Zhu, Y. S. Applications of adaptive filtering to ECG analysis: noise cancellation and arrhythmia detection. IEEE Trans. Biomedi. Eng. 38, 785\u2013794 (1991).","journal-title":"IEEE Trans. Biomedi. Eng."},{"key":"620_CR43","doi-asserted-by":"publisher","first-page":"1052","DOI":"10.1109\/TBME.1985.325514","volume":"BME-32","author":"JA Van Alste","year":"1985","unstructured":"Van Alste, J. A. & Schilder, T. S. Removal of base-line wander and power-line interference from the ECG by an efficient FIR filter with a reduced number of taps. IEEE Trans. Biomed. Eng. BME-32, 1052\u20131060 (1985).","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"620_CR44","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1016\/0010-4809(86)90037-6","volume":"19","author":"JA van Alst\u00e9","year":"1986","unstructured":"van Alst\u00e9, J. A., van Eck, W. & Herrmann, O. E. ECG baseline wander reduction using linear phase filters. Comput. Biomed. Res. 19, 417\u2013427 (1986).","journal-title":"Comput. Biomed. Res."},{"key":"620_CR45","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1093\/biomet\/81.3.425","volume":"81","author":"DL Donoho","year":"1994","unstructured":"Donoho, D. L. & Johnstone, I. M. Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425\u2013455 (1994).","journal-title":"Biometrika"},{"key":"620_CR46","unstructured":"Carreiras, C. et al. BioSPPy: Biosignal Processing in Python (GitHub, 2018); https:\/\/github.com\/PIA-Group\/BioSPPy\/"},{"key":"620_CR47","doi-asserted-by":"publisher","unstructured":"Turb\u00e9, H., Bjelogrlic, M., Lovis, C. & Mengaldo, G. Dataset: Evaluation of Post-Hoc Interpretability Methods in Time-Series Classification (Zenodo, 2023);: https:\/\/doi.org\/10.5281\/zenodo.7534770","DOI":"10.5281\/zenodo.7534770"},{"key":"620_CR48","doi-asserted-by":"publisher","unstructured":"Turb\u00e9, H, Bjelogrlic, M, Lovis, C, Mengaldo, G. hturbe\/InterpretTime: Initial Release to Replicate Results of the Submitted Article (Zenodo, 2023); https:\/\/doi.org\/10.5281\/zenodo.7560836","DOI":"10.5281\/zenodo.7560836"}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00620-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00620-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00620-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T00:08:50Z","timestamp":1679530130000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-023-00620-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,13]]},"references-count":48,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["620"],"URL":"https:\/\/doi.org\/10.1038\/s42256-023-00620-w","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,13]]},"assertion":[{"value":"7 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 January 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 March 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}