{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T09:10:11Z","timestamp":1770282611299,"version":"3.49.0"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T00:00:00Z","timestamp":1755561600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T00:00:00Z","timestamp":1755561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005005","name":"Ben-Gurion University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005005","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Model-agnostic feature attribution techniques are used to explain the decisions of complex machine learning (ML) models including ensemble models, and deep neural networks (DNNs). However, since complex ML models perform best when trained on low-level features, the explanations generated by these algorithms are often not interpretable or usable by humans. Recently proposed model-agnostic methods that support the generation of human-interpretable explanations are impractical because they require a fully invertible transformation function that maps the model\u2019s input features to human-interpretable features. While some practical human-interpretable explainability methods exist (e.g., concept-based methods), they typically require direct access to the model and are not fully model-agnostic. In this paper, we introduce Latent SHAP, a model-agnostic black-box feature attribution framework that provides human-interpretable explanations without necessitating a fully invertible transformation function. We validate the fidelity of Latent SHAP \u2019s explanations through quantitative faithfulness assessments on two controlled datasets\u2014a self-generated artificial dataset and the dSprites dataset. Furthermore, we showcase the practical utility of Latent SHAP in various real-world scenarios across domains such as computer vision, natural language processing, and cybersecurity. Each domain involves complex models (ensembles, DNNs, and LLMs), where invertible transformation functions are not available.<\/jats:p>","DOI":"10.1007\/s10994-025-06852-8","type":"journal-article","created":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T21:54:03Z","timestamp":1755640443000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Toward practical human-interpretable explanations"],"prefix":"10.1007","volume":"114","author":[{"given":"Alon","family":"Malach","sequence":"first","affiliation":[]},{"given":"Amiel","family":"Meiseles","sequence":"additional","affiliation":[]},{"given":"Ron","family":"Bitton","sequence":"additional","affiliation":[]},{"given":"Satoru","family":"Momiyama","sequence":"additional","affiliation":[]},{"given":"Toshinori","family":"Araki","sequence":"additional","affiliation":[]},{"given":"Jun","family":"Furukawa","sequence":"additional","affiliation":[]},{"given":"Yuval","family":"Elovici","sequence":"additional","affiliation":[]},{"given":"Asaf","family":"Shabtai","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,8,19]]},"reference":[{"key":"6852_CR1","doi-asserted-by":"publisher","first-page":"2644","DOI":"10.1109\/ICSMC.2001.972963","volume":"4","author":"P Aarabi","year":"2001","unstructured":"Aarabi, P., Hughes, D., Mohajer, K., & Emami, M. (2001). The automatic measurement of facial beauty. IEEE International Conference on Systems, Man and Cybernetics, 4, 2644\u201326474. https:\/\/doi.org\/10.1109\/ICSMC.2001.972963","journal-title":"IEEE International Conference on Systems, Man and Cybernetics"},{"issue":"4","key":"6852_CR2","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1002\/wics.101","volume":"2","author":"H Abdi","year":"2010","unstructured":"Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4), 433\u2013459.","journal-title":"Wiley interdisciplinary reviews: computational statistics"},{"key":"6852_CR3","doi-asserted-by":"publisher","DOI":"10.24432\/C5CC84","author":"T Almeida","year":"2012","unstructured":"Almeida, T., & Hidalgo, J. (2012). SMS spam collection. UCI Machine Learning Repository. https:\/\/doi.org\/10.24432\/C5CC84","journal-title":"UCI Machine Learning Repository"},{"issue":"1","key":"6852_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12911-020-01332-6","volume":"20","author":"J Amann","year":"2020","unstructured":"Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020). Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Medical Informatics and Decision Making, 20(1), 1\u20139.","journal-title":"BMC Medical Informatics and Decision Making"},{"issue":"1","key":"6852_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/sdata.2017.117","volume":"4","author":"S Bakas","year":"2017","unstructured":"Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., Freymann, J. B., Farahani, K., & Davatzikos, C. (2017). Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific Data, 4(1), 1\u201313.","journal-title":"Scientific Data"},{"key":"6852_CR5","first-page":"1","volume":"10","author":"S Bakas","year":"2018","unstructured":"Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R., Berger, C., Ha, S., Rozycki, M., & Prastawa, M. (2018). Identifying the best machine learning algorithms for brain tumor segmentation. Progression Assessment, and Overall Survival Prediction in the BRATS Challenge, 10, 1.","journal-title":"Progression Assessment, and Overall Survival Prediction in the BRATS Challenge"},{"key":"6852_CR7","doi-asserted-by":"crossref","unstructured":"Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017). Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6541\u20136549.","DOI":"10.1109\/CVPR.2017.354"},{"key":"6852_CR8","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"6852_CR9","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1109\/COMST.2015.2494502","volume":"18","author":"AL Buczak","year":"2015","unstructured":"Buczak, A. L., & Guven, E. (2015). A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys & Tutorials, 18(2), 1153\u20131176.","journal-title":"IEEE Communications Surveys & Tutorials"},{"key":"6852_CR10","doi-asserted-by":"publisher","unstructured":"Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171\u20134186. Association for Computational Linguistics, Minneapolis, Minnesota. https:\/\/doi.org\/10.18653\/v1\/N19-1423. https:\/\/aclanthology.org\/N19-1423.","DOI":"10.18653\/v1\/N19-1423"},{"key":"6852_CR11","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1201\/9781003187158-23","volume-title":"AI, Machine Learning and Deep Learning","author":"R Eckhardt","year":"2023","unstructured":"Eckhardt, R., & Bagui, S. (2023). A user-centric focus for detecting phishing emails. AI, Machine Learning and Deep Learning (pp. 313\u2013333). CRC Press."},{"issue":"12","key":"6852_CR12","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0078506","volume":"8","author":"F Eyben","year":"2013","unstructured":"Eyben, F., Weninger, F., Lehment, N. H., Schuller, B., & Rigoll, G. (2013). Affective video retrieval: Violence detection in hollywood movies by large-scale segmental feature extraction. PLoS ONE, 8(12), Article e78506.","journal-title":"PLoS ONE"},{"key":"6852_CR13","first-page":"1","volume":"32","author":"A Ghorbani","year":"2019","unstructured":"Ghorbani, A., Wexler, J., Zou, J. Y., & Kim, B. (2019). Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32, 1.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6852_CR14","unstructured":"Goyal, Y., Feder, A., Shalit, U., & Kim, B. (2019). Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv:1907.07165."},{"key":"6852_CR15","doi-asserted-by":"crossref","unstructured":"Guo, W., Mu, D., Xu, J., Su, P., Wang, G., & Xing, X. (2018). Lemna: Explaining deep learning based security applications. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 364\u2013379.","DOI":"10.1145\/3243734.3243792"},{"key":"6852_CR16","doi-asserted-by":"crossref","unstructured":"Hassan, W.U., Bates, A., & Marino, D. (2020). Tactical provenance analysis for endpoint detection and response systems. In 2020 IEEE Symposium on Security and Privacy (SP), pp. 1172\u20131189. IEEE.","DOI":"10.1109\/SP40000.2020.00096"},{"key":"6852_CR17","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026\u20131034.","DOI":"10.1109\/ICCV.2015.123"},{"key":"6852_CR18","doi-asserted-by":"crossref","unstructured":"Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., & Bolukbasi, T. (2021). Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 5050\u20135058.","DOI":"10.1109\/CVPR46437.2021.00501"},{"key":"6852_CR19","doi-asserted-by":"crossref","unstructured":"Kim, J., & Canny, J. (2017). Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).","DOI":"10.1109\/ICCV.2017.320"},{"key":"6852_CR20","unstructured":"Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., & Viegas, F. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pp. 2668\u20132677. PMLR."},{"key":"6852_CR21","unstructured":"Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097\u20131105."},{"key":"6852_CR22","unstructured":"Li, J., Li, D., Xiong, C., & Hoi, S. (2022). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888\u201312900. PMLR."},{"issue":"8","key":"6852_CR24","doi-asserted-by":"publisher","first-page":"7732","DOI":"10.1109\/TCYB.2021.3049630","volume":"52","author":"J Li","year":"2021","unstructured":"Li, J., Zhang, C., Zhou, J. T., Fu, H., Xia, S., & Hu, Q. (2021). Deep-lift: Deep label-specific feature learning for image annotation. IEEE Transactions on Cybernetics, 52(8), 7732\u20137741.","journal-title":"IEEE Transactions on Cybernetics"},{"key":"6852_CR23","unstructured":"Liu, Z., Luo, P., Wang, X., & Tang, X. (2018). Large-scale celebfaces attributes (celeba) dataset. Retrieved August 15(2018), 11."},{"key":"6852_CR25","first-page":"1","volume":"32","author":"G Lorberbom","year":"2019","unstructured":"Lorberbom, G., Gane, A., Jaakkola, T., & Hazan, T. (2019). Direct optimization through argmax for discrete variational auto-encoder. Advances in Neural Information Processing Systems, 32, 1.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6852_CR26","first-page":"1","volume":"30","author":"SM Lundberg","year":"2017","unstructured":"Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 1.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6852_CR27","unstructured":"Matthey, L., Higgins, I., Hassabis, D., & Lerchner, A. (2017). dSprites: Disentanglement testing Sprites dataset. https:\/\/github.com\/deepmind\/dsprites-dataset\/."},{"issue":"10","key":"6852_CR28","doi-asserted-by":"publisher","first-page":"1993","DOI":"10.1109\/TMI.2014.2377694","volume":"34","author":"BH Menze","year":"2015","unstructured":"Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M.-A., Arbel, T., Avants, B. B., Ayache, N., Buendia, P., Collins, D. L., Cordier, N., \u2026 Van Leemput, K. (2015). The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging, 34(10), 1993\u20132024. https:\/\/doi.org\/10.1109\/TMI.2014.2377694","journal-title":"IEEE Transactions on Medical Imaging"},{"key":"6852_CR29","unstructured":"Mijolla, D., Frye, C., Kunesch, M., Mansir, J., & Feige, I. (2020). Human-interpretable model explainability on high-dimensional data. arXiv preprint arXiv:2010.07384."},{"key":"6852_CR30","doi-asserted-by":"crossref","unstructured":"Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & M\u00fcller, K.-R. (2019). Layer-wise relevance propagation: An overview. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 193\u2013209.","DOI":"10.1007\/978-3-030-28954-6_10"},{"key":"6852_CR31","doi-asserted-by":"publisher","first-page":"2628","DOI":"10.1166\/jmihi.2020.3263","volume":"10","author":"AS Oliver","year":"2020","unstructured":"Oliver, A. S., Anuradha, M., John, J. J., Bellam, K., & Jayasankar, T. (2020). An efficient coding network based feature extraction with support vector machine based classification model for ct lung images. Journal Medical Imaging Health Informatics, 10, 2628\u20132633.","journal-title":"Journal Medical Imaging Health Informatics"},{"key":"6852_CR32","unstructured":"Ramaswamy, V.V., Kim, S.S., Meister, N., Fong, R., & Russakovsky, O. (2022). Elude: Generating interpretable explanations via a decomposition into labelled and unlabelled features. arXiv preprint arXiv:2206.07690."},{"key":"6852_CR33","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., & Guestrin, C. (2016). \"Why should i trust you?\" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135\u20131144.","DOI":"10.1145\/2939672.2939778"},{"key":"6852_CR34","doi-asserted-by":"crossref","unstructured":"Roy, A., Sun, J., Mahoney, R., Alonzi, L., Adams, S., & Beling, P. (2018). Deep learning detecting fraud in credit card transactions. In 2018 Systems and Information Engineering Design Symposium (SIEDS), pp. 129\u2013134. IEEE.","DOI":"10.1109\/SIEDS.2018.8374722"},{"issue":"5","key":"6852_CR35","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","volume":"1","author":"C Rudin","year":"2019","unstructured":"Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206\u2013215.","journal-title":"Nature Machine Intelligence"},{"key":"6852_CR36","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2019.113100","volume":"144","author":"S Sachan","year":"2020","unstructured":"Sachan, S., Yang, J.-B., Xu, D.-L., Benavides, D. E., & Li, Y. (2020). An explainable ai decision-support-system to automate loan underwriting. Expert Systems with Applications, 144, Article 113100.","journal-title":"Expert Systems with Applications"},{"key":"6852_CR37","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1016\/j.inffus.2020.03.013","volume":"61","author":"O Sagi","year":"2020","unstructured":"Sagi, O., & Rokach, L. (2020). Explainable decision forest: Transforming a decision forest into an interpretable tree. Information Fusion, 61, 124\u2013138.","journal-title":"Information Fusion"},{"issue":"1","key":"6852_CR38","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1021\/acscentsci.7b00512","volume":"4","author":"MH Segler","year":"2018","unstructured":"Segler, M. H., Kogej, T., Tyrchan, C., & Waller, M. P. (2018). Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, 4(1), 120\u2013131.","journal-title":"ACS Central Science"},{"key":"6852_CR39","unstructured":"Selvaraju, R.R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016). Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450."},{"key":"6852_CR40","doi-asserted-by":"publisher","unstructured":"Shapley, L. (2020). 7. A Value for n-Person Games. Contributions to the Theory of Games II (1953) 307-317., pp. 69\u201379. Princeton University Press, Princeton. https:\/\/doi.org\/10.1515\/9781400829156-012.","DOI":"10.1515\/9781400829156-012"},{"key":"6852_CR41","unstructured":"Smilkov, D., Thorat, N., Kim, B., Vi\u00e9gas, F., & Wattenberg, M. (2017). Smoothgrad: Removing noise by adding noise. arXiv preprint arXiv:1706.03825."},{"issue":"21","key":"6852_CR42","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1158\/0008-5472.CAN-17-0339","volume":"77","author":"JJ Van Griethuysen","year":"2017","unstructured":"Van Griethuysen, J. J., Fedorov, A., Parmar, C., Hosny, A., Aucoin, N., Narayan, V., Beets-Tan, R. G., Fillion-Robin, J.-C., Pieper, S., & Aerts, H. J. (2017). Computational radiomics system to decode the radiographic phenotype. Cancer Research, 77(21), 104\u2013107.","journal-title":"Cancer Research"},{"key":"6852_CR43","doi-asserted-by":"publisher","first-page":"46717","DOI":"10.1109\/ACCESS.2019.2906934","volume":"7","author":"R Vinayakumar","year":"2019","unstructured":"Vinayakumar, R., Alazab, M., Soman, K., Poornachandran, P., & Venkatraman, S. (2019). Robust intelligent malware detection using deep learning. IEEE Access, 7, 46717\u201346738.","journal-title":"IEEE Access"},{"key":"6852_CR44","first-page":"841","volume":"31","author":"S Wachter","year":"2017","unstructured":"Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31, 841.","journal-title":"Harv. JL & Tech."},{"key":"6852_CR45","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In Computer Vision\u2013ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6\u201312, 2014, Proceedings, Part I 13, pp. 818\u2013833. Springer.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"6852_CR46","doi-asserted-by":"crossref","unstructured":"Zhou, B., Sun, Y., Bau, D., & Torralba, A. (2018). Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 119\u2013134.","DOI":"10.1007\/978-3-030-01237-3_8"},{"key":"6852_CR47","doi-asserted-by":"crossref","unstructured":"Zilke, J.R., Loza\u00a0Menc\u00eda, E., & Janssen, F. (2016). Deepred\u2013rule extraction from deep neural networks. In International Conference on Discovery Science, pp. 457\u2013473. Springer.","DOI":"10.1007\/978-3-319-46307-0_29"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06852-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-025-06852-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06852-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T17:15:51Z","timestamp":1757438151000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-025-06852-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,19]]},"references-count":47,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["6852"],"URL":"https:\/\/doi.org\/10.1007\/s10994-025-06852-8","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,19]]},"assertion":[{"value":"7 August 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 June 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 July 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 August 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"209"}}