{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,31]],"date-time":"2025-08-31T10:09:43Z","timestamp":1756634983041,"version":"3.37.3"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:00:00Z","timestamp":1704844800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:00:00Z","timestamp":1704844800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007105","name":"Universit\u00e4t Mannheim","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007105","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We consider generating explanations for neural networks in cases where the network\u2019s training data is not accessible, for instance due to privacy or safety issues. Recently, Interpretation Nets (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathcal {I}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>I<\/mml:mi>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>-Nets) have been proposed as a sample-free approach to post-hoc, global model interpretability that does not require access to training data. They formulate interpretation as a machine learning task that maps network representations (parameters) to a representation of an interpretable function. In this paper, we extend the <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathcal {I}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>I<\/mml:mi>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>-Net framework to the cases of standard and soft decision trees as surrogate models. We propose a suitable decision tree representation and design of the corresponding <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathcal {I}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>I<\/mml:mi>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>-Net output layers. Furthermore, we make <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathcal {I}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>I<\/mml:mi>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>-Nets applicable to real-world tasks by considering more realistic distributions when generating the <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathcal {I}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>I<\/mml:mi>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>-Net\u2019s training data. We empirically evaluate our approach against traditional global, post-hoc interpretability approaches and show that it achieves superior results when the training data is not accessible.<\/jats:p>","DOI":"10.1007\/s10994-023-06428-4","type":"journal-article","created":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T17:02:27Z","timestamp":1704906147000},"page":"3633-3652","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Explaining neural networks without access to training data"],"prefix":"10.1007","volume":"113","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8151-9223","authenticated-orcid":false,"given":"Sascha","family":"Marton","sequence":"first","affiliation":[]},{"given":"Stefan","family":"L\u00fcdtke","sequence":"additional","affiliation":[]},{"given":"Christian","family":"Bartelt","sequence":"additional","affiliation":[]},{"given":"Andrej","family":"Tschalzev","sequence":"additional","affiliation":[]},{"given":"Heiner","family":"Stuckenschmidt","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,1,10]]},"reference":[{"unstructured":"Bhardwaj, K., Suda, N., & Marculescu, R. (2019). Dream distillation: A data-independent model compression framework. arXiv preprint arXiv:1905.07072","key":"6428_CR1"},{"key":"6428_CR2","volume-title":"Converting a trained neural network to a decision tree dectext-decision tree extractor","author":"O Boz","year":"2000","unstructured":"Boz, O., & Hillman, D. (2000). Converting a trained neural network to a decision tree dectext-decision tree extractor. Citeseer."},{"key":"6428_CR3","volume-title":"Classification and regression trees","author":"L Breiman","year":"1984","unstructured":"Breiman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. CRC Press."},{"doi-asserted-by":"crossref","unstructured":"Bucilu\u01ce, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 535\u2013541).","key":"6428_CR4","DOI":"10.1145\/1150402.1150464"},{"unstructured":"Craven, M., & Shavlik, J. (1995). Extracting tree-structured representations of trained networks. Advances in neural information processing systems8.","key":"6428_CR5"},{"unstructured":"Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608","key":"6428_CR6"},{"unstructured":"Dua, D., & Graff, C. (2017). UCI machine learning repository. http:\/\/archive.ics.uci.edu\/ml","key":"6428_CR7"},{"unstructured":"Frosst, N., & Hinton, G. (2017). Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784","key":"6428_CR8"},{"unstructured":"Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., & Anandkumar, A. (2018). Born again neural networks. arXiv preprint arXiv:1805.04770","key":"6428_CR9"},{"unstructured":"Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531","key":"6428_CR10"},{"doi-asserted-by":"crossref","unstructured":"Jin, H., Song, Q., & Hu, X. (2019). Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1946\u20131956). ACM.","key":"6428_CR11","DOI":"10.1145\/3292500.3330648"},{"doi-asserted-by":"crossref","unstructured":"Johansson, U., & Niklasson, L. (2009). Evolving decision trees using oracle guides. In 2009 IEEE Symposium on Computational Intelligence and Data Mining. IEEE.","key":"6428_CR12","DOI":"10.1109\/CIDM.2009.4938655"},{"issue":"12","key":"6428_CR13","doi-asserted-by":"publisher","first-page":"1999","DOI":"10.1016\/S0031-3203(98)00181-2","volume":"32","author":"R Krishnan","year":"1999","unstructured":"Krishnan, R., Sivakumar, G., & Bhattacharya, P. (1999). Extracting decision trees from trained neural networks. Pattern Recognition, 32(12), 1999\u20132009.","journal-title":"Pattern Recognition"},{"key":"6428_CR14","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436\u2013444.","journal-title":"Nature"},{"issue":"1","key":"6428_CR15","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1198\/000313008X270448","volume":"62","author":"LM Leemis","year":"2008","unstructured":"Leemis, L. M., & McQueston, J. T. (2008). Univariate distribution relationships. The American Statistician, 62(1), 45\u201353.","journal-title":"The American Statistician"},{"issue":"3","key":"6428_CR16","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton, Z. C. (2018). The mythos of model interpretability. Queue, 16(3), 31\u201357.","journal-title":"Queue"},{"doi-asserted-by":"crossref","unstructured":"Liu, X., Wang, X., & Matwin, S. (2018). Improving the interpretability of deep neural networks with knowledge distillation. arXiv preprint arXiv:1812.10924","key":"6428_CR17","DOI":"10.1109\/ICDMW.2018.00132"},{"unstructured":"Lopes, R.G., Fenu, S., & Starner, T. (2017). Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535","key":"6428_CR18"},{"issue":"3","key":"6428_CR19","doi-asserted-by":"publisher","first-page":"980","DOI":"10.3390\/app12030980","volume":"12","author":"S Marton","year":"2022","unstructured":"Marton, S., L\u00fcdtke, S., & Bartelt, C. (2022). Explanations for neural networks by neural networks. Applied Sciences, 12(3), 980.","journal-title":"Applied Sciences"},{"unstructured":"Molnar, C. (2020). Interpretable Machine Learning. Lulu. com.","key":"6428_CR20"},{"unstructured":"Mun, J. (2015). Understanding and choosing the right probability distributions. In Advanced analytical models: Over 800 models and 300 applications from the basel II accord to Wall Street and beyond (pp. 899\u2013917).","key":"6428_CR21"},{"doi-asserted-by":"crossref","unstructured":"Murphy, P. (1991). Constructive induction of m-of-n terms. In Proceedings of the Eighth International Workshop on Machine Learning (pp. 183\u2013187).","key":"6428_CR22","DOI":"10.1016\/B978-1-55860-200-7.50040-4"},{"unstructured":"Nayak, G.K., Mopuri, K.R., Shaj, V., Babu, R.V., & Chakraborty, A. (2019). Zero-shot knowledge distillation in deep networks. arXiv preprint arXiv:1905.08114","key":"6428_CR23"},{"unstructured":"Nguyen, D.T., Kasmarik, K.E., & Abbass, H.A. (2020). Towards interpretable ANNs: An exact transformation to multi-class multivariate decision trees.","key":"6428_CR24"},{"issue":"1","key":"6428_CR25","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1007\/BF00116251","volume":"1","author":"JR Quinlan","year":"1986","unstructured":"Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81\u2013106.","journal-title":"Machine Learning"},{"key":"6428_CR26","volume-title":"C4. 5: Programs for Machine Learning","author":"JR Quinlan","year":"2014","unstructured":"Quinlan, J. R. (2014). C4. 5: Programs for Machine Learning. Elsevier."},{"key":"6428_CR27","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-28954-6","volume-title":"Explainable AI: Interpreting explaining and visualizing deep learning","author":"W Samek","year":"2019","unstructured":"Samek, W., Montavon, G., Vedaldi, A., Hansen, L. K., & Muller, K.-R. (2019). Explainable AI: Interpreting explaining and visualizing deep learning. Springer."},{"unstructured":"Tan, S., Caruana, R., Hooker, G., Koch, P., & Gordo, A. (2018). Learning global additive explanations for neural nets using model distillation.","key":"6428_CR28"},{"key":"6428_CR29","doi-asserted-by":"publisher","first-page":"747","DOI":"10.1007\/s13042-020-01096-5","volume":"11","author":"X Wang","year":"2020","unstructured":"Wang, X., Zhao, Y., & Pourpanah, F. (2020). Recent advances in deep learning. International Journal of Machine Learning and Cybernetics, 11, 747\u2013750.","journal-title":"International Journal of Machine Learning and Cybernetics"},{"issue":"2","key":"6428_CR30","doi-asserted-by":"publisher","first-page":"2473","DOI":"10.1016\/j.eswa.2007.12.020","volume":"36","author":"I-C Yeh","year":"2009","unstructured":"Yeh, I.-C., & Lien, C.-H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2), 2473\u20132480.","journal-title":"Expert Systems with Applications"},{"doi-asserted-by":"crossref","unstructured":"Zhang, Q., Yang, Y., Ma, H., & Wu, Y.N. (2019). Interpreting cnns via decision trees. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (pp. 6261\u20136270).","key":"6428_CR31","DOI":"10.1109\/CVPR.2019.00642"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-023-06428-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-023-06428-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-023-06428-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,10]],"date-time":"2024-05-10T15:11:13Z","timestamp":1715353873000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-023-06428-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,10]]},"references-count":31,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["6428"],"URL":"https:\/\/doi.org\/10.1007\/s10994-023-06428-4","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"type":"print","value":"0885-6125"},{"type":"electronic","value":"1573-0565"}],"subject":[],"published":{"date-parts":[[2024,1,10]]},"assertion":[{"value":"5 June 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 August 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 October 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 January 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}