{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T23:24:35Z","timestamp":1778628275217,"version":"3.51.4"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"3-4","license":[{"start":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T00:00:00Z","timestamp":1657584000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T00:00:00Z","timestamp":1657584000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["438445824"],"award-info":[{"award-number":["438445824"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005722","name":"Ludwig-Maximilians-Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005722","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Explainable Artificial Intelligence (XAI) has mainly focused on static learning tasks so far. In this paper, we consider XAI in the context of online learning in dynamic environments, such as learning from real-time data streams, where models are learned incrementally and continuously adapted over the course of time. More specifically, we motivate the problem of<jats:italic>explaining model change<\/jats:italic>, i.e. explaining the difference between models before and after adaptation, instead of the models themselves. In this regard, we provide the first efficient model-agnostic approach to dynamically detecting, quantifying, and explaining significant model changes. Our approach is based on an adaptation of the well-known Permutation Feature Importance (PFI) measure. It includes two hyperparameters that control the sensitivity and directly influence explanation frequency, so that a human user can adjust the method to individual requirements and application needs. We assess and validate our method\u2019s efficacy on illustrative synthetic data streams with three popular model classes.<\/jats:p>","DOI":"10.1007\/s13218-022-00766-6","type":"journal-article","created":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T09:15:24Z","timestamp":1657617324000},"page":"211-224","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Agnostic Explanation of Model Change based on Feature Importance"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6921-0204","authenticated-orcid":false,"given":"Maximilian","family":"Muschalik","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3955-3510","authenticated-orcid":false,"given":"Fabian","family":"Fumagalli","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0935-5591","authenticated-orcid":false,"given":"Barbara","family":"Hammer","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9944-4108","authenticated-orcid":false,"given":"Eyke","family":"H\u00fcllermeier","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,7,12]]},"reference":[{"key":"766_CR1","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6:52138\u201352160. https:\/\/doi.org\/10.1109\/ACCESS.2018.2870052","journal-title":"IEEE Access"},{"issue":"6","key":"766_CR2","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1109\/69.250074","volume":"5","author":"R Agrawal","year":"1993","unstructured":"Agrawal R, Imielinski T, Swami A (1993) Database mining: a performance perspective. IEEE Trans Knowl Data Eng 5(6):914\u2013925. https:\/\/doi.org\/10.1109\/69.250074","journal-title":"IEEE Trans Knowl Data Eng"},{"issue":"3","key":"766_CR3","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garcia S, Gil-Lopez S, Molina D, Benjamins R, Chatila R, Herrera F (2020) Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible ai. inform fusion 58(3):82\u2013115. https:\/\/doi.org\/10.1016\/j.inffus.2019.12.012","journal-title":"Inform Fusion"},{"key":"766_CR4","doi-asserted-by":"publisher","unstructured":"Atkinson R, Shiffrin R (1968) Human memory: a proposed system and its control processes. In: Psychology of Learning and Motivation, 2, 89\u2013195. Academic Press. https:\/\/doi.org\/10.1016\/S0079-7421(08)60422-3","DOI":"10.1016\/S0079-7421(08)60422-3"},{"key":"766_CR5","doi-asserted-by":"publisher","unstructured":"Bahri M, Bifet A, Gama J, Gomes HM, Maniu S (2021) Data stream analysis: Foundations, major tasks and tools. Wiley Interdisciplin Rev Data Mining Knowl Discovery 11(3):e1405. https:\/\/doi.org\/10.1002\/widm.1405","DOI":"10.1002\/widm.1405"},{"key":"766_CR6","doi-asserted-by":"publisher","unstructured":"Bifet A, Gavald\u00e0 R (2007) Learning from time-changing data with adaptive windowing. In: Proceedings of International Conference on Data Mining (SIAM), pp. 443\u2013448. https:\/\/doi.org\/10.1137\/1.9781611972771.42","DOI":"10.1137\/1.9781611972771.42"},{"key":"766_CR7","doi-asserted-by":"publisher","unstructured":"Bifet A, Gavald\u00e0 R (2009) Adaptive learning from evolving data sreams. In: Proceedings of International Symposium on Intelligent Data Analysis (IDA), pp. 249\u2013260 . https:\/\/doi.org\/10.1007\/978-3-642-03915-7_22","DOI":"10.1007\/978-3-642-03915-7_22"},{"key":"766_CR8","doi-asserted-by":"publisher","unstructured":"Biswas R, Barz M, Sonntag D (2020) Towards explanatory interactive image captioning using top-down and bottom-up features, beam search and re-ranking. KI - K\u00fcnstliche Intelligenz 34(4):571\u2013584. https:\/\/doi.org\/10.1007\/s13218-020-00679-2","DOI":"10.1007\/s13218-020-00679-2"},{"key":"766_CR9","doi-asserted-by":"publisher","unstructured":"Boettcher M (2011) Contrast and change mining. WIREs data mining knowl discovery 1(3):215\u2013230, e1405. https:\/\/doi.org\/10.1002\/widm.27","DOI":"10.1002\/widm.27"},{"key":"766_CR10","doi-asserted-by":"publisher","unstructured":"Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Proceedings of international conference on computational statistics (COMPSTAT), pp. 177\u2013186 . https:\/\/doi.org\/10.1007\/978-3-7908-2604-3_16","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"766_CR11","doi-asserted-by":"publisher","unstructured":"Breiman L (2001) Random Forests. Mach Learn 45(1):5\u201332, e1405 https:\/\/doi.org\/10.1023\/A:1010933404324","DOI":"10.1023\/A:1010933404324"},{"key":"766_CR12","doi-asserted-by":"publisher","unstructured":"Burkart N, Huber MF (2021) A larning. J Artif Intellig Res 70:245\u2013317, e1405 https:\/\/doi.org\/10.1613\/jair.1.12228","DOI":"10.1613\/jair.1.12228"},{"key":"766_CR13","unstructured":"Covert I, Lundberg SM, Lee SI (2020) Understanding global feature contributions with additive importance measures. In: Proceedings of international conference on neural information processing systems (NeurIPS), pp. 17212\u201317223"},{"key":"766_CR14","unstructured":"Dasarathy BV (1991) Nearest neighbor (NN) Norms: Nn pattern classification techniques. IEEE Computer Society Press"},{"key":"766_CR15","doi-asserted-by":"publisher","unstructured":"Domingos P, Hulten G (2000) Mining high-speed data streams. In: Proceedings of International conference on knowledge discovery and data mining (KDD), pp. 71\u201380 . https:\/\/doi.org\/10.1145\/347090.347107","DOI":"10.1145\/347090.347107"},{"key":"766_CR16","doi-asserted-by":"publisher","unstructured":"Gama J, Fernandes R, Rocha R (2006) Decision trees for mining data streams. Intellig Data Anal 10(1):23\u201345, e1405. https:\/\/doi.org\/10.3233\/IDA-2006-10103","DOI":"10.3233\/IDA-2006-10103"},{"key":"766_CR17","doi-asserted-by":"publisher","unstructured":"Gama J, Medas P, Castillo G, Rodrigues P (2004) Learning with drift detection. In: Proceedings of Brazilian ligence (SBIA), pp. 286\u2013295 . https:\/\/doi.org\/10.1007\/978-3-540-28645-5_29","DOI":"10.1007\/978-3-540-28645-5_29"},{"key":"766_CR18","doi-asserted-by":"publisher","unstructured":"Gama J, \u017dliobait\u0117 I, Bifet A, Pechenizkiy M, Bouchachia A (2014) A survey on concept drift adaptation. ACM Comput Surv 46(4):1\u201337, e1405. https:\/\/doi.org\/10.1145\/2523813","DOI":"10.1145\/2523813"},{"key":"766_CR19","doi-asserted-by":"publisher","unstructured":"Gomes HM, Bifet A, Read J, Barddal JP, Enembreck F, Pfharinger B, Holmes G, Abdessalem T (2017) Adaptive random forests for evolving data stream classification. Mach Learn 106(9):1469\u20131495, e1405. https:\/\/doi.org\/10.1007\/s10994-017-5642-8","DOI":"10.1007\/s10994-017-5642-8"},{"key":"766_CR20","doi-asserted-by":"crossref","unstructured":"Hammer B, H\u00fcllermeier E (2021) Interpretable machine learning: On the problem of explaining model change. In: Proceedings of workshop computation intelligence (CI), pp. 1\u201310","DOI":"10.58895\/ksp\/1000138532-1"},{"key":"766_CR21","doi-asserted-by":"crossref","unstructured":"Hastie T, Tibshirani R, Friedman JH (2009) The elements of statistical learning: Data Mining, Inference, and Prediction, 2 edn. Springer","DOI":"10.1007\/978-0-387-84858-7"},{"key":"766_CR22","unstructured":"Hinder F, Hammer B (2020) Counterfactual explanations of concept drift. CoRR. arXiv:2006.12822"},{"key":"766_CR23","unstructured":"Hinder F, Jakob J, Hammer B (2020) Analysis of drifting features. CoRR. arXiv:2012.00499"},{"key":"766_CR24","doi-asserted-by":"publisher","unstructured":"Hoeffding W (1994) Probability inequalities for sums of bounded random variables. In: The Collected Works of Wassily Hoeffding, pp. 409\u2013426. Springer. https:\/\/doi.org\/10.1007\/978-1-4612-0865-5_26","DOI":"10.1007\/978-1-4612-0865-5_26"},{"key":"766_CR25","doi-asserted-by":"publisher","unstructured":"Hulten G, Spencer L, Domingos P (2001) Mining time-changing data streams. In: Proceedings of International conference on knowledge discovery and data mining (KDD), pp. 97\u2013106 . https:\/\/doi.org\/10.1145\/502512.502529","DOI":"10.1145\/502512.502529"},{"key":"766_CR26","doi-asserted-by":"publisher","unstructured":"Linardatos P, Papastefanopoulos V, Kotsiantis S (2020) Explainable AI: A review of machine learning interpretability methods. Entropy. https:\/\/doi.org\/10.3390\/e23010018","DOI":"10.3390\/e23010018"},{"key":"766_CR27","doi-asserted-by":"publisher","unstructured":"Losing V, Hammer B, Wersing H (2016) KNN classifier with self adjusting memory for heterogeneous concept drift. In: Proceedings of international conference on data mining (ICDM), pp. 291\u2013300 . https:\/\/doi.org\/10.1109\/ICDM.2016.0040","DOI":"10.1109\/ICDM.2016.0040"},{"key":"766_CR28","doi-asserted-by":"publisher","unstructured":"Losing V, Hammer B, Wersing H (2018) Incremental on-line learning: a review and comparison of state of the art algorithms. Neurocomputing 275:1261\u20131274, e1405. https:\/\/doi.org\/10.1016\/j.neucom.2017.06.084","DOI":"10.1016\/j.neucom.2017.06.084"},{"key":"766_CR29","doi-asserted-by":"publisher","unstructured":"Lu J, Liu A, Dong F, Gu F, Gama J, Zhang G (2018) Learning under concept drift: A Review. IEEE transactions on knowledge and data engineering pp. 2346\u20132363. https:\/\/doi.org\/10.1109\/TKDE.2018.2876857","DOI":"10.1109\/TKDE.2018.2876857"},{"key":"766_CR30","doi-asserted-by":"publisher","unstructured":"Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, Katz R, Himmelfarb J, Bansal N, Lee SI (2020) From local explanations to global understanding with explainable AI for Trees. Nat Mach Intellig 2(1):56\u201367, e1405. https:\/\/doi.org\/10.1038\/s42256-019-0138-9","DOI":"10.1038\/s42256-019-0138-9"},{"key":"766_CR31","unstructured":"Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. In: Proceedings of international conference on neural information processing systems (NeurIPS), pp. 4768\u20134777"},{"key":"766_CR32","doi-asserted-by":"publisher","unstructured":"Manapragada C, Webb GI, Salehi M (2018) Extremely fast decision tree. In: Proceedings of international conference on knowledge discovery and data mining (KDD), pp. 1953\u20131962 . https:\/\/doi.org\/10.1145\/3219819.3220005","DOI":"10.1145\/3219819.3220005"},{"key":"766_CR33","unstructured":"Molnar C (2019) Interpretable machine learning: A Guide for Making Black Box Models Explainable. Lulu.com"},{"key":"766_CR34","unstructured":"Montiel J, Halford M, Mastelini SM, Bolmier G, Sourty R, Vaysse R, Zouitine A, Gomes HM, Read J, Abdessalem T, Bifet A (2020) River: machine learning for streaming data in Python. CoRR. arXiv:2012.04740"},{"key":"766_CR35","unstructured":"Montiel J, Read J, Bifet A, Abdessalem T (2018) Scikit-Multiflow: A multi-output streaming framework. J Mach Learn Res 19(72):1\u20135"},{"key":"766_CR36","doi-asserted-by":"publisher","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) Why Should I Trust You? Explaining the Predictions of Any Classifier. In: Proceedings of international conference on knowledge discovery and data mining (KDD), pp. 1135\u20131144 . https:\/\/doi.org\/10.1145\/2939672.2939778","DOI":"10.1145\/2939672.2939778"},{"issue":"1","key":"766_CR37","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1007\/s10107-015-0901-6","volume":"156","author":"P Richt\u00e1rik","year":"2016","unstructured":"Richt\u00e1rik P, Tak\u00e1\u010d M (2016) Parallel coordinate descent methods for big data optimization. Math Program 156(1):433\u2013484. https:\/\/doi.org\/10.1007\/s10107-015-0901-6","journal-title":"Mathematical programming"},{"key":"766_CR38","doi-asserted-by":"publisher","unstructured":"Saffari A, Leistner C, Santner J, Godec M, Bischof H (2009) On-line Random Forests. In: Proceedings of International conference on computer vision workshops (ICCV Workshops), pp. 1393\u20131400. IEEE . https:\/\/doi.org\/10.1109\/ICCVW.2009.5457447","DOI":"10.1109\/ICCVW.2009.5457447"},{"issue":"3","key":"766_CR39","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/BF00116895","volume":"1","author":"JC Schlimmer","year":"1986","unstructured":"Schlimmer JC, Granger RH (1986) Incremental learning from noisy data. Mach Learn 1(3):317\u2013354. https:\/\/doi.org\/10.1007\/BF00116895","journal-title":"Mach Learn"},{"issue":"4","key":"766_CR40","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1007\/s12530-012-9059-0","volume":"3","author":"A Shaker","year":"2012","unstructured":"Shaker A, H\u00fcllermeier E (2012) IBLStreams: a system for instance-based classification and regression on data streams. Evolv Syst 3(4):235\u2013249. https:\/\/doi.org\/10.1007\/s12530-012-9059-0","journal-title":"Evolv Syst"},{"issue":"2","key":"766_CR41","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1111\/0824-7935.00142","volume":"17","author":"B Smyth","year":"2001","unstructured":"Smyth B, McKenna E (2001) Competence models and the maintenance problem. Comput Intellig 17(2):235\u2013249. https:\/\/doi.org\/10.1111\/0824-7935.00142","journal-title":"Comput Intellig"},{"key":"766_CR42","doi-asserted-by":"publisher","unstructured":"Teso S, Kersting K (2019) Explanatory interactive machine learning. In: Proceedings of AAAI\/ACM Conference on AI, Ethics, and Society (AIES), pp. 239\u2013245. https:\/\/doi.org\/10.1145\/3306618.3314293","DOI":"10.1145\/3306618.3314293"},{"issue":"5","key":"766_CR43","doi-asserted-by":"publisher","first-page":"1179","DOI":"10.1007\/s10618-018-0554-1","volume":"32","author":"GI Webb","year":"2018","unstructured":"Webb GI, Lee LK, Goethals B, Petitjean F (2018) Analyzing concept drift and shift from sample data. Data Min Knowl Discov 32(5):1179\u20131199. https:\/\/doi.org\/10.1007\/s10618-018-0554-1","journal-title":"Data Mining and Knowledge Discovery"},{"key":"766_CR44","unstructured":"Webb GI, Lee LK, Petitjean F, Goethals B (2017) Understanding concept drift. CoRR. arXiv:1704.00362"},{"issue":"1","key":"766_CR45","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1007\/BF00116900","volume":"23","author":"G Widmer","year":"1996","unstructured":"Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Mach Learn 23(1):69\u2013101. https:\/\/doi.org\/10.1007\/BF00116900","journal-title":"Mach Learn"},{"issue":"4","key":"766_CR46","doi-asserted-by":"publisher","first-page":"2233","DOI":"10.1109\/TII.2014.2300753","volume":"10","author":"LD Xu","year":"2014","unstructured":"Xu LD, He W, Li S (2014) Internet of things in industries: a survey. IEEE Transact Indust Inform 10(4):2233\u20132243. https:\/\/doi.org\/10.1109\/TII.2014.2300753","journal-title":"IEEE Transact Indust Inform"},{"key":"766_CR47","doi-asserted-by":"publisher","unstructured":"Zhang T (2004) Solving large scale linear prediction problems using stochastic gradient descent algorithms. In: Proceedings of international conference on machine learning (ICML), pp. 116\u2013124 . https:\/\/doi.org\/10.1145\/1015330.1015332","DOI":"10.1145\/1015330.1015332"},{"key":"766_CR48","doi-asserted-by":"publisher","unstructured":"\u017dliobait\u0117 I, Pechenizkiy M, Gama J (2016) An overview of concept drift applications, pp. 91\u2013114. Springer International Publishing.https:\/\/doi.org\/10.1007\/978-3-319-26989-4_4","DOI":"10.1007\/978-3-319-26989-4_4"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00766-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13218-022-00766-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00766-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,28]],"date-time":"2024-09-28T23:15:52Z","timestamp":1727565352000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13218-022-00766-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,12]]},"references-count":48,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["766"],"URL":"https:\/\/doi.org\/10.1007\/s13218-022-00766-6","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,12]]},"assertion":[{"value":"17 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}