{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T19:59:52Z","timestamp":1768679992139,"version":"3.49.0"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2022,7,30]],"date-time":"2022-07-30T00:00:00Z","timestamp":1659139200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,30]],"date-time":"2022-07-30T00:00:00Z","timestamp":1659139200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["823783"],"award-info":[{"award-number":["823783"]}],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["952215"],"award-info":[{"award-number":["952215"]}],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003407","name":"Ministero dell\u2019Istruzione, dell\u2019Universit\u00e1 e della Ricerca","doi-asserted-by":"publisher","award":["1062"],"award-info":[{"award-number":["1062"]}],"id":[{"id":"10.13039\/501100003407","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2022,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We introduce and study knowledge drift (KD), a special form of concept drift that occurs in hierarchical classification. Under KD the vocabulary of concepts, their individual distributions, and the <jats:italic>is-a<\/jats:italic> relations between them can all change over time. The main challenge is that, since the ground-truth concept hierarchy is unobserved, it is hard to tell apart different forms of KD. For instance, the introduction of a new <jats:italic>is-a<\/jats:italic> relation between two concepts might be confused with changes to those individual concepts, but it is far from equivalent. Failure to identify the right kind of KD compromises the concept hierarchy used by the classifier, leading to systematic prediction errors. Our key observation is that in human-in-the-loop applications like smart personal assistants the user knows what kind of drift occurred recently, if any. Motivated by this observation, we introduce <jats:sc>trckd<\/jats:sc>, a novel approach that combines two <jats:italic>automated<\/jats:italic> stages\u2014drift detection and adaptation\u2014with a new <jats:italic>interactive<\/jats:italic> disambiguation stage in which the user is asked to refine the machine\u2019s understanding of recently detected KD. In addition, <jats:sc>trckd<\/jats:sc> implements a simple but effective <jats:italic>knowledge-aware<\/jats:italic> adaptation strategy. Our simulations show that, when the structure of the concept hierarchy drifts, a handful of queries to the user are often enough to substantially improve prediction performance on both synthetic and realistic data.<\/jats:p>","DOI":"10.1007\/s10618-022-00845-0","type":"journal-article","created":{"date-parts":[[2022,7,30]],"date-time":"2022-07-30T08:06:47Z","timestamp":1659168407000},"page":"1865-1884","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Human-in-the-loop handling of knowledge drift"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7037-5797","authenticated-orcid":false,"given":"Andrea","family":"Bontempelli","sequence":"first","affiliation":[]},{"given":"Fausto","family":"Giunchiglia","sequence":"additional","affiliation":[]},{"given":"Andrea","family":"Passerini","sequence":"additional","affiliation":[]},{"given":"Stefano","family":"Teso","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,7,30]]},"reference":[{"key":"845_CR1","doi-asserted-by":"crossref","unstructured":"Bontempelli A, Teso S, Giunchiglia F, et\u00a0al (2020) Learning in the wild with incremental skeptical gaussian processes. In: IJCAI","DOI":"10.24963\/ijcai.2020\/399"},{"key":"845_CR2","doi-asserted-by":"crossref","unstructured":"Boult TE, et\u00a0al (2019) Learning and the unknown: surveying steps toward open world recognition. In: AAAI","DOI":"10.1609\/aaai.v33i01.33019801"},{"key":"845_CR3","doi-asserted-by":"crossref","unstructured":"Cao Y, Yang J (2015) Towards making systems forget with machine unlearning. In: 2015 IEEE symposium on security and privacy","DOI":"10.1109\/SP.2015.35"},{"key":"845_CR4","doi-asserted-by":"crossref","unstructured":"Chen J, L\u00e9cu\u00e9 F, Pan J, et\u00a0al (2017) Learning from ontology streams with semantic concept drift. In: Twenty-sixth international joint conference on artificial intelligence, international joint conferences on artificial intelligence organization","DOI":"10.24963\/ijcai.2017\/133"},{"key":"845_CR5","doi-asserted-by":"crossref","unstructured":"Cohen G, Afshar S, Tapson J, et\u00a0al (2017) Emnist: extending MNIST to handwritten letters. In: IJCNN","DOI":"10.1109\/IJCNN.2017.7966217"},{"key":"845_CR6","doi-asserted-by":"publisher","first-page":"546","DOI":"10.1016\/j.eswa.2017.10.003","volume":"92","author":"J Dem\u0161ar","year":"2018","unstructured":"Dem\u0161ar J, Bosni\u0107 Z (2018) Detecting concept drift in data streams using model explanation. Expert Syst Appl 92:546\u2013559","journal-title":"Expert Syst Appl"},{"key":"845_CR7","doi-asserted-by":"crossref","unstructured":"Ding K, Li J, Liu H (2019) Interactive anomaly detection on attributed networks. In: Proceedings of the twelfth ACM international conference on web search and data mining, pp 357\u2013365","DOI":"10.1145\/3289600.3290964"},{"key":"845_CR8","doi-asserted-by":"publisher","first-page":"E10313","DOI":"10.1073\/pnas.1800755115","volume":"115","author":"T Flesch","year":"2018","unstructured":"Flesch T, Balaguer J, Dekker R et al (2018) Comparing continual task learning in minds and machines. Proc Natl Acad Sci 115:E10313\u2013E10322","journal-title":"Proc Natl Acad Sci"},{"key":"845_CR9","doi-asserted-by":"publisher","first-page":"432","DOI":"10.1093\/biostatistics\/kxm045","volume":"9","author":"J Friedman","year":"2008","unstructured":"Friedman J, Hastie T, Tibshirani R (2008) Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9:432\u2013441","journal-title":"Biostatistics"},{"key":"845_CR10","doi-asserted-by":"publisher","first-page":"44-1","DOI":"10.1145\/2523813","volume":"46","author":"J Gama","year":"2014","unstructured":"Gama J, \u017dliobait\u0117 I, Bifet A et al (2014) A survey on concept drift adaptation. ACM Comput Surv 46:44\u20131","journal-title":"ACM Comput Surv"},{"key":"845_CR11","doi-asserted-by":"crossref","unstructured":"Giunchiglia F, Bignotti E, Zeni M (2017) Personal context modelling and annotation. In: PerCom","DOI":"10.1109\/PERCOMW.2017.7917543"},{"key":"845_CR12","first-page":"723","volume":"13","author":"A Gretton","year":"2012","unstructured":"Gretton A, Borgwardt KM, Rasch MJ et al (2012) A kernel two-sample test. JMLR 13:723\u2013773","journal-title":"JMLR"},{"key":"845_CR13","unstructured":"Jitkrittum W, Szab\u00f3 Z, Chwialkowski KP, et\u00a0al (2016) Interpretable distribution features with maximum testing power. In: NeurIPS"},{"key":"845_CR14","doi-asserted-by":"crossref","unstructured":"Kifer D, Ben-David S, Gehrke J (2004) Detecting change in data streams. In: VLDB","DOI":"10.1016\/B978-012088469-8.50019-X"},{"key":"845_CR15","unstructured":"Kim B, Khanna R, Koyejo OO (2016) Examples are not enough, learn to criticize! criticism for interpretability. In: Advances in neural information processing systems, vol 29"},{"key":"845_CR16","unstructured":"Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: ICLR\u201914"},{"key":"845_CR17","volume-title":"Probabilistic graphical models: principles and techniques","author":"D Koller","year":"2009","unstructured":"Koller D, Friedman N (2009) Probabilistic graphical models: principles and techniques. MIT Press, Cambridge"},{"key":"845_CR18","unstructured":"Lloyd JR, Ghahramani Z (2015) Statistical model criticism using kernel two sample tests. In: Advances in neural information processing systems"},{"key":"845_CR19","first-page":"2346","volume":"31","author":"J Lu","year":"2018","unstructured":"Lu J, Liu A, Dong F et al (2018) Learning under concept drift: a review. IEEE Trans Knowl Data Eng 31:2346\u20132363","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"845_CR20","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1109\/TKDE.2010.61","volume":"23","author":"M Masud","year":"2011","unstructured":"Masud M, Gao J, Khan L et al (2011) Classification and novel class detection in concept-drifting data streams under time constraints. IEEE Trans Knowl Data Eng 23:859\u2013874","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"845_CR21","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.neunet.2019.01.012","volume":"113","author":"GI Parisi","year":"2019","unstructured":"Parisi GI, Kemker R, Part JL et al (2019) Continual lifelong learning with neural networks: a review. Neural Netw 113:54\u201371","journal-title":"Neural Netw"},{"key":"845_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3406243","volume":"14","author":"R Paudel","year":"2020","unstructured":"Paudel R, Eberle W (2020) An approach for concept drift detection in a graph stream using discriminative subgraphs. TKDD 14:1\u201325","journal-title":"TKDD"},{"key":"845_CR23","unstructured":"P\u00e9rez-Cruz F (2009) Estimation of information theoretic measures for continuous random variables. In: NeurIPS"},{"key":"845_CR24","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/j.sigpro.2013.12.026","volume":"99","author":"MA Pimentel","year":"2014","unstructured":"Pimentel MA, Clifton DA, Clifton L et al (2014) A review of novelty detection. Signal Process 99:215\u2013249","journal-title":"Signal Process"},{"key":"845_CR25","doi-asserted-by":"crossref","unstructured":"Rebuffi SA, Kolesnikov A, Sperl G, et\u00a0al (2017) ICARL: incremental classifier and representation learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition","DOI":"10.1109\/CVPR.2017.587"},{"key":"845_CR26","doi-asserted-by":"crossref","unstructured":"Redko I, Morvant E, Habrard A, et\u00a0al (2020) A survey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv preprint arXiv:2004.11829","DOI":"10.1016\/B978-1-78548-236-6.50002-7"},{"key":"845_CR27","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I (2019) Sentence-bert: sentence embeddings using siamese bert-networks. In: EMNLP-IJCNLP","DOI":"10.18653\/v1\/D19-1410"},{"key":"845_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3363573","volume":"13","author":"M Roseberry","year":"2019","unstructured":"Roseberry M, Krawczyk B, Cano A (2019) Multi-label punitive kNN with self-adjusting memory for drifting data streams. TKDD 13:1\u201331","journal-title":"TKDD"},{"key":"845_CR29","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1109\/TPAMI.2017.2707495","volume":"40","author":"EM Rudd","year":"2017","unstructured":"Rudd EM, Jain LP, Scheirer WJ et al (2017) The extreme value machine. IEEE Trans Pattern Anal Mach Intell 40:762\u2013768","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"845_CR30","doi-asserted-by":"publisher","first-page":"1757","DOI":"10.1109\/TPAMI.2012.256","volume":"35","author":"WJ Scheirer","year":"2013","unstructured":"Scheirer WJ, de Rezende Rocha A, Sapkota A et al (2013) Toward open set recognition. IEEE Trans Pattern Anal Mach Intell 35:1757\u20131772","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"845_CR31","doi-asserted-by":"publisher","first-page":"2317","DOI":"10.1109\/TPAMI.2014.2321392","volume":"36","author":"WJ Scheirer","year":"2014","unstructured":"Scheirer WJ, Jain LP, Boult TE (2014) Probability models for open set recognition. IEEE Trans Pattern Anal Mach Intell 36:2317\u20132324","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"845_CR32","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/BF00116895","volume":"1","author":"JC Schlimmer","year":"1986","unstructured":"Schlimmer JC, Granger RH (1986) Incremental learning from noisy data. Mach Learn 1:317\u2013354","journal-title":"Mach Learn"},{"key":"845_CR33","doi-asserted-by":"publisher","first-page":"476","DOI":"10.1038\/s42256-020-0212-3","volume":"2","author":"P Schramowski","year":"2020","unstructured":"Schramowski P et al (2020) Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nat Mach Intell 2:476\u2013486","journal-title":"Nat Mach Intell"},{"key":"845_CR34","unstructured":"Snell J, Swersky K, Zemel R (2017) Prototypical networks for few-shot learning. In: NeurIPS"},{"key":"845_CR35","volume-title":"A literature survey on algorithms for multi-label learning","author":"MS Sorower","year":"2010","unstructured":"Sorower MS (2010) A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis"},{"key":"845_CR36","doi-asserted-by":"crossref","unstructured":"Spinosa EJ, de\u00a0Leon F, de Carvalho AP, Gama J (2007) Olindda: a cluster-based approach for detecting novelty and concept drift in data streams. In: Proceedings of the 2007 ACM symposium on applied computing, pp 448\u2013452","DOI":"10.1145\/1244002.1244107"},{"key":"845_CR37","unstructured":"Spyromitros-Xioufis, et\u00a0al (2011) Dealing with concept drift and class imbalance in multi-label stream classification. In: IJCAI"},{"key":"845_CR38","doi-asserted-by":"crossref","unstructured":"Stojanovic L, Maedche A, Motik B, et\u00a0al (2002) User-driven ontology evolution management. In: ECAW","DOI":"10.1007\/3-540-45810-7_27"},{"key":"845_CR39","first-page":"233-1","volume":"18","author":"Z Szab\u00f3","year":"2017","unstructured":"Szab\u00f3 Z, Sriperumbudur BK (2017) Characteristic and universal tensor product kernels. JMLR 18:233\u20131","journal-title":"JMLR"},{"key":"845_CR40","unstructured":"Tong S, Koller D (2001) Active learning for structure in Bayesian networks. In: International joint conference on artificial intelligence. Citeseer, pp 863\u2013869"},{"key":"845_CR41","unstructured":"Yao Y, Holder LB (2016) Detecting concept drift in classification over streaming graphs. In: Proceedings of the KDD workshop on mining and learning with graphs"},{"key":"845_CR42","doi-asserted-by":"crossref","unstructured":"Yu L, Twardowski B, Liu X, et\u00a0al (2020) Semantic drift compensation for class-incremental learning. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","DOI":"10.1109\/CVPR42600.2020.00701"},{"key":"845_CR43","doi-asserted-by":"publisher","first-page":"5592","DOI":"10.1109\/TNNLS.2018.2804443","volume":"29","author":"D Zambon","year":"2018","unstructured":"Zambon D, Alippi C, Livi L (2018) Concept drift and anomaly detection in graph streams. IEEE Trans Neural Netw Learn Syst 29:5592\u20135605","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"845_CR44","unstructured":"Zhang K, Sch\u00f6lkopf B, Muandet K, et\u00a0al (2013) Domain adaptation under target and conditional shift. In: ICML"},{"key":"845_CR45","doi-asserted-by":"crossref","unstructured":"Zhang ML, Zhang K (2010) Multi-label learning by exploiting label dependency. In: ACM SIGKDD","DOI":"10.1145\/1835804.1835930"},{"key":"845_CR46","doi-asserted-by":"publisher","first-page":"1249","DOI":"10.1109\/ACCESS.2019.2962059","volume":"8","author":"X Zheng","year":"2019","unstructured":"Zheng X, Li P, Chu Z et al (2019) A survey on multi-label data stream classification. IEEE Access 8:1249\u20131275","journal-title":"IEEE Access"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00845-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-022-00845-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00845-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,6]],"date-time":"2022-10-06T10:13:08Z","timestamp":1665051188000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-022-00845-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,30]]},"references-count":46,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,9]]}},"alternative-id":["845"],"URL":"https:\/\/doi.org\/10.1007\/s10618-022-00845-0","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,30]]},"assertion":[{"value":"8 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 May 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}