{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T16:07:00Z","timestamp":1787501220874,"version":"build-2736575974"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T00:00:00Z","timestamp":1637193600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T00:00:00Z","timestamp":1637193600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100012687","name":"Universit\u00e4t Kassel","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100012687","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Stream-based active learning (AL) strategies minimize the labeling effort by querying labels that improve the classifier\u2019s performance the most. So far, these strategies neglect the fact that an oracle or expert requires time to provide a queried label. We show that existing AL methods deteriorate or even fail under the influence of such verification latency. The problem with these methods is that they estimate a label\u2019s utility on the currently available labeled data. However, when this label would arrive, some of the current data may have gotten outdated and new labels have arrived. In this article, we propose to simulate the available data at the time when the label would arrive. Therefore, our method Forgetting and Simulating (FS) forgets outdated information and simulates the delayed labels to get more realistic utility estimates. We assume to know the label\u2019s arrival date a priori and the classifier\u2019s training data to be bounded by a sliding window. Our extensive experiments show that FS improves stream-based AL strategies in settings with both, constant and variable verification latency.<\/jats:p>","DOI":"10.1007\/s10994-021-06099-z","type":"journal-article","created":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T12:04:09Z","timestamp":1637237049000},"page":"2011-2036","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Stream-based active learning for sliding windows under the influence of verification latency"],"prefix":"10.1007","volume":"111","author":[{"given":"Tuan","family":"Pham","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Kottke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Georg","family":"Krempl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bernhard","family":"Sick","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,11,18]]},"reference":[{"key":"6099_CR1","doi-asserted-by":"crossref","unstructured":"Babcock, B., Babu, S., Datar, M., Motwani, R., & Widom, J. (2002). Models and issues in data stream systems. In ACM SIGMOD-SIGACT-SIGART, ACM, New York, NY, USA, PODS 02, pp 1\u201316.","DOI":"10.1145\/543613.543615"},{"key":"6099_CR2","doi-asserted-by":"crossref","unstructured":"Bondu, A., Lemaire, V., & Boulle, M. (2010). Exploration vs. exploitation in active learning: A bayesian approach. In IJCNN, IEEE, pp 1\u20137.","DOI":"10.1109\/IJCNN.2010.5596815"},{"key":"6099_CR3","unstructured":"Chapelle, O. (2005). Active learning for parzen window classifier. In AISTATS, Max\u2013Planck\u2013Gesellschaft, pp. 49\u201356."},{"key":"6099_CR4","doi-asserted-by":"crossref","unstructured":"Chaudhuri, A., Kakde, D., Sadek, C., Gonzalez, L., & Kong, S. (2017). The mean and median criteria for kernel bandwidth selection for support vector data description. In ICDM Workshops, pp. 842\u2013849.","DOI":"10.1109\/ICDMW.2017.116"},{"key":"6099_CR5","first-page":"679","volume-title":"NIPS","author":"DA Cohn","year":"1993","unstructured":"Cohn, D. A. (1993). Neural network exploration using optimal experiment design. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), NIPS (pp. 679\u2013686). Burlington: Morgan Kaufmann."},{"key":"6099_CR6","first-page":"10","volume-title":"Encyclopedia of Machine Learning","author":"D Cohn","year":"2010","unstructured":"Cohn, D. (2010). Active learning. In C. Sammut & G. I. Webb (Eds.), Encyclopedia of Machine Learning (pp. 10\u201314). New York: Springer."},{"key":"6099_CR7","doi-asserted-by":"crossref","unstructured":"dos Reis, D. M., Flach, P., Matwin, S., & Batista, G. (2016). Fast unsupervised online drift detection using incremental Kolmogorov\u2013Smirnov test. In SIGKDD, ACM, New York, NY, USA, KDD 16, pp. 1545\u20131554.","DOI":"10.1145\/2939672.2939836"},{"issue":"1","key":"6099_CR8","first-page":"12","volume":"25","author":"KB Dyer","year":"2014","unstructured":"Dyer, K. B., Capo, R., & Polikar, R. (2014). Compose: A semisupervised learning framework for initially labeled nonstationary streaming data. TNNLS, 25(1), 12\u201326.","journal-title":"TNNLS"},{"key":"6099_CR9","doi-asserted-by":"crossref","unstructured":"Frederickson, C., & Polikar, R. (2018). Resampling techniques for learning under extreme verification latency with class imbalance. In IJCNN, IEEE, pp. 1\u20138.","DOI":"10.1109\/IJCNN.2018.8489622"},{"key":"6099_CR10","doi-asserted-by":"crossref","unstructured":"Gama, J., Sebasti\u00e3o, R., & Rodrigues, P. P. (2009). Issues in evaluation of stream learning algorithms. In SIGKDD, Association for Computing Machinery, pp. 329\u2013338.","DOI":"10.1145\/1557019.1557060"},{"issue":"4","key":"6099_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2523813","volume":"46","author":"J Gama","year":"2014","unstructured":"Gama, J., Zliobaite, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. CSUR, 46(4), 1\u201344.","journal-title":"CSUR"},{"key":"6099_CR12","doi-asserted-by":"crossref","unstructured":"Grzenda, M., Gomes, H. M., & Bifet, A. (2019). Delayed labelling evaluation for data streams. Data Mining and Knowledge Discovery.","DOI":"10.1007\/s10618-019-00654-y"},{"key":"6099_CR13","unstructured":"Hammoodi, M., Stahl, F., & Tennant, M. (2016). Towards online concept drift detection with feature selection for data stream classification. In ECAI, Frontiers in Artificial Intelligence and Applications, vol 285, pp. 1549\u20131550."},{"issue":"1","key":"6099_CR14","first-page":"377","volume":"57","author":"V Hofer","year":"2013","unstructured":"Hofer, V., & Krempl, G. (2013). Drift mining in data: A framework for addressing drift in classification. CSDA, 57(1), 377\u2013391.","journal-title":"CSDA"},{"key":"6099_CR15","doi-asserted-by":"crossref","unstructured":"Ienco, D., Bifet, A., Zliobaite, I., & Pfahringer, B. (2013). Clustering based active learning for evolving data streams. In J. F\u00fcrnkranz, E. H\u00fcllermeier, & T. Higuchi (Eds.), Discovery Science. Lecture Notes in Artificial Intelligence, (Vol. 8140, pp. 79\u201393). Springer.","DOI":"10.1007\/978-3-642-40897-7_6"},{"key":"6099_CR16","unstructured":"Ienco, D., Pfahringer, B., & Zliobait\u0117, I. (2014). High density-focused uncertainty sampling for active learning over evolving stream data. In SIGKDD BigMine, pp. 133\u2013148."},{"key":"6099_CR17","doi-asserted-by":"crossref","unstructured":"Kelly, M. G., Hand, D. J., & Adams, N. M. (1999). The impact of changing populations on classifier performance. In SIGKDD, pp. 367\u2013371.","DOI":"10.1145\/312129.312285"},{"key":"6099_CR18","unstructured":"Klinkenberg, R., & Renz, I. (1998). Adaptive information filtering: Learning in the presence of concept drifts. In Workshop Notes of the AAAI-98\/ICML-98 workshop Learning for Text Categorization, AAAI Press, pp. 33\u201340."},{"key":"6099_CR19","doi-asserted-by":"crossref","unstructured":"Kottke, D., Herde, M., Minh, T. P., Benz, A., Mergard, P., Roghman, A., Sandrock, C., & Sick, B. (2021a). scikit-activeml: A library and toolbox for active learning algorithms. Preprints, 2021030194.","DOI":"10.20944\/preprints202103.0194.v1"},{"key":"6099_CR20","doi-asserted-by":"crossref","unstructured":"Kottke, D., Krempl, G., & Spiliopoulou, M. (2015). Probabilistic active learning in datastreams. In \u00c9. Fromont, T. D. Bie, & M. van Leeuwen (Eds.), IDA. Lecture Notes in Computer Science, (Vol. 9385, pp. 145\u2013157). Springer.","DOI":"10.1007\/978-3-319-24465-5_13"},{"key":"6099_CR21","doi-asserted-by":"publisher","first-page":"1199","DOI":"10.1007\/s10994-021-05986-9","volume":"110","author":"D Kottke","year":"2021","unstructured":"Kottke, D., Herde, M., Sandrock, C., Huseljic, D., Krempl, G., & Sick, B. (2021b). Toward optimal probabilistic active learning using a Bayesian approach. Machine Learning, 110, 1199\u20131231.","journal-title":"Machine Learning"},{"key":"6099_CR22","doi-asserted-by":"crossref","unstructured":"Krempl, G. (2011). The algorithm apt to classify in concurrence of latency and drift. In IDA, Springer, pp. 222\u2013233.","DOI":"10.1007\/978-3-642-24800-9_22"},{"key":"6099_CR23","doi-asserted-by":"crossref","unstructured":"Krempl, G., & Hofer, V. (2011). Classification in presence of drift and latency. In M. Spiliopoulou, H. Wang, D. Cook, J. Pei, W. Wang, O. Za\u00efane, & X. Wu (Eds.), ICDM Workshops. IEEE.","DOI":"10.1109\/ICDMW.2011.47"},{"key":"6099_CR24","doi-asserted-by":"crossref","unstructured":"Krempl, G., Lang, D., & Hofer, V. (2019). Temporal density extrapolation using a dynamic basis approach. Data Mining and Knowledge Discovery,33(5), 1323\u20131356. Special Issue of the ECML\/PKDD 2019 Journal Track.","DOI":"10.1007\/s10618-019-00636-0"},{"key":"6099_CR25","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1007\/978-3-319-24282-8_10","volume-title":"Discovery Science","author":"G Krempl","year":"2015","unstructured":"Krempl, G., Ha, T. C., & Spiliopoulou, M. (2015a). Clustering-based optimised probabilistic active learning (COPAL). In N. Japkowicz & S. Matwin (Eds.), Discovery Science (Vol. 9356, pp. 101\u2013115). New York: Springer."},{"key":"6099_CR26","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1007\/s10994-015-5504-1","volume":"100","author":"G Krempl","year":"2015","unstructured":"Krempl, G., Kottke, D., & Lemaire, V. (2015b). Optimised probabilistic active learning (OPAL) for fast, non-myopic, cost-sensitive active classification. Machine Learning, 100, 2.","journal-title":"Machine Learning"},{"issue":"4","key":"6099_CR27","first-page":"913","volume":"35","author":"P Kumar","year":"2020","unstructured":"Kumar, P., & Gupta, A. (2020). Active learning query strategies for classification, regression, and clustering: A survey. JCST, 35(4), 913\u2013945.","journal-title":"JCST"},{"key":"6099_CR28","unstructured":"Kuncheva, L. I. (2008). Classifier ensembles for detecting concept change in streaming data: Overview and perspectives. In O. Okun & G. Valentini (Eds.), SUEMA. Studies in Computational Intelligence, (Vol. 245, pp. 5\u201310). Springer."},{"key":"6099_CR29","doi-asserted-by":"crossref","unstructured":"Kuncheva, L. I., & S\u00e1nchez, J. S. (2008). Nearest neighbour classifiers for streaming data with delayed labelling. In ICDM, pp. 869\u2013874.","DOI":"10.1109\/ICDM.2008.33"},{"key":"6099_CR30","doi-asserted-by":"crossref","unstructured":"Lewis, D. D., & Gale, W. A. (1994). A sequential algorithm for training text classifiers. In SIGIR, Springer, New York, NY, USA, SIGIR 94, pp. 3\u201312.","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"6099_CR31","doi-asserted-by":"crossref","unstructured":"Loy, C. C., Hospedales, T. M., Xiang, T., & Gong, S. (2012). Stream-based joint exploration-exploitation active learning. In CVPR, IEEE pp. 1560\u20131567.","DOI":"10.1109\/CVPR.2012.6247847"},{"key":"6099_CR32","doi-asserted-by":"crossref","unstructured":"Marrs, G., Hickey, R., & Black, M. (2010). The impact of latency on online classification learning with concept drift. In Y. Bi & M. A. Williams (Eds.), KSEM. Lecture Notes in Computer Science, (Vol. 6291, pp. 459\u2013469). Springer.","DOI":"10.1007\/978-3-642-15280-1_42"},{"key":"6099_CR33","volume-title":"Machine Learning: A Probabilistic Perspective","author":"KP Murphy","year":"2012","unstructured":"Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. Cambridge: MIT Press."},{"key":"6099_CR34","doi-asserted-by":"crossref","unstructured":"Parreira, P., & Prati, R. (2019). Aprendizagem ativa em fluxo de dados com lat\u00eancia intermedi\u00e1ria. In ENIAC, SBC, pp. 365\u2013376","DOI":"10.5753\/eniac.2019.9298"},{"key":"6099_CR35","unstructured":"Pham, M. T., Kottke, D., Tsarenko, A., Gruhl, C., & Sick, B. (2020). Improving self-adaptation for multi-sensor activity recognition with active learning. In IJCNN."},{"key":"6099_CR36","doi-asserted-by":"crossref","unstructured":"Plasse, J., & Adams, N. (2016). Handling delayed labels in temporally evolving data streams. In IEEE BigData, pp. 2416\u20132424.","DOI":"10.1109\/BigData.2016.7840877"},{"issue":"1","key":"6099_CR37","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1109\/TED.2018.2878770","volume":"66","author":"R Razavi-Far","year":"2019","unstructured":"Razavi-Far, R., Hallaji, E., Saif, M., & Ditzler, G. (2019). A novelty detector and extreme verification latency model for nonstationary environments. IEEE TIE, 66(1), 561\u2013570.","journal-title":"IEEE TIE"},{"key":"6099_CR38","unstructured":"Roy, N., & McCallum, A. (2001). Toward optimal active learning through sampling estimation of error reduction. ICML (pp. 441\u2013448). San Francisco, CA, USA: Morgan Kaufmann."},{"key":"6099_CR39","unstructured":"Schlimmer, J. C., & Granger, R. H. (1986). Beyond incremental processing: Tracking concept drift. In AAAI, pp. 502\u2013507."},{"key":"6099_CR40","doi-asserted-by":"crossref","unstructured":"Settles, B. (2012). Active Learning. No. 18 in Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan and Claypool Publishers.","DOI":"10.2200\/S00429ED1V01Y201207AIM018"},{"key":"6099_CR41","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1007\/978-3-319-00969-8_28","volume":"226","author":"A Shaker","year":"2013","unstructured":"Shaker, A., & H\u00fcllermeier, E. (2013). Recovery analysis for adaptive learning from non-stationary data streams. Advances in Intelligent Systems and Computing, 226, 289\u2013298.","journal-title":"Advances in Intelligent Systems and Computing"},{"key":"6099_CR42","doi-asserted-by":"crossref","unstructured":"Souza, V., Pinho, T., & Batista, G. (2018). Evaluating stream classifiers with delayed labels information. In BRACIS, pp. 408\u2013413.","DOI":"10.1109\/BRACIS.2018.00077"},{"key":"6099_CR43","doi-asserted-by":"crossref","unstructured":"Umer, M. (2017). Learning extreme verification latency quickly with importance weighting: Fast compose and level_iw. PhD thesis, Rowan University.","DOI":"10.1109\/SSCI.2016.7849962"},{"key":"6099_CR44","unstructured":"Umer, M., & Polikar, R. (2020). Comparative analysis of extreme verification latency learning algorithms. arXiv:2011.14917."},{"key":"6099_CR45","doi-asserted-by":"crossref","unstructured":"Zhu, X., Zhang, P., Lin, X., & Shi, Y. (2007). Active learning from data streams. In ICDM, IEEE Computer Society, Washington, DC, USA, ICDM 07, pp. 757\u2013762.","DOI":"10.1109\/ICDM.2007.101"},{"key":"6099_CR46","doi-asserted-by":"crossref","unstructured":"\u017dliobait\u00e9, I. (2010). Change with delayed labeling: When is it detectable? In ICDM Workshops, pp. 843\u2013850.","DOI":"10.1109\/ICDMW.2010.49"},{"key":"6099_CR47","first-page":"27","volume":"25","author":"I Zliobaite","year":"2014","unstructured":"Zliobaite, I., Bifet, A., Pfahringer, B., & Holmes, G. (2014). Active learning with drifting streaming data. TNNLS, 25, 27\u201339.","journal-title":"TNNLS"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06099-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-06099-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06099-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,12]],"date-time":"2024-09-12T11:47:54Z","timestamp":1726141674000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-06099-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,18]]},"references-count":47,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["6099"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-06099-z","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,18]]},"assertion":[{"value":"31 January 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 July 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 October 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 November 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}