{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T06:14:07Z","timestamp":1774678447996,"version":"3.50.1"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2019,11,16]],"date-time":"2019-11-16T00:00:00Z","timestamp":1573862400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,11,16]],"date-time":"2019-11-16T00:00:00Z","timestamp":1573862400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004421","name":"Warsaw University of Technology","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004421","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2020,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>A large portion of the stream mining studies on classification rely on the availability of true labels immediately after making predictions. This approach is well exemplified by the test-then-train evaluation, where predictions immediately precede true label arrival. However, in many real scenarios, labels arrive with non-negligible latency. This raises the question of how to evaluate classifiers trained in such circumstances. This question is of particular importance when stream mining models are expected to refine their predictions between acquiring instance data and receiving its true label. In this work, we propose a novel evaluation methodology for data streams when verification latency takes place, namely continuous re-evaluation. It is applied to reference data streams and it is used to differentiate between stream mining techniques in terms of their ability to refine predictions based on newly arriving instances. Our study points out, discusses and shows empirically the importance of considering the delay of instance labels when evaluating classifiers for data streams.\n<\/jats:p>","DOI":"10.1007\/s10618-019-00654-y","type":"journal-article","created":{"date-parts":[[2019,11,16]],"date-time":"2019-11-16T06:02:21Z","timestamp":1573884141000},"page":"1237-1266","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Delayed labelling evaluation for data streams"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5440-4954","authenticated-orcid":false,"given":"Maciej","family":"Grzenda","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5276-637X","authenticated-orcid":false,"given":"Heitor Murilo","family":"Gomes","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8339-7773","authenticated-orcid":false,"given":"Albert","family":"Bifet","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,11,16]]},"reference":[{"issue":"1","key":"654_CR1","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/TKDE.2010.36","volume":"23","author":"H Abdulsalam","year":"2010","unstructured":"Abdulsalam H, Skillicorn DB, Martin P (2010) Classification using streaming random forests. IEEE Trans Knowl Data Eng 23(1):22\u201336","journal-title":"IEEE Trans Knowl Data Eng"},{"issue":"6","key":"654_CR2","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1109\/69.250074","volume":"5","author":"R Agrawal","year":"1993","unstructured":"Agrawal R, Imilielinski T, Swani A (1993) Database mining: a performance perspective. IEEE Trans Knowl Data Eng 5(6):914\u2013925","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"654_CR3","doi-asserted-by":"crossref","unstructured":"Almeida E, Ferreira C, Gama J (2013) Adaptive model rules from data streams. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 480\u2013492","DOI":"10.1007\/978-3-642-40988-2_31"},{"key":"654_CR4","doi-asserted-by":"crossref","unstructured":"Bifet A, Gavald\u00e0 R (2007) Learning from time-changing data with adaptive windowing. In: Proceedings of the 7th SIAM international conference on data mining, April 26\u201328, 2007, Minneapolis, Minnesota, USA. Society for Industrial and Applied Mathematics SIAM, pp 443\u2013448","DOI":"10.1137\/1.9781611972771.42"},{"key":"654_CR5","doi-asserted-by":"publisher","unstructured":"Bifet A, Gavald\u00e0 R (2009) Adaptive learning from evolving data streams. In: International symposium on intelligent data analysis. Springer, pp 249\u2013260. https:\/\/doi.org\/10.1007\/978-3-642-03915-7_22","DOI":"10.1007\/978-3-642-03915-7_22"},{"key":"654_CR6","unstructured":"Bifet A, Holmes G, Kirkby R, Pfahringer B (2011a) MOA data stream mining\u2014a practical approach. Centre for Open Software Innovation COSI"},{"key":"654_CR7","doi-asserted-by":"crossref","unstructured":"Bifet A, Holmes G, Pfahringer B, Read J, Kranen P, Kremer H, Jansen T, Seidl T (2011b) MOA: a real-time analytics open source framework. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 617\u2013620","DOI":"10.1007\/978-3-642-23808-6_41"},{"key":"654_CR8","doi-asserted-by":"crossref","unstructured":"Bifet A, Read J, \u017dliobait\u0117 I, Pfahringer B, Holmes G (2013) Pitfalls in benchmarking data stream classification and how to avoid them. Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, pp 465\u2013479","DOI":"10.1007\/978-3-642-40988-2_30"},{"key":"654_CR9","doi-asserted-by":"crossref","unstructured":"Bifet A, de\u00a0Francisci\u00a0Morales G, Read J, Holmes G, Pfahringer B (2015) Efficient online evaluation of big data stream classifiers. In: 21st ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 59\u201368","DOI":"10.1145\/2783258.2783372"},{"issue":"3","key":"654_CR10","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1016\/S0168-1699(99)00046-0","volume":"24","author":"JA Blackard","year":"1999","unstructured":"Blackard JA, Dean DJ (1999) Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Comput Electron Agric 24(3):131\u2013151","journal-title":"Comput Electron Agric"},{"issue":"1","key":"654_CR11","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L (2001) Random forests. Mach Learn 45(1):5\u201332","journal-title":"Mach Learn"},{"key":"654_CR12","doi-asserted-by":"publisher","DOI":"10.1201\/9781315139470","volume-title":"Classification and regression trees","author":"L Breiman","year":"2017","unstructured":"Breiman L (2017) Classification and regression trees. Routledge, New York"},{"issue":"4","key":"654_CR13","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1109\/MCI.2015.2471196","volume":"10","author":"G Ditzler","year":"2015","unstructured":"Ditzler G, Roveri M, Alippi C, Polikar R (2015) Learning in nonstationary environments: a survey. IEEE Comput Intell Mag 10(4):12\u201325","journal-title":"IEEE Comput Intell Mag"},{"key":"654_CR14","doi-asserted-by":"crossref","unstructured":"Domingos P, Hulten G (2000) Mining high-speed data streams. In: 6th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 71\u201380","DOI":"10.1145\/347090.347107"},{"key":"654_CR15","doi-asserted-by":"crossref","unstructured":"Fanaee-T H, Gama J (2013) Event labeling combining ensemble detectors and background knowledge. Progr Artif Intell. http:\/\/dx.doi.org\/10.1007\/s13748-013-0040-3","DOI":"10.1007\/s13748-013-0040-3"},{"key":"654_CR16","doi-asserted-by":"crossref","unstructured":"Gama J, Rodrigues P (2009) Issues in evaluation of stream learning algorithms. In: 15th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 329\u2013338","DOI":"10.1145\/1557019.1557060"},{"issue":"9","key":"654_CR17","doi-asserted-by":"publisher","first-page":"1469","DOI":"10.1007\/s10994-017-5642-8","volume":"106","author":"HM Gomes","year":"2017","unstructured":"Gomes HM, Bifet A, Read J, Barddal JP, Enembreck F, Pfharinger B, Holmes G, Abdessalem T (2017) Adaptive random forests for evolving data stream classification. Mach Learn 106(9):1469\u20131495. https:\/\/doi.org\/10.1007\/s10994-017-5642-8","journal-title":"Mach Learn"},{"key":"654_CR18","unstructured":"Gomes HM, Barddal JP, Boiko LE, Bifet A (2018) Adaptive random forests for data stream regression. In: Proceedings of the 26th European symposium on artificial neural networks (ESANN). pp 267\u2013272"},{"key":"654_CR19","doi-asserted-by":"crossref","unstructured":"Hulten G, Spencer L, Domingos P (2001) Mining time-changing data streams. In: 7th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 97\u2013106","DOI":"10.1145\/502512.502529"},{"issue":"1","key":"654_CR20","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1007\/s10618-010-0201-y","volume":"23","author":"E Ikonomovska","year":"2011","unstructured":"Ikonomovska E, Gama J, D\u017eeroski S (2011a) Learning model trees from evolving data streams. Data Min Knowl Discov 23(1):128\u2013168","journal-title":"Data Min Knowl Discov"},{"key":"654_CR21","unstructured":"Ikonomovska E, Gama J, Zenko B, Dzeroski S (2011b) Speeding-up Hoeffding-based regression trees with options. In: International conference on machine learning. Omnipress, pp 537\u2013544"},{"key":"654_CR22","doi-asserted-by":"crossref","unstructured":"Kuncheva LI, S\u00e1nchez JS (2008) Nearest neighbour classifiers for streaming data with delayed labelling. In: IEEE international conference on data mining. IEEE, pp 869\u2013874","DOI":"10.1109\/ICDM.2008.33"},{"key":"654_CR23","unstructured":"Kuo Ss, Lee Cm, Ko Cn (2014) Hybrid learning algorithm based neural networks for short-term load forecasting. In: International conference on fuzzy theory and its applications. IEEE, pp 105\u2013110"},{"key":"654_CR24","volume-title":"Big data: principles and best practices of scalable realtime data systems","author":"N Marz","year":"2015","unstructured":"Marz N, Warren J (2015) Big data: principles and best practices of scalable realtime data systems. Manning Publications Co., Greenwich"},{"issue":"6","key":"654_CR25","first-page":"859","volume":"23","author":"M Masud","year":"2011","unstructured":"Masud M, Gao J, Khan L, Han J, Thuraisingham BM (2011) Classification and novel class detection in concept-drifting data streams under time constraints. IEEE TKDE 23(6):859\u2013874","journal-title":"IEEE TKDE"},{"key":"654_CR26","doi-asserted-by":"crossref","unstructured":"Plasse J, Adams N (2016) Handling delayed labels in temporally evolving data streams. In: 2016 IEEE International Conference on Big Data (Big Data). IEEE, pp 2416\u20132424","DOI":"10.1109\/BigData.2016.7840877"},{"key":"654_CR27","doi-asserted-by":"crossref","unstructured":"Souza VMA, Silva DF, Batista GEAPA, Gama J (2015) Classification of evolving data streams with infinitely delayed labels. In: IEEE International Conference on Machine Learning and Applications. IEEE, pp 214\u2013219","DOI":"10.1109\/ICMLA.2015.174"},{"issue":"4","key":"654_CR28","doi-asserted-by":"publisher","first-page":"1897","DOI":"10.1109\/59.476055","volume":"10","author":"D Srinivasan","year":"1995","unstructured":"Srinivasan D, Chang CS, Liew AC (1995) Demand forecasting using fuzzy neural computation, with special emphasis on weekend and public holiday forecasting. IEEE Trans Power Syst 10(4):1897\u20131903","journal-title":"IEEE Trans Power Syst"},{"key":"654_CR29","unstructured":"\u017dliobait\u0117 I (2010) Change with delayed labeling: When is it detectable? In: IEEE International conference on data mining workshops. IEEE, pp 843\u2013850"},{"issue":"3","key":"654_CR30","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1007\/s10994-014-5441-4","volume":"98","author":"I \u017dliobait\u0117","year":"2015","unstructured":"\u017dliobait\u0117 I, Bifet A, Read J, Pfahringer B, Holmes G (2015) Evaluation methods and decision theory for classification of streaming data with temporal dependence. Mach Learn 98(3):455\u2013482","journal-title":"Mach Learn"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-019-00654-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10618-019-00654-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-019-00654-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,15]],"date-time":"2020-11-15T00:14:15Z","timestamp":1605399255000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10618-019-00654-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,16]]},"references-count":30,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,9]]}},"alternative-id":["654"],"URL":"https:\/\/doi.org\/10.1007\/s10618-019-00654-y","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,16]]},"assertion":[{"value":"20 November 2018","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 September 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 November 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with Ethical Standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}