{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,1]],"date-time":"2025-11-01T21:39:31Z","timestamp":1762033171819,"version":"build-2065373602"},"reference-count":37,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2013,12,13]],"date-time":"2013-12-13T00:00:00Z","timestamp":1386892800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Ubiquitous automated data collection at an unprecedented scale is making available streaming, real-time information flows in a wide variety of settings, transforming both science and industry. Learning algorithms deployed in such contexts often rely on single-pass inference, where the data history is never revisited. Learning may also need to be temporally adaptive to remain up-to-date against unforeseen changes in the data generating mechanism. Online Bayesian inference remains challenged by such transient, evolving data streams. Nonparametric modeling techniques can prove particularly ill-suited, as the complexity of the model is allowed to increase with the sample size. In this work, we take steps to overcome these challenges by porting information theoretic heuristics, such as exponential forgetting and active learning, into a fully Bayesian framework. We showcase our methods by augmenting a modern non-parametric modeling framework, dynamic trees, and illustrate its performance on a number of practical examples. The end product is a powerful streaming regression and classification tool, whose performance compares favorably to the state-of-the-art.<\/jats:p>","DOI":"10.3390\/e15125510","type":"journal-article","created":{"date-parts":[[2013,12,16]],"date-time":"2013-12-16T06:18:40Z","timestamp":1387174720000},"page":"5510-5535","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Information-Theoretic Data Discarding for Dynamic Trees on Data Streams"],"prefix":"10.3390","volume":"15","author":[{"given":"Christoforos","family":"Anagnostopoulos","sequence":"first","affiliation":[{"name":"Department of Mathematics, Imperial College London, South Kensington Campus, London SW7 2AZ, UK"}]},{"given":"Robert","family":"Gramacy","sequence":"additional","affiliation":[{"name":"Booth School of Business, The University of Chicago, 5807 South Woodlawn Avenue, Chicago, IL 60637, USA"}]}],"member":"1968","published-online":{"date-parts":[[2013,12,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1198\/jasa.2011.ap09769","article-title":"Dynamic trees for learning and design","volume":"106","author":"Taddy","year":"2011","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/BF00116900","article-title":"Learning in the presence of concept drift and hidden contexts","volume":"23","author":"Widmer","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_3","unstructured":"Hulten, G., Spencer, L., and Domings, P. (2000, January 20\u201323). Mining high-speed data streams. Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD \u201900, Boston, MA, USA."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Hulten, G., Spencer, L., and Domingos, P. (2001, January 26\u201329). Mining Time-Changing Data Streams. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD \u201901, San Francisco, CA, USA.","DOI":"10.1145\/502512.502529"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1959","DOI":"10.1162\/0899766041336396","article-title":"Online adaptive decision trees","volume":"16","author":"Basak","year":"2004","journal-title":"Neural Comput."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Street, W.N., and Kim, Y. (2001, January 26\u201329). A Streaming Ensemble Algorithm ({SEA}) for Large-Scale Classification. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD\u201901, San Francisco, CA, USA.","DOI":"10.1145\/502512.502568"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-3-540-25966-4_1","article-title":"Classifier Ensembles for Changing Environments","volume":"Volume 3077","author":"Roli","year":"2004","journal-title":"Proceedings of the 5th International Workshop on Multiple Classifier Systems"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wang, H., Fan, W., Yu, P.S., and Han, J. (2003, January 24\u201327). Mining Concept-Drifting Data Streams Using Ensemble Classifiers. Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD\u201903, Washington DC, WA, USA.","DOI":"10.1145\/956755.956778"},{"key":"ref_9","unstructured":"Kolter, J., and Maloof, M. (2003, January 19\u201322). Dynamic Weighted Majority: A New Ensemble Method for Tracking Concept Drift. Proceedings of the Third IEEE International Conference on Data Mining, Melbourne, FL, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"490","DOI":"10.1007\/978-3-540-72523-7_49","article-title":"An ensemble approach for incremental learning in nonstationary environments","volume":"4472","author":"Muhlbaier","year":"2007","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s10618-009-0130-9","article-title":"Flexible decision tree for data stream classification in the presence of concept change, noise and missing values","volume":"19","author":"Hashemi","year":"2009","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1002\/sam.10151","article-title":"Online linear and quadratic discriminant analysis with adaptive forgetting for streaming classification","volume":"5","author":"Anagnostopoulos","year":"2012","journal-title":"Stat. Anal. Data Min."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1007\/978-3-540-89689-0_55","article-title":"Adaptive learning rate for online linear discriminant classifiers","volume":"5342","author":"Kuncheva","year":"2008","journal-title":"Structural, Syntactic, and Statistical Pattern Recognition, Lecture Notes in Computer Science"},{"key":"ref_14","unstructured":"Haykin, S. (1996). Adaptive Filter Theory, Prentice-Hall, Inc."},{"key":"ref_15","first-page":"146","article-title":"Stochastic Learning","volume":"2600","author":"Bousquet","year":"2004","journal-title":"Advanced Lectures on Machine Learning"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Saad, D. (1998). On-Line Learning in Neural Networks, Cambridge University Press.","DOI":"10.1017\/CBO9780511569920"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"935","DOI":"10.1080\/01621459.1998.10473750","article-title":"Bayesian CART model search (with discussion)","volume":"93","author":"Chipman","year":"1998","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1023\/A:1013916107446","article-title":"Bayesian treed models","volume":"48","author":"Chipman","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Gramacy, R.B., and Taddy, M.A. (dynaTree: Dynamic Trees for Learning and Design, 2011). dynaTree: Dynamic Trees for Learning and Design, R Package Version 2.0.","DOI":"10.32614\/CRAN.package.dynaTree"},{"key":"ref_20","unstructured":"R Development Core Team (2010). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1214\/10-STS325","article-title":"Particle learning and smoothing","volume":"25","author":"Carvalho","year":"2010","journal-title":"Stat. Sci."},{"key":"ref_22","unstructured":"Lakshminarayanan, B., Roy, D.M., and Teh, Y.W. (2013, January 16\u201321). Top-Down Particle Filtering for Bayesian Decision Trees. Proceedings of the The 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA."},{"key":"ref_23","unstructured":"O\u2019Hagan, A., and Forster, J. (2004). Kendall\u2019s Advanced Theory of Statistics, Volume 2B, Bayesian Inference, Arnold Publishers."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1214\/aoms\/1177729893","article-title":"Adjustment of an inverse matrix corresponding to a change in one element of a given matrix","volume":"21","author":"Sherman","year":"1950","journal-title":"Ann. Math. Stat."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1162\/neco.1992.4.4.590","article-title":"Information\u2013based objective functions for active data selection","volume":"4","author":"MacKay","year":"1992","journal-title":"Neural Comput."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1071","DOI":"10.1016\/0893-6080(95)00137-9","article-title":"Neural Network Exploration using Optimal Experimental Design","volume":"6","author":"Cohn","year":"1996","journal-title":"Neural Networks"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Seo, S., Wallat, M., Graepel, T., and Obermayer, K. (2000, January 24\u201327). Gaussian Process Regression: Active Data Selection and Test Point Rejection. Proceedings of the IEEE International Joint Conference on Neural Networks, Como, Italy. Volume III.","DOI":"10.1007\/978-3-642-59802-9_4"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Gelman, A., Carlin, J.B., Stern, H.S., and Rubin, D.B. (2003). Bayesian Data Analysis, CRC Press.","DOI":"10.1201\/9780429258480"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Joshi, A., Porikli, F., and Papanikolopoulos, N. (2009, January 20\u201325). Multi-Class Active Learning for Image Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR \u201909, Miami, FL, USA.","DOI":"10.1109\/CVPRW.2009.5206627"},{"key":"ref_30","first-page":"1","article-title":"Multivariate adaptive regression splines","volume":"19","author":"Friedman","year":"1991","journal-title":"Ann. Stat."},{"key":"ref_31","unstructured":"Asuncion, A., and Newman, D. UCI Machine Learning Repository. Available online: http:\/\/www.ics.uci.edu\/mlearn\/MLRepository.html\/."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Saffari, A., Leistner, C., Santner, J., Godec, M., and Bischof, H. (2009, January 27September\u20134). On-Line Random Forests. Proceedings of the 3rd IEEE ICCV Workshop on on-Line Computer Vision, Kyoto, Japan.","DOI":"10.1109\/ICCVW.2009.5457447"},{"key":"ref_33","unstructured":"Harries, M. (1999). Splice-2 Comparative Evaluation: Electricity Pricing, School of Computer Science and Engineering, University of New South Wales. Technical Report TR-9905."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1198\/016214503388619229","article-title":"On optimality properties of the power prior","volume":"98","author":"Ibrahim","year":"2003","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1080\/00207179308923034","article-title":"On a general concept of forgetting","volume":"58","author":"Kulhavy","year":"1993","journal-title":"Int. J. Control"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1007\/s10994-009-5119-5","article-title":"Measuring classifier performance: A coherent alternative to the area under the ROC curve","volume":"77","author":"Hand","year":"2009","journal-title":"Mach. Learn."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/12\/5510\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:51:22Z","timestamp":1760219482000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/12\/5510"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,12,13]]},"references-count":37,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2013,12]]}},"alternative-id":["e15125510"],"URL":"https:\/\/doi.org\/10.3390\/e15125510","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2013,12,13]]}}}