{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T16:30:44Z","timestamp":1774542644878,"version":"3.50.1"},"reference-count":67,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2024,6,28]],"date-time":"2024-06-28T00:00:00Z","timestamp":1719532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Commission under Horizon Europe Programme","award":["101080756"],"award-info":[{"award-number":["101080756"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In this work, we present a novel methodology for performing the supervised classification of time-ordered noisy data; we call this methodology Entropic Sparse Probabilistic Approximation with Markov regularization (eSPA-Markov). It is an extension of entropic learning methodologies, allowing the simultaneous learning of segmentation patterns, entropy-optimal feature space discretizations, and Bayesian classification rules. We prove the conditions for the existence and uniqueness of the learning problem solution and propose a one-shot numerical learning algorithm that\u2014in the leading order\u2014scales linearly in dimension. We show how this technique can be used for the computationally scalable identification of persistent (metastable) regime affiliations and regime switches from high-dimensional non-stationary and noisy time series, i.e., when the size of the data statistics is small compared to their dimensionality and when the noise variance is larger than the variance in the signal. We demonstrate its performance on a set of toy learning problems, comparing eSPA-Markov to state-of-the-art techniques, including deep learning and random forests. We show how this technique can be used for the analysis of noisy time series from DNA and RNA Nanopore sequencing.<\/jats:p>","DOI":"10.3390\/e26070553","type":"journal-article","created":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T10:14:46Z","timestamp":1719828886000},"page":"553","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["On Entropic Learning from Noisy Time Series in the Small Data Regime"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1350-4390","authenticated-orcid":false,"given":"Davide","family":"Bassetti","sequence":"first","affiliation":[{"name":"Faculty of Mathematics, RPTU Kaiserslautern-Landau, Gottlieb-Daimler-Str. 48, 67663 Kaiserslautern, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9801-0538","authenticated-orcid":false,"given":"Luk\u00e1\u0161","family":"Posp\u00ed\u0161il","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Faculty of Civil Engineering, V\u0160B-TUO, Ludvika Podeste 1875\/17, 708 33 Ostrava, Czech Republic"}]},{"given":"Illia","family":"Horenko","sequence":"additional","affiliation":[{"name":"Faculty of Mathematics, RPTU Kaiserslautern-Landau, Gottlieb-Daimler-Str. 48, 67663 Kaiserslautern, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1214\/aos\/1176342709","article-title":"Consistent autoregressive spectral estimates","volume":"2","author":"Berk","year":"1974","journal-title":"Ann. Stat."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1016\/0047-259X(85)90027-2","article-title":"Prediction of multivariate time series by autoregressive model fitting","volume":"16","author":"Lewis","year":"1985","journal-title":"J. Multivar. Anal."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1093\/biomet\/46.3-4.306","article-title":"Efficient estimation of parameters in moving-average models","volume":"46","author":"Durbin","year":"1959","journal-title":"Biometrika"},{"key":"ref_4","unstructured":"Kedem, B., and Fokianos, K. (2005). Regression Models for Time Series Analysis, John Wiley & Sons."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MASSP.1986.1165342","article-title":"An introduction to hidden Markov models","volume":"3","author":"Rabiner","year":"1986","journal-title":"IEEE ASSP Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/0304-4076(90)90093-9","article-title":"Analysis of time series subject to changes in regime","volume":"45","author":"Hamilton","year":"1990","journal-title":"J. Econom."},{"key":"ref_7","unstructured":"Fr\u00fchwirth-Schnatter, S. (2006). Finite Mixture and Markov Switching Models, Springer."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1002\/j.1538-7305.1983.tb03114.x","article-title":"An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition","volume":"62","author":"Levinson","year":"1983","journal-title":"Bell Syst. Tech. J."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1109\/PROC.1973.9030","article-title":"The viterbi algorithm","volume":"61","author":"Forney","year":"1973","journal-title":"Proc. IEEE"},{"key":"ref_10","unstructured":"Lipton, Z.C., Berkowitz, J., and Elkan, C. (2015). A critical review of recurrent neural networks for sequence learning. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"162","DOI":"10.21629\/JSEE.2017.01.18","article-title":"Convolutional neural networks for time series classification","volume":"28","author":"Zhao","year":"2017","journal-title":"J. Syst. Eng. Electron."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_13","unstructured":"Kim, Y., Denton, C., Hoang, L., and Rush, A.M. (2017). Structured attention networks. arXiv."},{"key":"ref_14","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., and Zhu, J. (2019, January 9\u201314). Explainable AI: A brief survey on history, research areas, approaches and challenges. Proceedings of the Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China.","DOI":"10.1007\/978-3-030-32236-6_51"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3491209","article-title":"Trustworthy artificial intelligence: A review","volume":"55","author":"Kaur","year":"2022","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M., Preece, A., Julier, S., and Rao, R.M. (2017, January 4\u20138). Interpretability of deep learning models: A survey of results. Proceedings of the 2017 IEEE Smartworld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (Smartworld\/SCALCOM\/UIC\/ATC\/CBDcom\/IOP\/SCI), San Francisco, CA, USA.","DOI":"10.1109\/UIC-ATC.2017.8397411"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3197","DOI":"10.1007\/s10115-022-01756-8","article-title":"Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond","volume":"64","author":"Li","year":"2022","journal-title":"Knowl. Inf. Syst."},{"key":"ref_20","unstructured":"Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2021). On the opportunities and risks of foundation models. arXiv."},{"key":"ref_21","first-page":"6441","article-title":"Benchmarking deep learning interpretability in time series predictions","volume":"33","author":"Ismail","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1563","DOI":"10.1162\/neco_a_01296","article-title":"On a scalable entropic breaching of the overfitting barrier for small data problems in machine learning","volume":"32","author":"Horenko","year":"2020","journal-title":"Neural Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"eaaw0961","DOI":"10.1126\/sciadv.aaw0961","article-title":"Low-cost scalable discretization, prediction, and feature selection for complex systems","volume":"6","author":"Gerber","year":"2020","journal-title":"Sci. Adv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1220","DOI":"10.1162\/neco_a_01490","article-title":"eSPA+: Scalable entropy-optimal machine learning classification for small data problems","volume":"34","author":"Vecchi","year":"2022","journal-title":"Neural Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"102171","DOI":"10.1016\/j.isci.2021.102171","article-title":"A deeper look into natural sciences with physics-based and data-driven measures","volume":"24","author":"Rodrigues","year":"2021","journal-title":"Iscience"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e2119659119","DOI":"10.1073\/pnas.2119659119","article-title":"Cheap robust learning of data anomalies with analytically solvable entropic outlier sparsification","volume":"119","author":"Horenko","year":"2022","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e2214972120","DOI":"10.1073\/pnas.2214972120","article-title":"On cheap entropy-sparsified regression learning","volume":"120","author":"Horenko","year":"2023","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"101958","DOI":"10.1016\/j.ribaf.2023.101958","article-title":"Entropic approximate learning for financial decision-making in the small data regime","volume":"65","author":"Vecchi","year":"2023","journal-title":"Res. Int. Bus. Financ."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Horenko, I., Posp\u00ed\u0161il, L., Vecchi, E., Albrecht, S., Gerber, A., Rehbock, B., Stroh, A., and Gerber, S. (2022). Low-cost probabilistic 3D denoising with applications for ultra-low-radiation computed tomography. J. Imaging, 8.","DOI":"10.3390\/jimaging8060156"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1198","DOI":"10.1162\/neco_a_01664","article-title":"Gauge-Optimal Approximate Learning for Small Data Classification","volume":"36","author":"Vecchi","year":"2024","journal-title":"Neural Comput."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1348\/000711005X48266","article-title":"K-means clustering: A half-century synthesis","volume":"59","author":"Steinley","year":"2006","journal-title":"Br. J. Math. Stat. Psychol."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ahmed, M., Seraj, R., and Islam, S.M.S. (2020). The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9.","DOI":"10.3390\/electronics9081295"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1137\/080715962","article-title":"Finite element approach to clustering of multidimensional time series","volume":"32","author":"Horenko","year":"2010","journal-title":"SIAM J. Sci. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"107","DOI":"10.2140\/camcos.2018.13.107","article-title":"On a scalable nonparametric denoising of time series signals","volume":"13","author":"Gagliardini","year":"2018","journal-title":"Commun. Appl. Math. Comput. Sci."},{"key":"ref_35","unstructured":"Tikhonov, A.N., and Arsenin, V. (1977). Solutions of Ill-Posed Problems, Springer Science & Business Media."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Nocedal, J., and Wright, S.J. (1999). Numerical Optimization, Springer.","DOI":"10.1007\/b98874"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1146\/annurev-statistics-031017-100325","article-title":"Finite mixture models","volume":"6","author":"McLachlan","year":"2019","journal-title":"Annu. Rev. Stat. Its Appl."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1198\/016214502760047131","article-title":"Model-based clustering, discriminant analysis, and density estimation","volume":"97","author":"Fraley","year":"2002","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Lindsay, B.G. (1995). Mixture Models: Theory, Geometry, and Applications, IMS.","DOI":"10.1214\/cbms\/1462106013"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1093\/nsr\/nwt032","article-title":"Challenges of big data analysis","volume":"1","author":"Fan","year":"2014","journal-title":"Natl. Sci. Rev."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Hastie, T., Tibshirani, R., Friedman, J.H., and Friedman, J.H. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer.","DOI":"10.1007\/978-0-387-84858-7"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Reel, P.S., Reel, S., Pearson, E., Trucco, E., and Jefferson, E. (2021). Using machine learning approaches for multi-omics data analysis: A review. Biotechnol. Adv., 49.","DOI":"10.1016\/j.biotechadv.2021.107739"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Kang, M., Ko, E., and Mersha, T.B. (2022). A roadmap for multi-omics data integration using deep learning. Briefings Bioinform., 23.","DOI":"10.1093\/bib\/bbab454"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1037\/h0071325","article-title":"Analysis of a complex of statistical variables into principal components","volume":"24","author":"Hotelling","year":"1933","journal-title":"J. Educ. Psychol."},{"key":"ref_45","first-page":"13","article-title":"Dimensionality reduction: A comparative review","volume":"10","author":"Postma","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1038\/nbt.4314","article-title":"Dimensionality reduction for visualizing single-cell data using UMAP","volume":"37","author":"Becht","year":"2019","journal-title":"Nat. Biotechnol."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1186\/s41044-016-0014-0","article-title":"Big data preprocessing: Methods and prospects","volume":"1","author":"Luengo","year":"2016","journal-title":"Big Data Anal."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Xanthopoulos, P., Pardalos, P.M., Trafalis, T.B., Xanthopoulos, P., Pardalos, P.M., and Trafalis, T.B. (2013). Linear discriminant analysis. Robust Data Mining, Springer Briefs in Optimization; Springer.","DOI":"10.1007\/978-1-4419-9878-1"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1007\/s13042-013-0226-9","article-title":"Linear discriminant analysis for the small sample size problem: An overview","volume":"6","author":"Sharma","year":"2015","journal-title":"Int. J. Mach. Learn. Cybern."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"O\u2019Kane, T., Frederiksen, J.S., Frederiksen, C.S., and Horenko, I. (2024). Beyond the First Tipping Points of Southern Hemisphere Climate. Climate, 12.","DOI":"10.3390\/cli12060081"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Groom, M., Bassetti, D., Horenko, I., and O\u2019Kane, T.J. (2024). On the comparative utility of entropic learning versus deep learning for long-range ENSO prediction. Authorea, preprints.","DOI":"10.22541\/essoar.170688824.46505260\/v1"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"267","DOI":"10.2140\/camcos.2021.16.267","article-title":"Scalable computational measures for entropic detection of latent relations and their applications to magnetic imaging","volume":"16","author":"Horenko","year":"2021","journal-title":"Commun. Appl. Math. Comput. Sci."},{"key":"ref_53","unstructured":"Barisin, T., and Horenko, I. (2024). Towards Generalized Entropic Sparsification for Convolutional Neural Networks. arXiv."},{"key":"ref_54","unstructured":"Horenko, I. (2023). On existence, uniqueness and scalability of adversarial robustness measures for AI classifiers. arXiv."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Abida, K., Karray, F., and Sun, J. (2009, January 6\u20138). Comparison of GMM and fuzzy-GMM applied to phoneme classification. Proceedings of the 2009 3rd International Conference on Signals, Circuits and Systems (SCS), Medenine, Tunisia.","DOI":"10.1109\/ICSCS.2009.5412479"},{"key":"ref_56","unstructured":"Dost\u00e1l, Z. (2009). Optimal Quadratic Programming Algorithms, with Applications to Variational Inequalities, Springer."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1196","DOI":"10.1137\/S1052623497330963","article-title":"Nonmonotone spectral projected gradient methods on convex sets","volume":"10","author":"Birgin","year":"2000","journal-title":"SIAM J. Optim."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1016\/j.gpb.2016.05.004","article-title":"Oxford Nanopore MinION sequencing and genome assembly","volume":"14","author":"Lu","year":"2016","journal-title":"Genom. Proteom. Bioinform."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"1869","DOI":"10.1038\/s41467-019-09637-5","article-title":"Sequencing of human genomes with nanopore technology","volume":"10","author":"Bowden","year":"2019","journal-title":"Nat. Commun."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"239","DOI":"10.6339\/JDS.2007.05(2).396","article-title":"Singular spectrum analysis: Methodology and comparison","volume":"5","author":"Hassani","year":"2007","journal-title":"J. Data Sci."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Cristianini, N., and Shawe-Taylor, J. (2000). An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods, Cambridge University Press.","DOI":"10.1017\/CBO9780511801389"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_63","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1109\/34.192463","article-title":"A theory for multiresolution signal decomposition: The wavelet representation","volume":"11","author":"Mallat","year":"1989","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"1146","DOI":"10.1038\/nbt.1495","article-title":"The potential and challenges of nanopore sequencing","volume":"26","author":"Branton","year":"2008","journal-title":"Nat. Biotechnol."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1038\/nbt.4060","article-title":"Nanopore sequencing and assembly of a human genome with ultra-long reads","volume":"36","author":"Jain","year":"2018","journal-title":"Nat. Biotechnol."},{"key":"ref_67","unstructured":"Horenko, I., and Pospisil, L. (2023). Linearly-scalable learning of smooth low-dimensional patterns with permutation-aided entropic dimension reduction. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/7\/553\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:07:06Z","timestamp":1760108826000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/26\/7\/553"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,28]]},"references-count":67,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["e26070553"],"URL":"https:\/\/doi.org\/10.3390\/e26070553","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,28]]}}}