{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T10:24:26Z","timestamp":1792232666379,"version":"4.4.0"},"reference-count":45,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T00:00:00Z","timestamp":1643241600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In statistical inference, the information-theoretic performance limits can often be expressed in terms of a statistical divergence between the underlying statistical models (e.g., in binary hypothesis testing, the error probability is related to the total variation distance between the statistical models). As the data dimension grows, computing the statistics involved in decision-making and the attendant performance limits (divergence measures) face complexity and stability challenges. Dimensionality reduction addresses these challenges at the expense of compromising the performance (the divergence reduces by the data-processing inequality). This paper considers linear dimensionality reduction such that the divergence between the models is maximally preserved. Specifically, this paper focuses on Gaussian models where we investigate discriminant analysis under five f-divergence measures (Kullback\u2013Leibler, symmetrized Kullback\u2013Leibler, Hellinger, total variation, and \u03c72). We characterize the optimal design of the linear transformation of the data onto a lower-dimensional subspace for zero-mean Gaussian models and employ numerical algorithms to find the design for general Gaussian models with non-zero means. There are two key observations for zero-mean Gaussian models. First, projections are not necessarily along the largest modes of the covariance matrix of the data, and, in some situations, they can even be along the smallest modes. Secondly, under specific regimes, the optimal design of subspace projection is identical under all the f-divergence measures considered, rendering a degree of universality to the design, independent of the inference problem of interest.<\/jats:p>","DOI":"10.3390\/e24020188","type":"journal-article","created":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T21:59:55Z","timestamp":1643320795000},"page":"188","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Discriminant Analysis under f-Divergence Measures"],"prefix":"10.3390","volume":"24","author":[{"given":"Anmol","family":"Dwivedi","sequence":"first","affiliation":[{"name":"Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sihui","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Tajer","sequence":"additional","affiliation":[{"name":"Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,27]]},"reference":[{"key":"ref_1","unstructured":"Kunisky, D., Wein, A.S., and Bandeira, A.S. (2019). Notes on computational hardness of hypothesis testing: Predictions using the low-degree likelihood ratio. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gamarnik, D., Jagannath, A., and Wein, A.S. (2020). Low-degree hardness of random optimization problems. arXiv.","DOI":"10.1109\/FOCS46700.2020.00021"},{"key":"ref_3","first-page":"66","article-title":"Dimensionality reduction: A comparative review","volume":"10","author":"Postma","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lee, J.A., and Verleysen, M. (2007). Nonlinear Dimensionality Reduction, Springer Science.","DOI":"10.1007\/978-0-387-39351-3"},{"key":"ref_5","unstructured":"DeMers, D., and Cottrell, G.W. (1993, January 3\u20136). Non-linear dimensionality reduction. Proceedings of the Advances in Neural Information Processing Systems, Denver, CO, USA."},{"key":"ref_6","first-page":"2859","article-title":"Linear dimensionality reduction: Survey, insights, and generalizations","volume":"16","author":"Cunningham","year":"2015","journal-title":"J. Mach. Learn. Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1080\/14786440109462720","article-title":"On lines and planes of closest fit to systems of points in space","volume":"2","author":"Pearson","year":"1901","journal-title":"Philos. Mag."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/BF02288367","article-title":"The approximation of one matrix by another of lower rank","volume":"1","author":"Eckart","year":"1936","journal-title":"Psychometrika"},{"key":"ref_9","unstructured":"Jolliffe, I. (2002). Principal Component Analysis, Springer."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1007\/BF02288916","article-title":"Multidimensional scaling: I. Theory and method","volume":"17","author":"Torgerson","year":"1952","journal-title":"Psychometrika"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Cox, T.F., and Cox, M.A. (2008). Multidimensional scaling. Handbook of Data Visualization, Springer.","DOI":"10.1007\/978-3-540-33037-0_14"},{"key":"ref_12","unstructured":"Borg, I., and Groenen, P.J. (2005). Modern Multidimensional Scaling: Theory and Applications, Springer."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Izenman, A.J. (2013). Linear discriminant analysis. Modern Multivariate Statistical Techniques, Springer.","DOI":"10.1007\/978-0-387-78189-1_8"},{"key":"ref_14","first-page":"1307","article-title":"Sufficient dimensionality reduction","volume":"3","author":"Globerson","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1111\/j.1469-1809.1936.tb02137.x","article-title":"The use of multiple measurements in taxonomic problems","volume":"7","author":"Fisher","year":"1936","journal-title":"Ann. Eugen."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1111\/j.2517-6161.1948.tb00008.x","article-title":"The utilization of multiple measurements in problems of biological classification","volume":"10","author":"Rao","year":"1948","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_17","unstructured":"Fukunaga, K. (2013). Introduction to Statistical Pattern Recognition, Elsevier."},{"key":"ref_18","first-page":"1","article-title":"Linear discriminant analysis- A brief tutorial","volume":"18","author":"Suresh","year":"1998","journal-title":"Inst. Signal Inf. Process."},{"key":"ref_19","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_20","unstructured":"Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer Science & Business Media."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_23","first-page":"745","article-title":"On the general definition of the amount of information","volume":"11","author":"Gelfand","year":"1956","journal-title":"Dokl. Akad. Nauk SSSR"},{"key":"ref_24","first-page":"379","article-title":"Eine Informationstheoretische Ungleichung und ihre Anwendung auf den Bewis der Ergodizit\u00e4t von Markhoffschen Ketten","volume":"8","year":"1948","journal-title":"Magy. Tudom\u00e1nyos Akad. Mat. Kut. Int\u00e9zet\u00e9nek K\u00f6zlem\u00e9nyei"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1111\/j.2517-6161.1966.tb00626.x","article-title":"General Class of Coefficients of Divergence of One Distribution from Another","volume":"28","author":"Ali","year":"1966","journal-title":"J. R. Stat. Soc."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1143\/JPSJ.18.328","article-title":"Markov Processes and the H-Theorem","volume":"18","author":"Morimoto","year":"1963","journal-title":"J. Phys. Soc. Jpn."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/S0019-9958(71)90065-9","article-title":"Information-theoretical considerations on estimation problems","volume":"19","author":"Arimoto","year":"1971","journal-title":"Inf. Control"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.1109\/18.149496","article-title":"Distribution estimation consistent in total variation and in two types of information divergence","volume":"38","author":"Barron","year":"1992","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"999","DOI":"10.1109\/18.669143","article-title":"About the asymptotic accuracy of Barron density estimates","volume":"44","author":"Berlinet","year":"1998","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_30","unstructured":"Gyorfi, L., Morvai, G., and Vajda, I. (2000, January 25\u201330). Information-theoretic methods in testing the goodness of fit. Proceedings of the IEEE International Symposium on Information Theory, Sorrento, Italy."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"4394","DOI":"10.1109\/TIT.2006.881731","article-title":"On Divergences and Informations in Statistics and Information Theory","volume":"52","author":"Liese","year":"2006","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1109\/TCOM.1967.1089532","article-title":"The Divergence and Bhattacharyya Distance Measures in Signal Selection","volume":"15","author":"Kailath","year":"1967","journal-title":"IEEE Trans. Commun. Technol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1109\/TIT.1980.1056249","article-title":"Robust decision design using a distance criterion","volume":"26","author":"Poor","year":"1980","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1109\/18.54897","article-title":"Information-theoretic asymptotics of Bayes methods","volume":"36","author":"Clarke","year":"1990","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"3230","DOI":"10.1109\/TIT.2011.2137353","article-title":"On Pairs of f-divergences and their joint range","volume":"57","author":"Harremoes","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"5973","DOI":"10.1109\/TIT.2016.2603151","article-title":"f-Divergence Inequalities","volume":"62","author":"Sason","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Sason, I. (2018). On f-divergence: Integral representations, local behavior, and inequalities. Entropy, 20.","DOI":"10.3390\/e20050383"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Rao, C.R., and Statistiker, M. (1973). Linear Statistical Inference and Its Applications, Wiley.","DOI":"10.1002\/9780470316436"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Poor, H.V., and Hadjiliadis, O. (2008). Quickest Detection, Cambridge University Press.","DOI":"10.1017\/CBO9780511754678"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1111\/j.1467-842X.2004.00328.x","article-title":"Criteria for linear model selection based on Kullback\u2019s symmetric divergence","volume":"46","author":"Cavanaugh","year":"2004","journal-title":"Aust. N. Z. J. Stat."},{"key":"ref_41","unstructured":"Devroye, L., Mehrabian, A., and Reddad, T. (2020). The total variation distance between high-dimensional Gaussians. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1214\/aos\/1176346587","article-title":"Optimal detection of a change in distribution","volume":"13","author":"Pollak","year":"1985","journal-title":"Ann. Stat."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1109\/JSTSP.2008.2011112","article-title":"Information preserving component analysis: Data projections for flow cytometry analysis","volume":"3","author":"Carter","year":"2009","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1007\/s10107-012-0584-1","article-title":"A feasible method for optimization with orthogonality constraints","volume":"142","author":"Wen","year":"2013","journal-title":"Math. Program."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1137\/S0895479895290954","article-title":"The geometry of algorithms with orthogonality constraints","volume":"20","author":"Edelman","year":"1998","journal-title":"SIAM J. Matrix Anal. Appl."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/2\/188\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:08:55Z","timestamp":1760134135000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/2\/188"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,27]]},"references-count":45,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["e24020188"],"URL":"https:\/\/doi.org\/10.3390\/e24020188","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,27]]}}}