{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T04:56:00Z","timestamp":1768712160573,"version":"3.49.0"},"reference-count":57,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,2,12]],"date-time":"2020-02-12T00:00:00Z","timestamp":1581465600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Canonical Correlation Analysis (CCA) is a linear representation learning method that seeks maximally correlated variables in multi-view data. Nonlinear CCA extends this notion to a broader family of transformations, which are more powerful in many real-world applications. Given the joint probability, the Alternating Conditional Expectation (ACE) algorithm provides an optimal solution to the nonlinear CCA problem. However, it suffers from limited performance and an increasing computational burden when only a finite number of samples is available. In this work, we introduce an information-theoretic compressed representation framework for the nonlinear CCA problem (CRCCA), which extends the classical ACE approach. Our suggested framework seeks compact representations of the data that allow a maximal level of correlation. This way, we control the trade-off between the flexibility and the complexity of the model. CRCCA provides theoretical bounds and optimality conditions, as we establish fundamental connections to rate-distortion theory, the information bottleneck and remote source coding. In addition, it allows a soft dimensionality reduction, as the compression level is determined by the mutual information between the original noisy data and the extracted signals. Finally, we introduce a simple implementation of the CRCCA framework, based on lattice quantization.<\/jats:p>","DOI":"10.3390\/e22020208","type":"journal-article","created":{"date-parts":[[2020,2,14]],"date-time":"2020-02-14T10:09:57Z","timestamp":1581674997000},"page":"208","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5899-5608","authenticated-orcid":false,"given":"Amichai","family":"Painsky","sequence":"first","affiliation":[{"name":"The Industrial Engineering Department, Tel Aviv University, Tel Aviv 6997801, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1290-0482","authenticated-orcid":false,"given":"Meir","family":"Feder","sequence":"additional","affiliation":[{"name":"The School of Electrical Engineering, Tel Aviv University, Tel Aviv 6997801, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8086-4436","authenticated-orcid":false,"given":"Naftali","family":"Tishby","sequence":"additional","affiliation":[{"name":"The School of Computer Science and Engineering and the Interdisciplinary Center for Neural Computation, The Hebrew University of Jerusalem, Jerusalem 9190401, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1093\/biomet\/28.3-4.321","article-title":"Relations between two sets of variates","volume":"28","author":"Hotelling","year":"1936","journal-title":"Biometrika"},{"key":"ref_2","unstructured":"Arora, R., and Livescu, K. (2012, January 14). Kernel CCA for multi-view learning of acoustic features using articulatory measurements. Proceedings of the Symposium on Machine Learning in Speech and Language Processing, Portland, OR, USA."},{"key":"ref_3","unstructured":"Dhillon, P., Foster, D.P., and Ungar, L.H. (2011). Multi-view learning of word embeddings via cca. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., and Lazebnik, S. (2014). Improving image-sentence embeddings using large weakly annotated photo collections. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-10593-2_35"},{"key":"ref_5","unstructured":"Slaney, M., and Covell, M. (2011). Facesync: A linear operator for measuring synchronization of video facial images and audio tracks. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kim, T.K., Wong, S.F., and Cipolla, R. (2007). Tensor canonical correlation analysis for action classification. Computer Vision and Pattern Recognition, IEEE.","DOI":"10.1109\/CVPR.2007.383137"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1111\/j.2044-8317.1983.tb00765.x","article-title":"Nonlinear canonical correlation","volume":"36","year":"1983","journal-title":"Br. J. Math. Stat. Psychol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1080\/01621459.1985.10478157","article-title":"Estimating optimal transformations for multiple regression and correlation","volume":"80","author":"Breiman","year":"1985","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_9","unstructured":"Akaho, S. (2001, January 15\u201319). A Kernel Method For Canonical Correlation Analysis. Proceedings of the International Meeting of the Psychometric Society, Osaka, Japan."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1109\/TNN.2007.891186","article-title":"Variational Bayesian approach to canonical correlation analysis","volume":"18","author":"Wang","year":"2007","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_11","first-page":"965","article-title":"Bayesian canonical correlation analysis","volume":"14","author":"Klami","year":"2013","journal-title":"J. Mach. Learn. Res."},{"key":"ref_12","unstructured":"Andrew, G., Arora, R., Bilmes, J.A., and Livescu, K. (2013, January 16\u201321). Deep canonical correlation analysis. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA."},{"key":"ref_13","unstructured":"Michaeli, T., Wang, W., and Livescu, K. (2016, January 19\u201324). Nonparametric canonical correlation analysis. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2639","DOI":"10.1162\/0899766042321814","article-title":"Canonical correlation analysis: An overview with application to learning methods","volume":"16","author":"Hardoon","year":"2004","journal-title":"Neural Comput."},{"key":"ref_15","unstructured":"Wang, W., Arora, R., Livescu, K., and Bilmes, J.A. (2015, January 6\u201311). On Deep Multi-View Representation Learning. Proceedings of the 33rd International Conference on Machine Learning, Lille, France."},{"key":"ref_16","unstructured":"Arora, R., Mianjy, P., and Marinov, T. (2016, January 19\u201324). Stochastic optimization for multiview representation learning using partial least squares. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_17","unstructured":"Arora, R., Marinov, T.V., Mianjy, P., and Srebro, N. (2017, January 4\u20139). Stochastic Approximation for Canonical Correlation Analysis. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, NY, USA."},{"key":"ref_18","unstructured":"Wang, W., Arora, R., Livescu, K., and Srebro, N. (October, January 30). Stochastic optimization for deep CCA via nonlinear orthogonal iterations. Proceedings of the 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA."},{"key":"ref_19","unstructured":"Cover, T.M., and Thomas, J.A. (2012). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_20","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, Monticello, IL, USA."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1007\/BF02294765","article-title":"Interpreting canonical correlation analysis through biplots of structure correlations and weights","volume":"55","year":"1990","journal-title":"Psychometrika"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1080\/02664760500054202","article-title":"Enriched biplots for canonical correlation analysis","volume":"32","author":"Graffelman","year":"2005","journal-title":"J. Appl. Stat."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1093\/biostatistics\/kxp008","article-title":"A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis","volume":"10","author":"Witten","year":"2009","journal-title":"Biostatistics"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2202\/1544-6115.1470","article-title":"Extensions of sparse canonical correlation analysis with applications to genomic data","volume":"8","author":"Witten","year":"2009","journal-title":"Stat. Appl. Genet. Mol. Biol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.gexplo.2018.07.014","article-title":"Exploration of geochemical data with compositional canonical biplots","volume":"194","author":"Graffelman","year":"2018","journal-title":"J. Geochem. Explor."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1214\/aoms\/1177706532","article-title":"The structure of bivariate distributions","volume":"29","author":"Lancaster","year":"1958","journal-title":"Ann. Math. Stat."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1017\/S1446788700026707","article-title":"The general theory of canonical correlation and its relation to functional analysis","volume":"2","author":"Hannan","year":"1961","journal-title":"J. Aust. Math. Soc."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/0167-9473(94)90136-8","article-title":"OVERALS: Nonlinear canonical correlation with k sets of variables","volume":"18","author":"Dijksterhuis","year":"1994","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_29","unstructured":"Gifi, A. (1990). Nonlinear Multivariate Analysis, Wiley-VCH."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Makur, A., Kozynski, F., Huang, S.L., and Zheng, L. (October, January 30). An efficient algorithm for information decomposition and extraction. Proceedings of the 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA.","DOI":"10.1109\/ALLERTON.2015.7447113"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1142\/S012906570000034X","article-title":"Kernel and nonlinear canonical correlation analysis","volume":"10","author":"Lai","year":"2000","journal-title":"Int. J. Neural Syst."},{"key":"ref_32","unstructured":"Uurtio, V., Bhadra, S., and Rousu, J. (2019, January 10\u201315). Large-Scale Sparse Kernel Canonical Correlation Analysis. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"18280","DOI":"10.1021\/acs.iecr.9b03069","article-title":"Accelerated Kernel Canonical Correlation Analysis with Fault Relevance for Nonlinear Process Fault Isolation","volume":"58","author":"Yu","year":"2019","journal-title":"Ind. Eng. Chem. Res."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Sch\u00f6lkopf, B., Herbrich, R., and Smola, A.J. (2001). A generalized representer theorem. International Conference on Computational Learning Theory, Springer.","DOI":"10.1007\/3-540-44581-1_27"},{"key":"ref_35","first-page":"1","article-title":"Kernel independent component analysis","volume":"3","author":"Bach","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1137\/090771806","article-title":"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions","volume":"53","author":"Halko","year":"2011","journal-title":"SIAM Rev."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1016\/0005-1098(78)90005-5","article-title":"Modeling by shortest data description","volume":"14","author":"Rissanen","year":"1978","journal-title":"Automatica"},{"key":"ref_38","unstructured":"Chigirev, D.V., and Bialek, W. (2003, January 8\u201313). Optimal Manifold Representation of Data: An Information Theoretic Approach. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Vera, M., Piantanida, P., and Vega, L.R. (2018, January 17\u201322). The Role of the Information Bottleneck in Representation Learning. Proceedings of the 2018 IEEE International Symposium on Information Theory (ISIT), Vail, CO, USA.","DOI":"10.1109\/ISIT.2018.8437679"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1109\/TIT.1962.1057738","article-title":"Information transmission with additional noise","volume":"8","author":"Dobrushin","year":"1962","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"406","DOI":"10.1109\/TIT.1970.1054469","article-title":"Transmission of noisy information to a noisy receiver with minimum distortion","volume":"16","author":"Wolf","year":"1970","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_42","first-page":"165","article-title":"Information bottleneck for Gaussian variables","volume":"6","author":"Chechik","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1109\/18.556117","article-title":"Empirical quantizer design in the presence of source noise or channel noise","volume":"43","author":"Linder","year":"1997","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_44","unstructured":"Gy\u00f6rfi, L., Kohler, M., Krzyzak, A., and Walk, H. (2006). A Distribution-Free Theory of Nonparametric Regression, Springer."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"867","DOI":"10.1109\/TIT.2007.913565","article-title":"Quantization for nonparametric regression","volume":"54","author":"Wegkamp","year":"2008","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Zamir, R. (2014). Lattice Coding for Signals and Networks: A Structured Coding Approach to Quantization, Modulation, and Multiuser Information Theory, Cambridge University Press.","DOI":"10.1017\/CBO9781139045520"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1109\/TIT.1985.1057034","article-title":"On universal quantization","volume":"31","author":"Ziv","year":"1985","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zamir, R. (2009). Lattices are everywhere. Information Theory and Applications Workshop, IEEE.","DOI":"10.1109\/ITA.2009.5044976"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1109\/18.119699","article-title":"On universal quantization by randomized uniform\/lattice quantizers","volume":"38","author":"Zamir","year":"1992","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1340","DOI":"10.1109\/18.532876","article-title":"Information rates of pre\/post-filtered dithered quantizers","volume":"42","author":"Zamir","year":"1996","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Velloso, E., Bulling, A., Gellersen, H., Ugulino, W., and Fuks, H. (2013, January 7\u20138). Qualitative activity recognition of weight lifting exercises. Proceedings of the 4th Augmented Human International Conference, Stuttgart, Germany.","DOI":"10.1145\/2459236.2459256"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1093\/biomet\/40.3-4.237","article-title":"The population frequencies of species and the estimation of population parameters","volume":"40","author":"Good","year":"1953","journal-title":"Biometrika"},{"key":"ref_54","unstructured":"Orlitsky, A., and Suresh, A.T. (2015, January 7\u201312). Competitive distribution estimation: Why is good-turing good. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Painsky, A., and Wornell, G. (2018, January 17\u201322). On the universality of the logistic loss function. Proceedings of the 2018 IEEE International Symposium on Information Theory (ISIT), Vail, CO, USA.","DOI":"10.1109\/ISIT.2018.8437786"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Painsky, A., and Wornell, G.W. (2019). Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss. IEEE Trans. Inf. Theory.","DOI":"10.1109\/TIT.2019.2958705"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1739","DOI":"10.1162\/neco.2006.18.8.1739","article-title":"Multivariate information bottleneck","volume":"18","author":"Slonim","year":"2006","journal-title":"Neural Comput."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/208\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:57:04Z","timestamp":1760173024000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/208"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,12]]},"references-count":57,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["e22020208"],"URL":"https:\/\/doi.org\/10.3390\/e22020208","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,12]]}}}