{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T05:03:20Z","timestamp":1768539800469,"version":"3.49.0"},"reference-count":33,"publisher":"MIT Press","issue":"10","license":[{"start":{"date-parts":[[2022,8,26]],"date-time":"2022-08-26T00:00:00Z","timestamp":1661472000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,9,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Disentanglement is a useful property in representation learning, which increases the interpretability of generative models such as variational autoencoders (VAE), generative adversarial models, and their many variants. Typically in such models, an increase in disentanglement performance is traded off with generation quality. In the context of latent space models, this work presents a representation learning framework that explicitly promotes disentanglement by encouraging orthogonal directions of variations. The proposed objective is the sum of an autoencoder error term along with a principal component analysis reconstruction error in the feature space. This has an interpretation of a restricted kernel machine with the eigenvector matrix valued on the Stiefel manifold. Our analysis shows that such a construction promotes disentanglement by matching the principal directions in the latent space with the directions of orthogonal variation in data space. In an alternating minimization scheme, we use the Cayley ADAM algorithm, a stochastic optimization method on the Stiefel manifold along with the Adam optimizer. Our theoretical discussion and various experiments show that the proposed model is an improvement over many VAE variants in terms of both generation quality and disentangled representation learning.<\/jats:p>","DOI":"10.1162\/neco_a_01528","type":"journal-article","created":{"date-parts":[[2022,8,26]],"date-time":"2022-08-26T22:05:21Z","timestamp":1661551521000},"page":"2009-2036","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":10,"title":["Disentangled Representation Learning and Generation With Manifold Optimization"],"prefix":"10.1162","volume":"34","author":[{"given":"Arun","family":"Pandey","sequence":"first","affiliation":[{"name":"KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, B-3001 Leuven, Belgium arun.pandey@esat.kuleuven.be"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Micha\u00ebl","family":"Fanuel","sequence":"additional","affiliation":[{"name":"Universit\u00e9 de Lille, CNRS, Centrale Lille, F-59000 Lille, France michael.fanuel@univ-lille.fr"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joachim","family":"Schreurs","sequence":"additional","affiliation":[{"name":"KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Kasteelpark Arenberg 10, B-3001 Leuven, Belgium joachim.schreurs@esat.kuleuven.be"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Johan A. K.","family":"Suykens","sequence":"additional","affiliation":[{"name":"KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Kasteelpark Arenberg 10, B-3001 Leuven, Belgium johan.suykens@esat.kuleuven.be"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2022,9,12]]},"reference":[{"key":"2022091222424699800_B1","doi-asserted-by":"crossref","DOI":"10.1515\/9781400830244","volume-title":"Optimization algorithms on matrix manifolds","author":"Absil","year":"2008"},{"key":"2022091222424699800_B2","first-page":"2258","volume-title":"Advances in neural information processing systems, 27","author":"Avron","year":"2014"},{"issue":"8","key":"2022091222424699800_B3","doi-asserted-by":"publisher","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2022091222424699800_B4","author":"Burgess","year":"2018","journal-title":"3Dshapes dataset."},{"key":"2022091222424699800_B5","article-title":"Understanding disentangling in \u03b2-VAE","author":"Burgess","year":"2017","journal-title":"NIPS 2017 Workshop on Learning Disentangled Representations: From Perception to Control."},{"key":"2022091222424699800_B6","first-page":"2610","volume-title":"Advances in neural information processing systems","author":"Chen","year":"2018"},{"key":"2022091222424699800_B7","first-page":"708","volume-title":"Advances in neural information processing systems","author":"Dupont","year":"2018"},{"key":"2022091222424699800_B8","article-title":"A framework for the quantitative evaluation of disentangled representations","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Eastwood","year":"2018"},{"key":"2022091222424699800_B9","article-title":"From variational to deterministic autoencoders","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Ghosh","year":"2020"},{"key":"2022091222424699800_B10","first-page":"6629","volume-title":"Advances in neural information processing systems, 30","author":"Heusel","year":"2017"},{"key":"2022091222424699800_B11","first-page":"6","article-title":"Beta-VAE: Learning basic visual concepts with a constrained variational framework","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Higgins","year":"2017"},{"key":"2022091222424699800_B12","first-page":"1765","article-title":"What kind of a graphical model is the brain?","volume-title":"Proceedings of the 19th International Joint Conference on Artificial Intelligence","author":"Hinton","year":"2005"},{"key":"2022091222424699800_B13","article-title":"Progressive growing of GANs for improved quality, stability, and variation","volume-title":"Proceedings of the International Conference on Learning Representations.","author":"Karras","year":"2017"},{"key":"2022091222424699800_B14","first-page":"2649","article-title":"Disentangling by factorising","volume-title":"Proceedings of the Thirty-Fifth International Conference on Machine Learning","author":"Kim","year":"2018"},{"key":"2022091222424699800_B15","article-title":"Auto-encoding variational Bayes","author":"Kingma","year":"2014","journal-title":"Proceedings of the International Conference on Learning Representations."},{"key":"2022091222424699800_B16","article-title":"Variational inference of disentangled latent concepts from unlabeled observations","author":"Kumar","year":"2018","journal-title":"Proceedings of the International Conference on Learning Representations."},{"key":"2022091222424699800_B17","author":"LeCun","year":"2010","journal-title":"MNIST handwritten digit database."},{"key":"2022091222424699800_B18","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2004.1315150","article-title":"Learning methods for generic object recognition with invariance to pose and lighting","volume-title":"Proceedings of the Conference on Computer Vision and Pattern Recognition.","author":"LeCun","year":"2004"},{"key":"2022091222424699800_B19","article-title":"Efficient Riemannian optimization on the Stiefel manifold via the Cayley transform","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Li","year":"2020"},{"key":"2022091222424699800_B20","article-title":"Challenging common assumptions in the unsupervised learning of disentangled representations","volume-title":"Proceedings of the International Conference on Machine Learning.","author":"Locatello","year":"2019"},{"key":"2022091222424699800_B21","article-title":"Disentangling factors of variations using few labels","volume-title":"International Conference on Learning Representations.","author":"Locatello","year":"2020"},{"key":"2022091222424699800_B22","author":"Matthey","year":"2017","journal-title":"dSprites: Disentanglement testing Sprites dataset."},{"key":"2022091222424699800_B23","volume-title":"Introductory lectures on convex optimization: A basic course.","author":"Nesterov","year":"2014"},{"key":"2022091222424699800_B24","article-title":"Reading digits in natural images with unsupervised feature learning","author":"Netzer","year":"2011","journal-title":"NIPS Workshop on Deep Learning and Unsupervised Feature Learning."},{"key":"2022091222424699800_B25","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-64583-0_54","article-title":"Robust generative restricted kernel machines using weighted conjugate feature duality","volume-title":"Proceedings of the Sixth International Conference on Machine Learning, Optimization, and Data Science","author":"Pandey","year":"2020"},{"key":"2022091222424699800_B26","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1016\/j.neunet.2020.12.010","article-title":"Generative restricted kernel machines: A framework for multi-view generation and disentangled feature learning","volume":"135","author":"Pandey","year":"2021","journal-title":"Neural Networks"},{"key":"2022091222424699800_B27","article-title":"Deep visual analogy-making","volume-title":"Advances in neural information processing systems","author":"Reed","year":"2015"},{"key":"2022091222424699800_B28","article-title":"Variational inference with normalizing flows","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Rezende","year":"2015"},{"key":"2022091222424699800_B29","doi-asserted-by":"crossref","first-page":"12398","DOI":"10.1109\/CVPR.2019.01269","article-title":"Variational autoencoders pursue PCA directions (by accident)","volume-title":"Proceedings of the 2019 IEEE\/CVF conference on Computer Vision and Pattern Recognition","author":"Rol\u00ednek","year":"2019"},{"key":"2022091222424699800_B30","article-title":"Deep Boltzmann machines","volume-title":"Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics","author":"Salakhutdinov","year":"2009"},{"issue":"8","key":"2022091222424699800_B31","doi-asserted-by":"publisher","first-page":"2123","DOI":"10.1162\/neco_a_00984","article-title":"Deep restricted kernel machines using conjugate feature duality","volume":"29","author":"Suykens","year":"2017","journal-title":"Neural Computation"},{"key":"2022091222424699800_B32","author":"Xiao","year":"2017","journal-title":"Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms."},{"issue":"3","key":"2022091222424699800_B33","doi-asserted-by":"publisher","first-page":"991","DOI":"10.1214\/16-AOS1472","article-title":"Randomized sketches for kernels: Fast and optimal nonparameteric regression","volume":"45","author":"Yang","year":"2017","journal-title":"Annals of Statistics"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/34\/10\/2009\/2042454\/neco_a_01528.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/34\/10\/2009\/2042454\/neco_a_01528.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,12]],"date-time":"2022-09-12T22:43:58Z","timestamp":1663022638000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/34\/10\/2009\/112570\/Disentangled-Representation-Learning-and"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,12]]},"references-count":33,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,9,12]]},"published-print":{"date-parts":[[2022,9,12]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01528","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,10]]},"published":{"date-parts":[[2022,9,12]]}}}