{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:39:19Z","timestamp":1787027959478,"version":"build-2736575974"},"reference-count":22,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T00:00:00Z","timestamp":1635120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100009708","name":"Novo Nordisk Fonden","doi-asserted-by":"publisher","award":["NNF20OC0062606"],"award-info":[{"award-number":["NNF20OC0062606"]}],"id":[{"id":"10.13039\/501100009708","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Autoencoders are commonly used in representation learning. They consist of an encoder and a decoder, which provide a straightforward method to map n-dimensional data in input space to a lower m-dimensional representation space and back. The decoder itself defines an m-dimensional manifold in input space. Inspired by manifold learning, we showed that the decoder can be trained on its own by learning the representations of the training samples along with the decoder weights using gradient descent. A sum-of-squares loss then corresponds to optimizing the manifold to have the smallest Euclidean distance to the training samples, and similarly for other loss functions. We derived expressions for the number of samples needed to specify the encoder and decoder and showed that the decoder generally requires much fewer training samples to be well-specified compared to the encoder. We discuss the training of autoencoders in this perspective and relate it to previous work in the field that uses noisy training examples and other types of regularization. On the natural image data sets MNIST and CIFAR10, we demonstrated that the decoder is much better suited to learn a low-dimensional representation, especially when trained on small data sets. Using simulated gene regulatory data, we further showed that the decoder alone leads to better generalization and meaningful representations. Our approach of training the decoder alone facilitates representation learning even on small data sets and can lead to improved training of autoencoders. We hope that the simple analyses presented will also contribute to an improved conceptual understanding of representation learning.<\/jats:p>","DOI":"10.3390\/e23111403","type":"journal-article","created":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T21:40:21Z","timestamp":1635198021000},"page":"1403","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A Manifold Learning Perspective on Representation Learning: Learning Decoder and Representations without an Encoder"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6016-8337","authenticated-orcid":false,"given":"Viktoria","family":"Schuster","sequence":"first","affiliation":[{"name":"Center for Health Data Science, University of Copenhagen, 2200 Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5147-6282","authenticated-orcid":false,"given":"Anders","family":"Krogh","sequence":"additional","affiliation":[{"name":"Center for Health Data Science, University of Copenhagen, 2200 Copenhagen, Denmark"},{"name":"Department of Computer Science, University of Copenhagen, 2100 Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Rumelhart, D., Hinton, G., and Williams, R. (1986). Learning Internal Representations by Error Propagation, MIT Press. Chapter 8.","DOI":"10.21236\/ADA164453"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation Learning: A Review and New Perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_3","unstructured":"Tschannen, M., Bachem, O., and Lucic, M. (2018). Recent Advances in Autoencoder-Based Representation Learning. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A. (2008, January 5\u20139). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning, ACM, Helsinki, Finland.","DOI":"10.1145\/1390156.1390294"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1016\/j.neucom.2015.02.023","article-title":"Laplacian Auto-Encoders: An explicit learning of nonlinear data manifold","volume":"160","author":"Jia","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_6","unstructured":"Cottrell, G., Munro, P., and Zipser, D. (1987, January 16\u201318). Learning Internal Representations from Gray-Scale Images: An Example of Extensional Programming. Proceedings of the Ninth Annual Conference of the Cognitive Science Society, Seattle, WA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1615","DOI":"10.1121\/1.395916","article-title":"Learning the Hidden Structure of Speech","volume":"83","author":"Elman","year":"1988","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1007\/BF00332918","article-title":"Auto-association by multilayer perceptrons and singular value decomposition","volume":"59","author":"Bourlard","year":"1988","journal-title":"Biol. Cybern."},{"key":"ref_9","unstructured":"Getoor, L., and Scheffer, T. (2011). Contractive Auto-Encoders: Explicit Invariance During Feature Extraction. ICML, Omnipress."},{"key":"ref_10","unstructured":"Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2010). Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Han, T., Lu, Y., Zhu, S.C., and Wu, Y.N. (2017, January 4\u20139). Alternating Back-Propagation for Generator Network. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10902"},{"key":"ref_12","first-page":"733","article-title":"A Cost Function for Internal Representations","volume":"Volume 2","author":"Touretzky","year":"1989","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"Lecun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_14","unstructured":"Krizhevsky, A. (2021, October 23). Learning Multiple Layers of Features from Tiny Images. Available online: https:\/\/www.cs.toronto.edu\/kriz\/learning-features-2009-TR.pdfhere."},{"key":"ref_15","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/BF00344251","article-title":"Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position","volume":"36","author":"Fukushima","year":"1980","journal-title":"Biol. Cybern."},{"key":"ref_17","unstructured":"F\u00fcrnkranz, J., and Joachims, T. (2010). Rectified Linear Units Improve Restricted Boltzmann Machines. ICML, Omnipress."},{"key":"ref_18","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1214\/aoms\/1177729586","article-title":"A Stochastic Approximation Method","volume":"22","author":"Robbins","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_20","unstructured":"Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., and Doll\u00e1r, P. (2020, January 13\u201319). Designing Network Design Spaces. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01044"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Xie, S., Girshick, R., Doll\u00e1r, P., Tu, Z., and He, K. (2017, January 21\u201326). Aggregated Residual Transformations for Deep Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.634"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/11\/1403\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:23:24Z","timestamp":1760167404000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/11\/1403"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,25]]},"references-count":22,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["e23111403"],"URL":"https:\/\/doi.org\/10.3390\/e23111403","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,25]]}}}