{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T14:32:26Z","timestamp":1781533946870,"version":"3.54.5"},"reference-count":44,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T00:00:00Z","timestamp":1775606400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Interdisciplinary Research Center for Intelligent Secure Systems (IRC-ISS) at King Fahd University of Petroleum &amp; Minerals","award":["INSS2522"],"award-info":[{"award-number":["INSS2522"]}]},{"award":["INSS2522"],"award-info":[{"award-number":["INSS2522"]}],"id":[{"id":"https:\/\/ror.org\/03yez3163","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Autoencoder-based models have become a fundamental component of unsupervised and self-supervised learning in natural language processing (NLP), enabling models to learn compact latent representations through input reconstruction. From early denoising autoencoders to probabilistic variational autoencoders (VAEs) and transformer-based masked autoencoding, reconstruction-driven objectives have played a significant role in shaping modern approaches to text representation and generation. This review provides a comprehensive analysis of the evolution of autoencoder architectures and training objectives in NLP, and synthesizes applications of VAEs across language modeling, controllable text generation, machine translation, sentiment modeling, and multilingual representation learning. Although previous surveys have examined deep generative models or representation learning in NLP, there remains a lack of a unified review that systematically connects classical autoencoder variants, variational formulations, and modern transformer-based masked autoencoders within a single conceptual framework. To address this gap, this work consolidates architectural developments, training objectives, and major application domains under a reconstruction-based learning perspective, offering a structured comparison of modeling choices, datasets, and evaluation practices. Our analysis highlights the strengths and limitations of existing approaches, discusses the ongoing influence of autoencoder-style learning in NLP, and outlines future research directions focused on improving training stability, designing more structured latent spaces, and enhancing multilingual representation learning.<\/jats:p>","DOI":"10.3390\/computers15040232","type":"journal-article","created":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T15:26:39Z","timestamp":1775661999000},"page":"232","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Autoencoders in Natural Language Processing: A Comprehensive Review"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-8495-7753","authenticated-orcid":false,"given":"Moussa","family":"Redah","sequence":"first","affiliation":[{"name":"Information and Computer Science Department, King Fahd University of Petroleum and Minerals (KFUPM), Dhahran 31261, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6760-3506","authenticated-orcid":false,"given":"Wasfi G.","family":"Al-Khatib","sequence":"additional","affiliation":[{"name":"Information and Computer Science Department, King Fahd University of Petroleum and Minerals (KFUPM), Dhahran 31261, Saudi Arabia"},{"name":"Interdisciplinary Research Center for Intelligent Secure Systems (IRC-ISS), King Fahd University of Petroleum and Minerals (KFUPM), Dhahran 31261, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,4,8]]},"reference":[{"key":"ref_1","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the NAACL-HLT, Minneapolis, MN, USA."},{"key":"ref_2","unstructured":"Yang, Z., Hu, Z., Salakhutdinov, R., and Berg-Kirkpatrick, T. (2017, January 6\u201311). Improved variational autoencoders for text modeling using dilated convolutions. Proceedings of the 34th International Conference on Machine Learning, ICML\u201917, Sydney, NSW, Australia."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wang, S., Li, Z., Ge, S., Xie, J., and Zhou, M. (2022, January 22\u201327). Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), Dublin, Ireland.","DOI":"10.18653\/v1\/2022.acl-long.185"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: An updated guideline for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, London, UK, 13\u201314 May 2014, Association for Computing Machinery.","DOI":"10.1145\/2601248.2601268"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"110176","DOI":"10.1016\/j.asoc.2023.110176","article-title":"A comprehensive survey on design and application of autoencoder in deep learning","volume":"138","author":"Li","year":"2023","journal-title":"Appl. Soft Comput."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1007\/s10462-023-10662-6","article-title":"Autoencoders and their applications in machine learning: A survey","volume":"57","author":"Berahmand","year":"2024","journal-title":"Artif. Intell. Rev."},{"key":"ref_8","unstructured":"Sutskever, I., Vinyals, O., and Le, Q.V. (2014, January 8\u201313). Sequence to sequence learning with neural networks. Proceedings of the 28th International Conference on Neural Information Processing Systems, Cambridge, MA, USA."},{"key":"ref_9","unstructured":"Amiriparian, S., Freitag, M.J., Cummins, N., and Schuller, B. (2017, January 16\u201317). Sequence to Sequence Autoencoders for Unsupervised Representation Learning from Audio. Proceedings of the Workshop on Detection and Classification of Acoustic Scenes and Events, Munich, Germany."},{"key":"ref_10","unstructured":"Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (2014). An Autoencoder Approach to Learning Bilingual Word Representations. Proceedings of the Advances in Neural Information Processing Systems, Montreal, Quebec, Canada, 8\u201313 December 2014, Curran Associates, Inc."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzm\u00e1n, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020, January 5\u201310). Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the ACL, Online.","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"ref_12","unstructured":"Cheng, K., Lu, W., and Zhang, R. (August, January 28). Variational Semi-Supervised Aspect-Term Sentiment Analysis. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3402884","article-title":"Condition-Transforming Variational Autoencoder for Generating Diverse Short Text Conversations","volume":"19","author":"Ruan","year":"2020","journal-title":"ACM Trans. Asian Low-Resour. Lang. Inf. Process."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhao, K., Ding, H., Ye, K., and Cui, X. (2021). A Transformer-Based Hierarchical Variational AutoEncoder Combined Hidden Markov Model for Long Text Generation. Entropy, 23.","DOI":"10.3390\/e23101277"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wang, W., Gan, Z., Xu, H., Zhang, R., Wang, G., Shen, D., Chen, C., and Carin, L. (2019, January 2\u20137). Topic-Guided Variational Autoencoders for Text Generation. Proceedings of the NAACL-HLT, Minneapolis, MI, USA.","DOI":"10.18653\/v1\/N19-1015"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Mai, F., Pappas, N., Montero, I., Smith, N.A., and Henderson, J. (2020). Plug and play autoencoders for conditional text generation. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.491"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1194","DOI":"10.1016\/j.fmre.2024.01.001","article-title":"A recent survey on controllable text generation: A causal perspective","volume":"5","author":"Wang","year":"2025","journal-title":"Fundam. Res."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Montero, I., Pappas, N., and Smith, N.A. (2021, January 7\u201311). Sentence Bottleneck Autoencoders from Transformer Language Models. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic.","DOI":"10.18653\/v1\/2021.emnlp-main.137"},{"key":"ref_19","unstructured":"Tay, Y., Dehghani, M., Tran, V.Q., Garcia, X., Wei, J., Wang, X., Chung, H.W., Shakeri, S., Bahri, D., and Schuster, T. (2023). UL2: Unifying Language Learning Paradigms. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kim, Y.J., and Lee, S.P. (2024). A Generation of Enhanced Data by Variational Autoencoders and Diffusion Modeling. Electronics, 13.","DOI":"10.3390\/electronics13071314"},{"key":"ref_21","unstructured":"Li, J., Li, D., Savarese, S., and Hoi, S. (2023, January 23\u201329). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Proceedings of the International Conference on Machine Learning, PMLR, Honolulu, HI, USA."},{"key":"ref_22","first-page":"5885","article-title":"InfoVAE: Balancing Learning and Inference in Variational Autoencoders","volume":"33","author":"Zhao","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Bowman, S., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. (2016, January 11\u201312). Generating sentences from a continuous space. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany.","DOI":"10.18653\/v1\/K16-1002"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Alayba, A.M. (2025). Arabic Natural Language Processing (NLP): A Comprehensive Review of Challenges, Techniques, and Emerging Trends. Computers, 14.","DOI":"10.3390\/computers14110497"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Akram, M.W., Salman, M., Bashir, M.F., Salman, S.M.S., Gadekallu, T.R., and Javed, A.R. (2022). A Novel Deep Auto-Encoder Based Linguistics Clustering Model for Social Text. Transactions on Asian and Low-Resource Language Information Processi, ACM.","DOI":"10.1145\/3527838"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Brunner, G., Wang, Y., Wattenhofer, R., and Weigelt, M. (2018, January 5\u20137). Disentangling the latent space of (variational) autoencoders for NLP. Proceedings of the UK Workshop on Computational Intelligence, Nottingham, UK.","DOI":"10.1007\/978-3-319-97982-3_13"},{"key":"ref_27","unstructured":"Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Li, C., Gao, X., Li, Y., Peng, B., Zhang, Y., Huang, D., He, X., and Gao, J. (2020, January 16\u201320). OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online Event.","DOI":"10.18653\/v1\/2020.emnlp-main.378"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018, January 1). GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium.","DOI":"10.18653\/v1\/W18-5446"},{"key":"ref_30","unstructured":"Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2019, January 8\u201314). Superglue: A stickier benchmark for general-purpose language understanding systems. Proceedings of the Advances in Neural Information Processing Systems 32, Vancouver, BC, Canada."},{"key":"ref_31","unstructured":"He, J., Spokoyny, D., Neubig, G., and Berg-Kirkpatrick, T. (2019, January 6\u20139). Lagging Inference Networks and Posterior Collapse in Variational Autoencoders. Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Fu, H., Li, C., Liu, X., Gao, J., Celikyilmaz, A., and Carin, L. (2019, January 2\u20137). Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Minneapolis, MN, USA.","DOI":"10.18653\/v1\/N19-1021"},{"key":"ref_33","unstructured":"Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., and Ray, A. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Song, T., Sun, J., Liu, X., and Peng, W. (2024, January 20\u201325). Scale-VAE: Preventing Posterior Collapse in Variational Autoencoder. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), Torino, Italy.","DOI":"10.63317\/2imbghwp6u9v"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ok, C., Lee, G., and Lee, K. (2022). Informative Language Encoding by Variational Autoencoders Using Transformer. Appl. Sci., 12.","DOI":"10.3390\/app12167968"},{"key":"ref_36","unstructured":"Kim, Y., Wiseman, S., Miller, A.C., Sontag, D., and Rush, A.M. (2018, January 10\u201315). Semi-Amortized Variational Autoencoders. Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_37","first-page":"9","article-title":"Language Models are Unsupervised Multitask Learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI"},{"key":"ref_38","unstructured":"Ma, X., Li, Y., Gao, J., and Chen, Z. (2019, January 15). Mutual Posterior-Divergence Regularization for Variational Autoencoders. Proceedings of the 37th International Conference on Machine Learning (ICML) Workshop, Long Beach, CA, USA."},{"key":"ref_39","unstructured":"SAEBench, A. (2025). SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Models. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020, January 5\u201310). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"ref_41","first-page":"5485","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref_42","unstructured":"Lan, T., Zhou, Y.H., Ma, Z.A., Sun, F., Sun, R.Q., Luo, J., Tu, R.C., Huang, H., Xu, C., and Wu, Z. (2025). A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation, Association for Computational Linguistics.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_44","unstructured":"Lin, C.Y. (2004, January 21\u201326). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/4\/232\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T04:10:47Z","timestamp":1775794247000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/4\/232"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,8]]},"references-count":44,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,4]]}},"alternative-id":["computers15040232"],"URL":"https:\/\/doi.org\/10.3390\/computers15040232","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,8]]}}}