{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T05:30:39Z","timestamp":1767850239989,"version":"3.49.0"},"reference-count":38,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,4,14]],"date-time":"2025-04-14T00:00:00Z","timestamp":1744588800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"International Chinese Language Education Research Program","award":["23YH35C"],"award-info":[{"award-number":["23YH35C"]}]},{"name":"International Chinese Language Education Research Program","award":["62177001"],"award-info":[{"award-number":["62177001"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["23YH35C"],"award-info":[{"award-number":["23YH35C"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62177001"],"award-info":[{"award-number":["62177001"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>This paper introduces a novel variational autoencoder model termed DVAE to prevent posterior collapse in text modeling. DVAE employs a dual-path architecture within its decoder: path A and path B. Path A makes the direct input of text instances into the decoder, whereas path B replaces a subset of word tokens in the text instances with a generic unknown token before their input into the decoder. A stopping strategy is implemented, wherein both paths are concurrently active during the early phases of training. As the model progresses towards convergence, path B is removed. To further refine the performance, a KL weight dropout method is employed, which randomly sets certain dimensions of the KL weight to zero during the annealing process. DVAE compels the latent variables to encode more information about the input texts through path B and fully utilize the expressiveness of the decoder, as well as avoiding the local optimum when path B is active through path A and the stopping strategy. Furthermore, the KL weight dropout method augments the number of active units within the latent variables. Experimental results show the excellent performance of DVAE in density estimation, representation learning, and text generation.<\/jats:p>","DOI":"10.3390\/e27040423","type":"journal-article","created":{"date-parts":[[2025,4,14]],"date-time":"2025-04-14T06:18:36Z","timestamp":1744611516000},"page":"423","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Preventing Posterior Collapse with DVAE for Text Modeling"],"prefix":"10.3390","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6390-1442","authenticated-orcid":false,"given":"Tianbao","family":"Song","sequence":"first","affiliation":[{"name":"School of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing 100048, China"}]},{"given":"Zongyi","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing 100048, China"}]},{"given":"Xin","family":"Liu","sequence":"additional","affiliation":[{"name":"The 15th Research Institute of China Electronics Technology Group Corporation, Beijing 100083, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4462-8660","authenticated-orcid":false,"given":"Jingbo","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Information Engineering, Minzu University of China, Beijing 100081, China"},{"name":"National Language Resource Monitoring and Research Center of Minority Language, Minzu University of China, Beijing 100081, China"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,14]]},"reference":[{"key":"ref_1","unstructured":"Kingma, D.P., and Welling, M. (2014, January 14\u201316). Auto-encoding variational bayes. Proceedings of the International Conference on Learning Representations, Banff, AB, Canada."},{"key":"ref_2","unstructured":"Rezende, D.J., Mohamed, S., and Wierstra, D. (2014, January 21\u201326). Stochastic backpropagation and approximate inference in deep generative models. Proceedings of the International Conference on Machine Learning, Beijing, China."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"106447","DOI":"10.1016\/j.engappai.2023.106447","article-title":"Decoupled variational autoencoder with interactive attention for affective text generation","volume":"123","author":"Chen","year":"2023","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1016\/j.ins.2022.10.052","article-title":"HVAE: A deep generative model via hierarchical variational auto-encoder for multi-view document modeling","volume":"623","author":"Bai","year":"2023","journal-title":"Inf. Sci."},{"key":"ref_5","unstructured":"Kviman, O., Mol\u00e9n, R., Hotti, A., Kurt, S., Elvira, V., and Lagergren, J. (2023, January 23\u201329). Cooperation in the latent space: The benefits of adding mixture components in variational autoencoders. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., and Bengio, S. (2016, January 11\u201312). Generating sentences from a continuous space. Proceedings of the SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany.","DOI":"10.18653\/v1\/K16-1002"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3288","DOI":"10.1109\/TNNLS.2024.3359275","article-title":"EVAE: Evolutionary variational autoencoder","volume":"36","author":"Wu","year":"2024","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Fu, H., Li, C., Liu, X., Gao, J., Celikyilmaz, A., and Carin, L. (2019, January 3\u20135). Cyclical annealing schedule: A simple approach to mitigating KL vanishing. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, MN, USA.","DOI":"10.18653\/v1\/N19-1021"},{"key":"ref_9","unstructured":"Dieng, A.B., Kim, Y., Rush, A.M., and Blei, D.M. (2019, January 16\u201318). Avoiding latent variable collapse with generative skip models. Proceedings of the International Conference on Artificial Intelligence and Statistics, Okinawa, Japan."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ghose, A., and Poupart, P. (2023, January 6\u201310). Contrastive Deterministic Autoencoders For Language Modeling. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore.","DOI":"10.18653\/v1\/2023.findings-emnlp.567"},{"key":"ref_11","unstructured":"Kim, Y., Wiseman, S., Miller, A., Sontag, D., and Rush, A. (2018, January 10\u201315). Semi-amortized variational autoencoders. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_12","unstructured":"He, J., Spokoyny, D., Neubig, G., and Berg-Kirkpatrick, T. (2019, January 6\u20139). Lagging inference networks and posterior collapse in variational autoencoders. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_13","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016, January 2\u20134). Beta-VAE: Learning basic visual concepts with a constrained variational framework. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R.A., and Murphy, K. (2018, January 10\u201315). Fixing a broken ELBO. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.","DOI":"10.1007\/978-3-319-67401-8_2"},{"key":"ref_15","unstructured":"Kim, H., and Mnih, A. (2018, January 10\u201315). Disentangling by factorising. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_16","unstructured":"Chen, R.T., Li, X., Grosse, R.B., and Duvenaud, D.K. (2018, January 3\u20138). Isolating sources of disentanglement in variational autoencoders. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_17","unstructured":"Kingma, D.P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. (2016, January 5\u201310). Improved variational inference with inverse autoregressive flow. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Pelsmaeker, T., and Aziz, W. (2020, January 5\u201310). Effective estimation of deep generative language models. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.646"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhu, Q., Bi, W., Liu, X., Ma, X., Li, X., and Wu, D. (2020, January 5\u201310). A batch normalized inference network keeps the KL vanishing away. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.235"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Shen, D., Qin, C., Wang, C., Zhu, H., Chen, E., and Xiong, H. (2021, January 19\u201327). Regularizing variational autoencoder with diversity and uncertainty awareness. Proceedings of the International Joint Conference on Artificial Intelligence, Montreal, QC, Canada.","DOI":"10.24963\/ijcai.2021\/408"},{"key":"ref_21","unstructured":"Razavi, A., van den Oord, A., Poole, B., and Vinyals, O. (2019, January 6\u20139). Preventing posterior collapse with delta-VAEs. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1162\/tacl_a_00030","article-title":"Generating sentences by editing prototypes","volume":"6","author":"Guu","year":"2018","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_23","unstructured":"Davidson, T.R., Falorsi, L., De Cao, N., Kipf, T., and Tomczak, J.M. (2018, January 6\u20138). Hyperspherical variational autoencoders. Proceedings of the Conference on Uncertainty in Artificial Intelligence, Monterey, CA, USA."},{"key":"ref_24","unstructured":"Van den Oord, A., Vinyals, O., and Kavukcuoglu, K. (2017, January 4\u20139). Neural discrete representation learning. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zhao, T., Lee, K., and Eskenazi, M. (2018, January 15\u201320). Unsupervised discrete sentence representation learning for interpretable neural dialog generation. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.","DOI":"10.18653\/v1\/P18-1101"},{"key":"ref_26","unstructured":"Zhao, S., Song, J., and Ermon, S. (February, January 27). InfoVAE: Balancing learning and inference in variational autoencoders. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_27","unstructured":"Zheng, H., Yao, J., Zhang, Y., Tsang, I.W., and Wang, J. (February, January 27). Understanding VAEs in fisher-shannon plane. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_28","unstructured":"Ma, X., Zhou, C., and Hovy, E. (2019, January 6\u20139). MAE: Mutual posterior-divergence regularization for variational autoEncoders. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Li, Y., Cheng, L., Yin, F., Zhang, M.M., and Theodoridis, S. (2023, January 4\u201310). Overcoming posterior collapse in variational autoencoders via EM-type training. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece.","DOI":"10.1109\/ICASSP49357.2023.10096746"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Li, B., He, J., Neubig, G., Berg-Kirkpatrick, T., and Yang, Y. (2019, January 3\u20137). A surprisingly effective fix for deep latent variable modeling of text. Proceedings of the Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, Hong Kong, China.","DOI":"10.18653\/v1\/D19-1370"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Semeniuta, S., Severyn, A., and Barth, E. (2017, January 9\u201311). A hybrid convolutional variational autoencoder for text generation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.","DOI":"10.18653\/v1\/D17-1066"},{"key":"ref_32","unstructured":"Yang, Z., Hu, Z., Salakhutdinov, R., and Berg-Kirkpatrick, T. (2017, January 6\u201311). Improved variational autoencoders for text modeling using dilated convolutions. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_33","unstructured":"Petit, A., and Corro, C.F. (2021, January 13). Preventing posterior collapse in variational autoencoders for text generation via decoder regularization. Proceedings of the NeurIPS Workshop on Deep Generative Models and Downstream Applications, Online."},{"key":"ref_34","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2016, January 2\u20134). Deep variational information bottleneck. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_35","unstructured":"Yeung, S., Kannan, A., Dauphin, Y., and Feifei, L. (2017). Tackling over-pruning in variational autoencoders. arXiv."},{"key":"ref_36","unstructured":"Burda, Y., Grosse, R., and Salakhutdinov, R. (2016, January 2\u20134). Importance weighted autoencoders. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_37","unstructured":"Shen, T., Lei, T., Barzilay, R., and Jaakkola, T. (2017, January 4\u20139). Style transfer from non-parallel text by cross-alignment. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_38","unstructured":"Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. (2021, January 6\u201314). MAUVE: Measuring the gap between neural text and human text using divergence frontiers. Proceedings of the Advances in Neural Information Processing Systems, Online."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/27\/4\/423\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:14:01Z","timestamp":1760030041000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/27\/4\/423"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,14]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["e27040423"],"URL":"https:\/\/doi.org\/10.3390\/e27040423","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,14]]}}}