{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T14:23:13Z","timestamp":1775830993473,"version":"3.50.1"},"reference-count":53,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,9,29]],"date-time":"2021-09-29T00:00:00Z","timestamp":1632873600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2018YFC1604000"],"award-info":[{"award-number":["2018YFC1604000"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The Variational AutoEncoder (VAE) has made significant progress in text generation, but it focused on short text (always a sentence). Long texts consist of multiple sentences. There is a particular relationship between each sentence, especially between the latent variables that control the generation of the sentences. The relationships between these latent variables help in generating continuous and logically connected long texts. There exist very few studies on the relationships between these latent variables. We proposed a method for combining the Transformer-Based Hierarchical Variational AutoEncoder and Hidden Markov Model (HT-HVAE) to learn multiple hierarchical latent variables and their relationships. This application improves long text generation. We use a hierarchical Transformer encoder to encode the long texts in order to obtain better hierarchical information of the long text. HT-HVAE\u2019s generation network uses HMM to learn the relationship between latent variables. We also proposed a method for calculating the perplexity for the multiple hierarchical latent variable structure. The experimental results show that our model is more effective in the dataset with strong logic, alleviates the notorious posterior collapse problem, and generates more continuous and logically connected long text.<\/jats:p>","DOI":"10.3390\/e23101277","type":"journal-article","created":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T00:03:42Z","timestamp":1632960222000},"page":"1277","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["A Transformer-Based Hierarchical Variational AutoEncoder Combined Hidden Markov Model for Long Text Generation"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2293-9209","authenticated-orcid":false,"given":"Kun","family":"Zhao","sequence":"first","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0851-1994","authenticated-orcid":false,"given":"Hongwei","family":"Ding","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8728-1719","authenticated-orcid":false,"given":"Kai","family":"Ye","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6079-009X","authenticated-orcid":false,"given":"Xiaohui","family":"Cui","sequence":"additional","affiliation":[{"name":"Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1007\/s00521-016-2401-x","article-title":"Text classification based on deep belief network and softmax regression","volume":"29","author":"Jiang","year":"2016","journal-title":"Neural Comput. Appl."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kowsari, K., Brown, D.E., Heidarysafa, M., Meimandi, K., Gerber, M., and Barnes, L.E. (2017, January 18\u201321). HDLTex: Hierarchical Deep Learning for Text Classification. Proceedings of the 16th IEEE International Conference on Machine Learning and Applications (ICMLA), Cancun, Mexico.","DOI":"10.1109\/ICMLA.2017.0-134"},{"key":"ref_3","unstructured":"Kowsari, K., Heidarysafa, M., Brown, D.E., Meimandi, K., and Barnes, L.E. (2018, January 9\u201311). RMDL: Random Multimodel Deep Learning for Classification. Proceedings of the ICISDM \u201918, Lakeland, FL, USA."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lai, S., Xu, L., Liu, K., and Zhao, J. (2015, January 25\u201330). Recurrent Convolutional Neural Networks for Text Classification. Proceedings of the AAAI 2015, Austin, TX, USA.","DOI":"10.1609\/aaai.v29i1.9513"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wang, Z., Wang, X., An, B., Yu, D., and Chen, C. (2020). Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.101"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Wang, G., Li, C., Gan, Z., Brockett, C., and Dolan, B. (2020). POINTER: Constrained Text Generation via Insertion-based Generative Pre-training. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.698"},{"key":"ref_7","unstructured":"Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. (2020). Plug and Play Language Models: A Simple Approach to Controlled Text Generation. arXiv."},{"key":"ref_8","unstructured":"Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2021, September 02). Language Models are Unsupervised Multitask Learners. Available online: https:\/\/cdn.openai.com\/better-language-models\/language_models_are_unsupervised_multitask_learners.pdf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1162\/tacl_a_00297","article-title":"AMR-To-Text Generation with Graph Transformer","volume":"8","author":"Wang","year":"2020","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_10","unstructured":"Koncel-Kedziorski, R., Bekal, D., Luan, Y., Lapata, M., and Hajishirzi, H. (2019, January 6\u20137). Text Generation from Knowledge Graphs with Graph Transformers. Proceedings of the NAACL-HLT 2019, Minneapolis, MN, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.datak.2017.06.001","article-title":"Learning multiple layers of knowledge representation for aspect based sentiment analysis","volume":"114","author":"Pham","year":"2018","journal-title":"Data Knowl. Eng."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Tao, J., Zhou, L., and Feeney, C. (2019, January 8\u201311). I Understand What You Are Saying: Leveraging Deep Learning Techniques for Aspect Based Sentiment Analysis. Proceedings of the HICSS 2019, Maui, HI, USA.","DOI":"10.24251\/HICSS.2019.057"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1083","DOI":"10.1016\/j.eswa.2014.08.036","article-title":"A multi-label classification based approach for sentiment classification","volume":"42","author":"Liu","year":"2015","journal-title":"Expert Syst. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1016\/j.knosys.2016.07.019","article-title":"Learning word dependencies in text by means of a deep recurrent belief network","volume":"108","author":"Chaturvedi","year":"2016","journal-title":"Knowl. Based Syst."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kim, J., Jang, S., Choi, S., and Park, E.L. (2020). Text Classification using Capsules. arXiv.","DOI":"10.1016\/j.neucom.2019.10.033"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Aly, R., Remus, S., and Biemann, C. (August, January 28). Hierarchical Multi-label Classification of Text with Capsule Networks. Proceedings of the ACL 2019, Florence, Italy.","DOI":"10.18653\/v1\/P19-2045"},{"key":"ref_17","unstructured":"Ren, H., and Lu, H. (2018). Compositional coding capsule network with k-means routing for text classification. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep Learning","volume":"521","author":"Goodfellow","year":"2015","journal-title":"Nature"},{"key":"ref_19","unstructured":"Sutskever, I., Vinyals, O., and Le, Q.V. (2014, January 8\u201311). Sequence to Sequence Learning with Neural Networks. Proceedings of the NIPS 2014, Montreal, QC, Canada."},{"key":"ref_20","unstructured":"Kingma, D.P., and Welling, M. (2021, September 02). Auto-Encoding Variational Bayes. Available online: https:\/\/arxiv.org\/abs\/1312.6114."},{"key":"ref_21","unstructured":"Rezende, D.J., Mohamed, S., and Wierstra, D. (2014, January 21\u201326). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the ICML 2014, Beijing, China."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., J\u00f3zefowicz, R., and Bengio, S. (2016, January 11\u201312). Generating Sentences from a Continuous Space. Proceedings of the CoNLL 2016, Berlin, Germany.","DOI":"10.18653\/v1\/K16-1002"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_24","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is All you Need. arXiv."},{"key":"ref_25","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 6\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the NAACL-HLT 2019, Minneapolis, MN, USA."},{"key":"ref_26","unstructured":"Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H. (2019). Unified Language Model Pre-training for Natural Language Understanding and Generation. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"ref_28","first-page":"140:1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shen, D., \u00c7elikyilmaz, A., Zhang, Y., Chen, L., Wang, X.E., Gao, J., and Carin, L. (2019). Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models. arXiv.","DOI":"10.18653\/v1\/P19-1200"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, W., Gan, Z., Xu, H., Zhang, R., Wang, G., Shen, D., Chen, C., and Carin, L. (2019). Topic-Guided Variational Autoencoders for Text Generation. arXiv.","DOI":"10.18653\/v1\/N19-1015"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhang, X., Yang, Y., Yuan, S., Shen, D., and Carin, L. (2019). Syntax-Infused Variational Autoencoder for Text Generation. arXiv.","DOI":"10.18653\/v1\/P19-1199"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Liu, D., and Liu, G. (2019, January 14\u201319). A Transformer-Based Variational Autoencoder for Sentence Generation. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8852155"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Li, C., Gao, X., Li, Y., Li, X., Peng, B., Zhang, Y., and Gao, J. (2020). Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.378"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, T., and Wan, X. (2019, January 10\u201316). T-CVAE: Transformer-Based Conditioned Variational Autoencoder for Story Completion. Proceedings of the IJCAI 2019, Macao, China.","DOI":"10.24963\/ijcai.2019\/727"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merrienboer, B., G\u00fcl\u00e7ehre, \u00c7., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_36","unstructured":"Burda, Y., Grosse, R.B., and Salakhutdinov, R. (2021, September 02). Importance Weighted Autoencoders. Available online: https:\/\/arxiv.org\/abs\/1509.00519."},{"key":"ref_37","unstructured":"Miao, Y., Yu, L., and Blunsom, P. (2016). Neural Variational Inference for Text Processing. arXiv."},{"key":"ref_38","unstructured":"Yang, Z., Hu, Z., Salakhutdinov, R., and Berg-Kirkpatrick, T. (2017). Improved Variational Autoencoders for Text Modeling using Dilated Convolutions. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Semeniuta, S., Severyn, A., and Barth, E. (2017). A Hybrid Convolutional Variational Autoencoder for Text Generation. arXiv.","DOI":"10.18653\/v1\/D17-1066"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Serban, I., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A.C., and Bengio, Y. (2017, January 4\u20139). A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. Proceedings of the AAAI 2017, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10983"},{"key":"ref_41","unstructured":"Kim, Y., Wiseman, S., Miller, A., Sontag, D., and Rush, A.M. (2018, January 10\u201315). Semi-Amortized Variational Autoencoders. Proceedings of the ICML 2018, Stockholm, Sweden."},{"key":"ref_42","unstructured":"Kaiser, \u0141., Roy, A., Vaswani, A., Parmar, N., Bengio, S., Uszkoreit, J., and Shazeer, N.M. (2018, January 10\u201315). Fast Decoding in Sequence Models using Discrete Latent Variables. Proceedings of the ICML 2018, Stockholm, Sweden."},{"key":"ref_43","unstructured":"Hashimoto, T., Guu, K., Oren, Y., and Liang, P. (2018, January 3\u20138). A Retrieve-and-Edit Framework for Predicting Structured Outputs. Proceedings of the NeurIPS 2018, Montreal, QC, Canada."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zhao, T., Zhao, R., and Esk\u00e9nazi, M. (2017, January 6\u201315). Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders. Proceedings of the ACL 2017, Austin, TX, USA.","DOI":"10.18653\/v1\/P17-1061"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Li, B., He, J., Neubig, G., Berg-Kirkpatrick, T., and Yang, Y. (2019, January 3\u20137). A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text. Proceedings of the EMNLP\/IJCNLP 2019, Hong Kong, China.","DOI":"10.18653\/v1\/D19-1370"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Fu, H., Li, C., Liu, X., Gao, J., \u00c7elikyilmaz, A., and Carin, L. (2019, January 2\u20137). Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing. Proceedings of the NAACL 2019, Minneapolis, MN, USA.","DOI":"10.18653\/v1\/N19-1021"},{"key":"ref_47","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C.P., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017, January 24\u201326). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. Proceedings of the ICLR 2017, Toulon, France."},{"key":"ref_48","unstructured":"Dieng, A.B., Kim, Y., Rush, A.M., and Blei, D. (2019). Avoiding Latent Variable Collapse With Generative Skip Models. arXiv."},{"key":"ref_49","unstructured":"He, J., Spokoyny, D., Neubig, G., and Berg-Kirkpatrick, T. (2019). Lagging Inference Networks and Posterior Collapse in Variational Autoencoders. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Fang, L., Li, C., Gao, J., Dong, W.J., and Chen, C. (2019, January 3\u20137). Implicit Deep Latent Variable Models for Text Generation. Proceedings of the EMNLP\/IJCNLP 2019, Hong Kong, China.","DOI":"10.18653\/v1\/D19-1407"},{"key":"ref_51","unstructured":"Zhao, J., Kim, Y., Zhang, K., Rush, A.M., and LeCun, Y. (2018, January 10\u201315). Adversarially Regularized Autoencoders. Proceedings of the ICML 2018, Stockholm, Sweden."},{"key":"ref_52","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6\u201312). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the ACL 2002, Philadelphia, PA, USA."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. (2018, January 8\u201312). Texygen: A Benchmarking Platform for Text Generation Models. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.","DOI":"10.1145\/3209978.3210080"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/10\/1277\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:07:29Z","timestamp":1760166449000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/10\/1277"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,29]]},"references-count":53,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["e23101277"],"URL":"https:\/\/doi.org\/10.3390\/e23101277","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,29]]}}}