{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T07:09:57Z","timestamp":1777705797540,"version":"3.51.4"},"reference-count":16,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2021,12,24]],"date-time":"2021-12-24T00:00:00Z","timestamp":1640304000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"published-print":{"date-parts":[[2022,3,31]]},"abstract":"<jats:p>Probabilistic Bayesian methods are widely used in the machine learning domain. Variational Autoencoder (VAE) is a common architecture for solving the Language Modeling task in a self-supervised way. VAE consists of a concept of latent variables inside the model. Latent variables are described as a random variable that is fit by the data. Up to now, in the majority of cases, latent variables are considered normally distributed. The normal distribution is a well-known distribution that can be easily included in any pipeline. Moreover, the normal distribution is a good choice when the Central Limit Theorem (CLT) holds. It makes it effective when one is working with i.i.d. (independent and identically distributed) random variables. However, the conditions of CLT in Natural Language Processing are not easy to check. So, the choice of distribution family is unclear in the domain. This paper studies the priors selection impact of continuous distributions in the Low-Resource Language Modeling task with VAE. The experiment shows that there is a statistical difference between the different priors in the encoder-decoder architecture. We showed that family distribution hyperparameter is important in the Low-Resource Language Modeling task and should be considered for the model training.<\/jats:p>","DOI":"10.3233\/jifs-219243","type":"journal-article","created":{"date-parts":[[2021,12,24]],"date-time":"2021-12-24T10:27:41Z","timestamp":1640341661000},"page":"4541-4549","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Prior latent distribution comparison for the RNN Variational Autoencoder in low-resource language modeling"],"prefix":"10.1177","volume":"42","author":[{"given":"Yevhen","family":"Kostiuk","sequence":"first","affiliation":[{"name":"Taras Shevchenko National University of Kyiv, Kyiv, Ukraine"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mykola","family":"Lukashchuk","sequence":"additional","affiliation":[{"name":"Instituto Polit\u00e9cnico Nacional, Centro de Investigaci\u00f3n en Computaci\u00f3n, Mexico City, Mexico"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander","family":"Gelbukh","sequence":"additional","affiliation":[{"name":"Instituto Polit\u00e9cnico Nacional, Centro de Investigaci\u00f3n en Computaci\u00f3n, Mexico City, Mexico"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Grigori","family":"Sidorov","sequence":"additional","affiliation":[{"name":"Instituto Polit\u00e9cnico Nacional, Centro de Investigaci\u00f3n en Computaci\u00f3n, Mexico City, Mexico"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2021,12,24]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"KingmaD. and WellingM. Auto-Encoding Variational Bayes Presented at the 2nd International Conference on Learning Representations ICLR 2014 Banff AB Canada Conference Track Proceedings. (2014) Apr. 14\u201316."},{"key":"e_1_3_2_3_2","unstructured":"TomczakJ. and WellingM. VAE with aVampPrior in International Conference on Artificial Intelligence and Statistics Mar. (2018) pp. 1214\u20131223."},{"key":"e_1_3_2_4_2","unstructured":"MakhzaniA. and FreyB. k-Sparse Autoencoders in International Conference on Learning Representations (ICLR) Dec. 2014."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","unstructured":"TakahashiH. Iwata TomoharuT. YamanakaY. YamadaM. and YagiS. Variational Autoencoder with Implicit Optimal Priors in AAAI Conference 2012 doi: 10.1609\/aaai.v33i01.33015066.","DOI":"10.1609\/aaai.v33i01.33015066"},{"key":"e_1_3_2_6_2","unstructured":"DevlinJ. ChangM.W. LeeK. and ToutanovaK. BERT: Pretraining of Deep Bidirectional Transformers for Language Understandingm. 2019 arXiv:1810.04805."},{"key":"e_1_3_2_7_2","unstructured":"MikolovT. ChenK. CorradoG. and DeanJ. Efficient Estimation of Word Representations in Vector Space in International Conference on Learning Representations 2013 Scottsdale Arizona arXiv:1301.3781."},{"key":"e_1_3_2_8_2","unstructured":"ZhangT. WuF. KatiyarA. WeinbergerK. and ArtziY. Revisiting Few-sample BERT Fine-tuning 2019 arXiv:2006.05987."},{"key":"e_1_3_2_9_2","doi-asserted-by":"crossref","unstructured":"ZhangZ. HanX. LiuZ. JiangX. SunM. and LiuQ. ERNIE: Enhanced Language Representation with Informative Entities 2019 arXiv:1905.07129.","DOI":"10.18653\/v1\/P19-1139"},{"key":"e_1_3_2_10_2","unstructured":"YangZ. DaiZ. YangY. CarbonellJ. SalakhutdinovR. and LeQ. XLNet: Generalized Autoregressive Pretraining for Language Understanding in Advances in Neural Information Processing Systems eds. H. Wallach H. Larochelle A. Beygelzimer F. Alch\u00e9-Buc E. Fox and R. Garnett (2019) pp. 5753\u20135763.."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00300"},{"key":"e_1_3_2_12_2","unstructured":"LanZ. ChenM. GoodmanS. GimpelK. SharmaP. and SoricutR. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2020) arXiv:1909.11942."},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"BowmanS. VilnisL. VinyalsO. DaiA. JozefowiczR. and BengioS. Generating Sentences from a Continuous Space (2016) arXiv:1511.06349.","DOI":"10.18653\/v1\/K16-1002"},{"key":"e_1_3_2_14_2","unstructured":"MaoJ. XuW. YangY. WangJ. HuangZ. and YuilleA. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2015) arXiv:1412.6632."},{"key":"e_1_3_2_15_2","unstructured":"Bra\u017einskasA. HavrylovS. and TitovI. Embedding Words as Distributions with a Bayesian Skip-gram Model in Proceedings of the 27th International Conference on Computational Linguistics Aug. (2015) pp. 1775\u20131789 https:\/\/www.aclweb.org\/anthology\/C18-1151."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2013.06.018"},{"key":"e_1_3_2_17_2","unstructured":"ChyzakF. and NielsenF. A closed-form formula for the Kullback-Leibler divergence between Cauchy distributions (2019) arXiv:1905.10965."}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-219243","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JIFS-219243","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-219243","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:45:15Z","timestamp":1777455915000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JIFS-219243"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,24]]},"references-count":16,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,3,31]]}},"alternative-id":["10.3233\/JIFS-219243"],"URL":"https:\/\/doi.org\/10.3233\/jifs-219243","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,24]]}}}