{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T03:19:46Z","timestamp":1792034386428,"version":"4.4.0"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2021,7,7]],"date-time":"2021-07-07T00:00:00Z","timestamp":1625616000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,7]],"date-time":"2021-07-07T00:00:00Z","timestamp":1625616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"crossref","award":["JP20J11448"],"award-info":[{"award-number":["JP20J11448"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Sequential variational autoencoders (VAEs) with a global latent variable <jats:italic>z<\/jats:italic> have been studied for disentangling the global features of data, which is useful for several downstream tasks. To further assist the sequential VAEs in obtaining meaningful <jats:italic>z<\/jats:italic>, existing approaches introduce a regularization term that maximizes the mutual information (MI) between the observation and <jats:italic>z<\/jats:italic>. However, by analyzing the sequential VAEs from the information-theoretic perspective, we claim that simply maximizing the MI encourages the latent variable to have redundant information, thereby preventing the disentanglement of global features. Based on this analysis, we derive a novel regularization method that makes <jats:italic>z<\/jats:italic> informative while encouraging disentanglement. Specifically, the proposed method removes redundant information by minimizing the MI between <jats:italic>z<\/jats:italic> and the local features by using adversarial training. In the experiments, we trained two sequential VAEs, state-space and autoregressive model variants, using speech and image datasets. The results indicate that the proposed method improves the performance of downstream classification and data generation tasks, thereby supporting our information-theoretic perspective for the learning of global features.<\/jats:p>","DOI":"10.1007\/s10994-021-06032-4","type":"journal-article","created":{"date-parts":[[2021,7,7]],"date-time":"2021-07-07T18:01:51Z","timestamp":1625680911000},"page":"2239-2266","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Information-theoretic regularization for learning global features by sequential VAE"],"prefix":"10.1007","volume":"110","author":[{"given":"Kei","family":"Akuzawa","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yusuke","family":"Iwasawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yutaka","family":"Matsuo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,7,7]]},"reference":[{"key":"6032_CR1","unstructured":"Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R. A., & Murphy, K. (2018) Fixing a broken elbo. In International conference on machine learning (pp. 159\u2013168)."},{"key":"6032_CR2","doi-asserted-by":"publisher","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","volume":"35","author":"Y Bengio","year":"2013","unstructured":"Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 1798\u20131828. https:\/\/doi.org\/10.1109\/TPAMI.2013.50","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"6032_CR3","doi-asserted-by":"crossref","unstructured":"Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL conference on computational natural language learning (pp. 10\u201321).","DOI":"10.18653\/v1\/K16-1002"},{"key":"6032_CR4","first-page":"2172","volume-title":"Advances in neural information processing systems","author":"X Chen","year":"2016","unstructured":"Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 29, pp. 2172\u20132180). Red Hook: Curran Associates Inc."},{"key":"6032_CR5","unstructured":"Chen, X., Kingma, D. P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., et al. (2017). Variational lossy autoencoder. In Proceedings of the 5th international conference on learning representations."},{"key":"6032_CR6","first-page":"1531","volume":"5","author":"F Fleuret","year":"2004","unstructured":"Fleuret, F. (2004). Fast binary feature selection with conditional mutual information. Journal of Machine Learning Research, 5, 1531\u20131555.","journal-title":"Journal of Machine Learning Research"},{"key":"6032_CR7","unstructured":"Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., et al. (2016). Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59), 1\u201335."},{"key":"6032_CR8","volume-title":"Timit acoustic-phonetic continuous speech corpus","author":"J Garofolo","year":"1992","unstructured":"Garofolo, J., Lamel, L., Fisher, W., Fiscus, J., Pallett, D., Dahlgren, N., & Zue, V. (1992). Timit acoustic-phonetic continuous speech corpus. Philadelphia: Linguistic Data Consortium."},{"key":"6032_CR9","first-page":"13475","volume":"32","author":"K Gregor","year":"2019","unstructured":"Gregor, K., Rezende, D. J., Besse, F., Wu, Y., Merzic, H., & van den Oord, A. (2019). Shaping belief states with generative environment models for RL. Advances in Neural Information Processing Systems, 32, 13475\u201313487.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6032_CR10","unstructured":"Gulrajani, I., Kumar, K., Ahmed, F., Taiga, A. A., Visin, F., Vazquez, D., et al. (2017). Pixelvae: A latent variable model for natural images. In 5th international conference on learning representations."},{"key":"6032_CR11","unstructured":"He, J., Spokoyny, D., Neubig, G., & Berg-Kirkpatrick, T. (2019). Lagging inference networks and posterior collapse in variational autoencoders. In International conference on learning representations."},{"key":"6032_CR12","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., et al. (2017). beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations."},{"key":"6032_CR13","first-page":"517","volume-title":"Advances in neural information processing systems","author":"JT Hsieh","year":"2018","unstructured":"Hsieh, J. T., Liu, B., Huang, D. A., Fei-Fei, L. F., & Niebles, J. C. (2018). Learning to decompose and disentangle representations for video prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 31, pp. 517\u2013526). Red Hook: Curran Associates Inc."},{"key":"6032_CR14","first-page":"1878","volume":"30","author":"WN Hsu","year":"2017","unstructured":"Hsu, W. N., Zhang, Y., & Glass, J. (2017). Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in Neural Information Processing Systems, 30, 1878\u20131889.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6032_CR15","unstructured":"Jaiswal, A., Brekelmans, R., Moyer, D., Steeg, G. V., AbdAlmageed, W., & Natarajan, P. (2019) Discovery and separation of features for invariant representation learning. CoRR abs\/1912.00646, arxiv:1912.00646"},{"key":"6032_CR16","unstructured":"Kim, H., & Mnih, A. (2018). Disentangling by factorising. In Proceedings of the 35th international conference on machine learning (ICML)."},{"key":"6032_CR17","first-page":"4743","volume-title":"Advances in neural information processing systems","author":"DP Kingma","year":"2016","unstructured":"Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., & Welling, M. (2016). Improved variational inference with inverse autoregressive flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 29, pp. 4743\u20134751). Red Hook: Curran Associates Inc."},{"key":"6032_CR18","first-page":"2539","volume-title":"Advances in neural information processing systems","author":"TD Kulkarni","year":"2015","unstructured":"Kulkarni, T. D., Whitney, W. F., Kohli, P., & Tenenbaum, J. (2015). Deep convolutional inverse graphics network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 28, pp. 2539\u20132547). Red Hook: Curran Associates Inc."},{"key":"6032_CR19","first-page":"2590","volume-title":"Advances in neural information processing systems","author":"AH Liu","year":"2018","unstructured":"Liu, A. H., Liu, Y. C., Yeh, Y. Y., & Wang, Y. C. F. (2018). A unified feature disentangler for multi-domain image translation and manipulation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 31, pp. 2590\u20132599). Red Hook: Curran Associates Inc."},{"key":"6032_CR20","unstructured":"Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch\u00f6lkopf, B., et al. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th international conference on machine learning (Vol.\u00a097, pp. 4114\u20134124)."},{"key":"6032_CR21","unstructured":"Lucas, J., Tucker, G., Grosse, R. B., & Norouzi, M. (2019). Don\u2019t blame the elbo! A linear VAE perspective on posterior collapse. Advances in Neural Information Processing Systems, 32, 9403\u20139413."},{"key":"6032_CR22","doi-asserted-by":"crossref","unstructured":"Lucas, T., Verbeek, J. (2018). Auxiliary guided autoregressive variational autoencoders. In Machine learning and knowledge discovery in databases\u2014European conference, ECML PKDD (pp. 443\u2013458).","DOI":"10.1007\/978-3-030-10925-7_27"},{"key":"6032_CR23","unstructured":"Ma, X., Zhou, C., & Hovy, E. (2019). MAE: Mutual posterior-divergence regularization for variational autoencoders. In International conference on learning representations."},{"key":"6032_CR24","first-page":"1975","volume":"30","author":"A Makhzani","year":"2017","unstructured":"Makhzani, A., & Frey, B. J. (2017). Pixelgan autoencoders. Advances in Neural Information Processing Systems, 30, 1975\u20131985.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6032_CR25","doi-asserted-by":"crossref","unstructured":"Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. In International conference on learning representations. https:\/\/openreview.net\/forum?id=B1QRgziT-.","DOI":"10.1007\/978-3-030-03243-2_860-1"},{"key":"6032_CR26","unstructured":"Mohamed, S., & Lakshminarayanan, B. (2017). Learning in implicit generative models. CoRR abs\/1610.03483, arxiv:1610.03483."},{"key":"6032_CR27","first-page":"9084","volume":"31","author":"D Moyer","year":"2018","unstructured":"Moyer, D., Gao, S., Brekelmans, R., Galstyan, A., & Ver Steeg, G. (2018). Invariant representations without adversarial training. Advances in Neural Information Processing Systems, 31, 9084\u20139093.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6032_CR28","unstructured":"Mukherjee, S., Asnani, H., & Kannan, S. (2019). CCMI: Classifier based conditional mutual information estimation. In International conference on uncertainity in artificial intelligence."},{"key":"6032_CR29","first-page":"1089","volume":"20","author":"X Nguyen","year":"2008","unstructured":"Nguyen, X., Wainwright, M. J., & Jordan, M. I. (2008). Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. Advances in Neural Information Processing Systems, 20, 1089\u20131096.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6032_CR30","unstructured":"Nie, W., Karras, T., Garg, A., Debhath, S., Patney, A., Patel, A. B., et al. (2020) Semi-supervised stylegan for disentanglement learning. CoRR abs\/2003.03461, arxiv:2003.03461."},{"key":"6032_CR32","unstructured":"Park, Y., Kim, C., & Kim, G. (2019). Variational Laplace autoencoders. In Proceedings of the 36th international conference on machine learning, PMLR, proceedings of machine learning research."},{"key":"6032_CR33","unstructured":"Poole, B., Ozair, S., Van Den\u00a0Oord, A., Alemi, A., & Tucker, G. (2019). On variational bounds of mutual information. In Proceedings of the 36th international conference on machine learning (pp. 5171\u20135180)."},{"key":"6032_CR34","first-page":"12268","volume":"32","author":"S Ravuri","year":"2019","unstructured":"Ravuri, S., & Vinyals, O. (2019). Classification accuracy score for conditional generative models. Advances in Neural Information Processing Systems, 32, 12268\u201312279.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6032_CR35","unstructured":"Razavi, A., van\u00a0den Oord, A., Poole, B., & Vinyals, O. (2019). Preventing posterior collapse with delta-VAEs. In International conference on learning representations."},{"key":"6032_CR36","unstructured":"Shu, R., Chen, Y., Kumar, A., Ermon, S., & Poole, B. (2020). Weakly supervised disentanglement with guarantees. In International conference on learning representations."},{"issue":"5","key":"6032_CR37","doi-asserted-by":"publisher","first-page":"1009","DOI":"10.1007\/s10463-011-0343-8","volume":"64","author":"M Sugiyama","year":"2012","unstructured":"Sugiyama, M., Suzuki, T., & Kanamori, T. (2012). Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64(5), 1009\u20131044.","journal-title":"Annals of the Institute of Statistical Mathematics"},{"key":"6032_CR31","unstructured":"van\u00a0den Oord, A., Li, Y., & Vinyals, O. (2019). Representation learning with contrastive predictive coding. arxiv:1807.03748."},{"key":"6032_CR38","unstructured":"Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. CoRR abs\/1708.07747, arxiv:1708.07747."},{"key":"6032_CR39","first-page":"585","volume-title":"Advances in neural information processing systems","author":"Q Xie","year":"2017","unstructured":"Xie, Q., Dai, Z., Du, Y., Hovy, E., & Neubig, G. (2017). Controllable invariance through adversarial feature learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 30, pp. 585\u2013596). Red Hook: Curran Associates Inc."},{"key":"6032_CR40","unstructured":"Yingzhen, L., & Mandt, S. (2018). Disentangled sequential autoencoder. In Proceedings of the 35th international conference on machine learning (pp. 5670\u20135679)."},{"key":"6032_CR41","doi-asserted-by":"crossref","unstructured":"Zhao, S., Song, J., & Ermon, S. (2019). Infovae: Balancing learning and inference in variational autoencoders. In The thirty-third AAAI conference on artificial intelligence (pp. 5885\u20135892).","DOI":"10.1609\/aaai.v33i01.33015885"},{"key":"6032_CR42","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Min, M. R., Kadav, A., & Graf, H. P. (2020). S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR).","DOI":"10.1109\/CVPR42600.2020.00657"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06032-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-06032-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06032-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,8,17]],"date-time":"2021-08-17T16:16:37Z","timestamp":1629216997000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-06032-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,7]]},"references-count":42,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2021,8]]}},"alternative-id":["6032"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-06032-4","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,7]]},"assertion":[{"value":"31 January 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 May 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 June 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 July 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}