{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,7]],"date-time":"2026-02-07T19:08:00Z","timestamp":1770491280894,"version":"3.49.0"},"reference-count":25,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2020,8,27]],"date-time":"2020-08-27T00:00:00Z","timestamp":1598486400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["200021_182063"],"award-info":[{"award-number":["200021_182063"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In this paper, we consider an information bottleneck (IB) framework for semi-supervised classification with several families of priors on latent space representation. We apply a variational decomposition of mutual information terms of IB. Using this decomposition we perform an analysis of several regularizers and practically demonstrate an impact of different components of variational model on the classification accuracy. We propose a new formulation of semi-supervised IB with hand crafted and learnable priors and link it to the previous methods such as semi-supervised versions of VAE (M1 + M2), AAE, CatGAN, etc. We show that the resulting model allows better understand the role of various previously proposed regularizers in semi-supervised classification task in the light of IB framework. The proposed IB semi-supervised model with hand-crafted and learnable priors is experimentally validated on MNIST under different amount of labeled data.<\/jats:p>","DOI":"10.3390\/e22090943","type":"journal-article","created":{"date-parts":[[2020,8,27]],"date-time":"2020-08-27T09:47:02Z","timestamp":1598521622000},"page":"943","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Variational Information Bottleneck for Semi-Supervised Classification"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0416-9674","authenticated-orcid":false,"given":"Slava","family":"Voloshynovskiy","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Geneva, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8537-5204","authenticated-orcid":false,"given":"Olga","family":"Taran","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Geneva, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mouad","family":"Kondah","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Geneva, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taras","family":"Holotyak","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Geneva, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3184-8509","authenticated-orcid":false,"given":"Danilo","family":"Rezende","sequence":"additional","affiliation":[{"name":"DeepMind, London N1C 4AG, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,27]]},"reference":[{"key":"ref_1","unstructured":"Kingma, D.P., Mohamed, S., Rezende, D.J., and Welling, M. (2014). Semi-supervised learning with deep generative models. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_2","unstructured":"Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B. (2015). Adversarial autoencoders. arXiv."},{"key":"ref_3","unstructured":"Springenberg, J.T. (2015). Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv."},{"key":"ref_4","unstructured":"Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. arXiv."},{"key":"ref_5","unstructured":"Federici, M., Dutta, A., Forr\u00e9, P., Kushman, N., and Akata, Z. (2020). Learning Robust Representations via Multi-View Information Bottleneck. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tishby, N., and Zaslavsky, N. (May, January 26). Deep learning and the information bottleneck principle. Proceedings of the 2015 IEEE Information Theory Workshop (ITW), Jerusalem, Israel.","DOI":"10.1109\/ITW.2015.7133169"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2897","DOI":"10.1109\/TPAMI.2017.2784440","article-title":"Information dropout: Learning optimal representations through noisy computation","volume":"40","author":"Achille","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","unstructured":"Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C.A. (2019). Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_9","unstructured":"Grandvalet, Y., and Bengio, Y. (2004). Semi-supervised learning by entropy minimization. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_10","unstructured":"Lee, D.H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML Workshop: Challenges in Representation Learning (WREPL), ICML."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3207","DOI":"10.1162\/NECO_a_00052","article-title":"Deep, big, simple neural nets for handwritten digit recognition","volume":"22","author":"Meier","year":"2010","journal-title":"Neural Comput."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q.V. (2018). Autoaugment: Learning augmentation policies from data. arXiv.","DOI":"10.1109\/CVPR.2019.00020"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2225","DOI":"10.1109\/TPAMI.2019.2909031","article-title":"Learning representations for neural network-based classification using the information bottleneck principle","volume":"42","author":"Amjad","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2016). Deep variational information bottleneck. arXiv."},{"key":"ref_15","unstructured":"Voloshynovskiy, S., Kondah, M., Rezaeifar, S., Taran, O., Hotolyak, T., and Rezende, D.J. (2019). Information bottleneck through variational glasses. NeurIPS Workshop on Bayesian Deep Learning, Vancouver Convention Center."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"U\u011fur, Y., and Zaidi, A. (2020). Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding. Entropy, 22.","DOI":"10.3390\/e22020213"},{"key":"ref_17","unstructured":"Maal\u00f8e, L., S\u00f8nderby, C.K., S\u00f8nderby, S.K., and Winther, O. (2016). Auxiliary deep generative models. arXiv."},{"key":"ref_18","unstructured":"\u015amieja, M., Wo\u0142czyk, M., Tabor, J., and Geiger, B.C. (2019). SeGMA: Semi-Supervised Gaussian Mixture Auto-Encoder. arXiv."},{"key":"ref_19","unstructured":"Makhzani, A., and Frey, B.J. (2017). Pixelgan autoencoders. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_20","unstructured":"Cover, T.M., and Thomas, J.A. (2012). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_21","unstructured":"Kingma, D., and Welling, M. (2014). Auto-Encoding Variational Bayes. arXiv."},{"key":"ref_22","unstructured":"Rezende, D.J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. arXiv."},{"key":"ref_23","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017, January 24\u201326). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_24","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_25","first-page":"5","article-title":"Reading Digits in Natural Images with Unsupervised Feature Learning","volume":"2011","author":"Netzer","year":"2011","journal-title":"NIPS Workshop on Deep Learning and Unsupervised Feature Learning"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/9\/943\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:03:50Z","timestamp":1760177030000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/9\/943"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,27]]},"references-count":25,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["e22090943"],"URL":"https:\/\/doi.org\/10.3390\/e22090943","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,27]]}}}