{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T02:35:22Z","timestamp":1775097322222,"version":"3.50.1"},"reference-count":22,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,1,22]],"date-time":"2020-01-22T00:00:00Z","timestamp":1579651200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Combining the information bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proven successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the deep variational information bottleneck and the assumptions needed for its derivation. The two assumed properties of the data, X and Y, and their latent representation T, take the form of two Markov chains     T \u2212 X \u2212 Y     and     X \u2212 T \u2212 Y    . Requiring both to hold during the optimisation process can be limiting for the set of potential joint distributions     P ( X , Y , T )    . We, therefore, show how to circumvent this limitation by optimising a lower bound for the mutual information between T and Y:     I ( T ; Y )    , for which only the latter Markov chain has to be satisfied. The mutual information     I ( T ; Y )     can be split into two non-negative parts. The first part is the lower bound for     I ( T ; Y )    , which is optimised in deep variational information bottleneck (DVIB) and cognate models in practice. The second part consists of two terms that measure how much the former requirement     T \u2212 X \u2212 Y     is violated. Finally, we propose interpreting the family of information bottleneck models as directed graphical models, and show that in this framework, the original and deep information bottlenecks are special cases of a fundamental IB model.<\/jats:p>","DOI":"10.3390\/e22020131","type":"journal-article","created":{"date-parts":[[2020,1,22]],"date-time":"2020-01-22T11:17:57Z","timestamp":1579691877000},"page":"131","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["On the Difference between the Information Bottleneck and the Deep Information Bottleneck"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1565-1781","authenticated-orcid":false,"given":"Aleksander","family":"Wieczorek","sequence":"first","affiliation":[{"name":"Department of Mathematics and Computer Science, University of Basel, CH-4051 Basel, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Volker","family":"Roth","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, University of Basel, CH-4051 Basel, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,22]]},"reference":[{"key":"ref_1","unstructured":"Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems 27, Curran Associates, Inc."},{"key":"ref_2","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv."},{"key":"ref_3","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2016). Deep variational information bottleneck. arXiv."},{"key":"ref_4","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (2000). The information bottleneck method. arXiv."},{"key":"ref_5","first-page":"165","article-title":"Information bottleneck for Gaussian variables","volume":"6","author":"Chechik","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_6","unstructured":"Rey, M., Roth, V., and Fuchs, T. (2014, January 21\u201326). Sparse meta-Gaussian information bottleneck. Proceedings of the 31st International Conference on Machine Learning (ICML-14), Beijing, China."},{"key":"ref_7","unstructured":"Wieczorek, A., Wieser, M., Murezzan, D., and Roth, V. (May, January 30). Learning Sparse Latent Representations with the Deep Copula Information Bottleneck. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Tishby, N., and Zaslavsky, N. (May, January 26). Deep learning and the information bottleneck principle. Proceedings of the 2015 IEEE Information Theory Workshop, ITW, Jerusalem, Israel.","DOI":"10.1109\/ITW.2015.7133169"},{"key":"ref_9","unstructured":"Shwartz-Ziv, R., and Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"124020","DOI":"10.1088\/1742-5468\/ab3985","article-title":"On the information bottleneck theory of deep learning","volume":"2019","author":"Saxe","year":"2019","journal-title":"J. Stat. Mech.-Theory Exp."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"964","DOI":"10.1109\/TIT.2007.915715","article-title":"Lautum information","volume":"54","author":"Palomar","year":"2008","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_12","unstructured":"Alemi, A.A., Poole, B., Fischer, I., Dillon, J.V., Saurous, R.A., and Murphy, K. (2017). Fixing a broken ELBO. arXiv."},{"key":"ref_13","unstructured":"Alemi, A.A., and Fischer, I. (2018). GILBO: One metric to measure them all. arXiv."},{"key":"ref_14","first-page":"7908","article-title":"Gaussian lower bound for the information bottleneck limit","volume":"18","author":"Painsky","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Hotelling, H. (1992). Relations between two sets of variates. Breakthroughs in Statistics, Springer.","DOI":"10.1007\/978-1-4612-4380-9_14"},{"key":"ref_16","unstructured":"Zhao, S., Song, J., and Ermon, S. (February, January 27). Infovae: Balancing learning and inference in variational autoencoders. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_17","unstructured":"Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2897","DOI":"10.1109\/TPAMI.2017.2784440","article-title":"Information dropout: Learning optimal representations through noisy computation","volume":"40","author":"Achille","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","first-page":"1947","article-title":"Emergence of invariance and disentanglement in deep representations","volume":"19","author":"Achille","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lauritzen, S.L. (1996). Graphical Models, Clarendon Press.","DOI":"10.1093\/oso\/9780198522195.001.0001"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Pearl, J. (2009). Causality, Cambridge University Press.","DOI":"10.1017\/CBO9780511803161"},{"key":"ref_22","unstructured":"Bach, F.R., and Jordan, M.I. (2005). A Probabilistic Interpretation of Canonical Correlation Analysis, University of California."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/131\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:30:02Z","timestamp":1760362202000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/131"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,22]]},"references-count":22,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["e22020131"],"URL":"https:\/\/doi.org\/10.3390\/e22020131","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,22]]}}}