{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T19:22:36Z","timestamp":1792696956720,"version":"4.5.0"},"reference-count":39,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,7,29]],"date-time":"2021-07-29T00:00:00Z","timestamp":1627516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP16K00228"],"award-info":[{"award-number":["JP16K00228"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP18KK0308"],"award-info":[{"award-number":["JP18KK0308"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Task-nuisance decomposition describes why the information bottleneck loss I(z;x)\u2212\u03b2I(z;y) is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, I(z;n) can be decreased by reducing I(z;x) since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information I(z;x|y) provides an alternative upper bound for I(z;n). This bound is applicable even if z is not a sufficient representation of x, that is, I(z;y)\u2260I(x;y). We used mutual information neural estimation (MINE) to estimate I(z;x|y). Experiments demonstrated that I(z;x|y) is smaller than I(z;x) for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when I(z;x|y) is used instead of I(z;x).<\/jats:p>","DOI":"10.3390\/e23080974","type":"journal-article","created":{"date-parts":[[2021,7,29]],"date-time":"2021-07-29T10:47:46Z","timestamp":1627555666000},"page":"974","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Information Bottleneck Analysis by a Conditional Mutual Information Bound"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5628-9961","authenticated-orcid":false,"given":"Taro","family":"Tezuka","sequence":"first","affiliation":[{"name":"Faculty of Library, Information and Media Science, University of Tsukuba, Tsukuba, Ibaraki 305-8577, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shizuma","family":"Namekawa","sequence":"additional","affiliation":[{"name":"Graduate School of Library, Information and Media Studies, University of Tsukuba, Tsukuba, Ibaraki 305-8577, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2897","DOI":"10.1109\/TPAMI.2017.2784440","article-title":"Information dropout: Learning optimal representations through noisy computation","volume":"40","author":"Achille","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gabri\u00e9, M., Manoel, A., Luneau, C., Barbier, J., Macris, N., Krzakala, F., and Zdeborov\u00e1, L. (2018). Entropy and mutual information in models of deep neural networks. Advances in Neural Information Processing Systems, The MIT Press.","DOI":"10.1088\/1742-5468\/ab3430"},{"key":"ref_3","unstructured":"Hjelm, R.D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. (2019, January 6\u20139). Learning deep representations by mutual information estimation and maximization. Proceedings of the 2019 International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_4","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2019, January 6\u20139). Deep Variational Information Bottleneck. Proceedings of the 2019 International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.neunet.2019.05.003","article-title":"Understanding autoencoders with information theoretic concepts","volume":"117","author":"Yu","year":"2019","journal-title":"Neural Netw."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1109\/TNNLS.2020.2968509","article-title":"Understanding convolutional neural networks with information theory: An initial exploration","volume":"32","author":"Yu","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_7","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the 37th annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA."},{"key":"ref_8","first-page":"1","article-title":"Emergence of Invariance and Disentanglement in Deep Representations","volume":"18","author":"Achille","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_9","unstructured":"Slonim, N., and Tishby, N. (December, January 29). Agglomerative information bottleneck. Proceedings of the 12th International Conference on Neural Information Processing Systems, Denver, CO, USA."},{"key":"ref_10","first-page":"165","article-title":"Information Bottleneck for Gaussian Variables","volume":"6","author":"Chechik","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Harremo\u00ebs, P., and Tishby, N. (2007, January 24\u201329). The information bottleneck revisited or how to choose a good distortion measure. Proceedings of the 2007 IEEE International Symposium on Information Theory, Nice, France.","DOI":"10.1109\/ISIT.2007.4557285"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2696","DOI":"10.1016\/j.tcs.2010.04.006","article-title":"Learning and generalization with the information bottleneck","volume":"411","author":"Shamir","year":"2010","journal-title":"Theor. Comput. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tishby, N., and Zaslavsky, N. (2015, January 11\u201315). Deep learning and the information bottleneck principle. Proceedings of the IEEE Information Theory Workshop (ITW), Jeju Island, Korea.","DOI":"10.1109\/ITW.2015.7133169"},{"key":"ref_14","unstructured":"Schwartz-Ziv, R., and Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv."},{"key":"ref_15","unstructured":"Balda, E.R., Behboodi, A., and Mathar, R. (2018). On the Trajectory of Stochastic Gradient Descent in the Information Plane. arXiv."},{"key":"ref_16","unstructured":"Goldfeld, Z., van den Berg, E., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., and Polyanskiy, Y. (2018). Estimating Information Flow in Deep Neural Networks. arXiv."},{"key":"ref_17","unstructured":"Chelombiev, I., Houghton, C., and O\u2019Donnell, C. (2019, January 6\u20139). Adaptive estimators show information compression in deep neural networks. Proceedings  of the 2019 International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_18","unstructured":"Achille, A., Paolini, G., and Soatto, S. (2019). Where is the Information in a Deep Neural Network?. arXiv."},{"key":"ref_19","unstructured":"Darlow, L.N., and Storkey, A. (2020). What Information Does a ResNet Compress?. arXiv."},{"key":"ref_20","unstructured":"Geiger, B.C. (2020). On Information Plane Analyses of Neural Network Classifiers\u2014A Review. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wieczorek, A., and Roth, V. (2020). On the Difference between the Information Bottleneck and the Deep Information Bottleneck. Entropy, 22.","DOI":"10.3390\/e22020131"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Goldfeld, Z., and Polyanskiy, Y. (2020). The Information Bottleneck Problem and Its Applications in Machine Learning. arXiv.","DOI":"10.1109\/JSAIT.2020.2991561"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2225","DOI":"10.1109\/TPAMI.2019.2909031","article-title":"Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle","volume":"42","author":"Amjad","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Fischer, I. (2020). The conditional entropy bottleneck. Entropy, 22.","DOI":"10.3390\/e22090999"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Geiger, B.C., and Fischer, I.S. (2020). A comparison of variational bounds for the information bottleneck functional. Entropy, 22.","DOI":"10.3390\/e22111229"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Yu, X., Yu, S., and Princ\u00edpe, J.C. (2021, January 6\u201311). Deep Deterministic Information Bottleneck with Matrix-Based Entropy Functional. Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.","DOI":"10.1109\/ICASSP39728.2021.9414151"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1109\/TIT.2014.2370058","article-title":"Measures of entropy from data using infinitely divisible kernels","volume":"61","author":"Giraldo","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_28","first-page":"2960","article-title":"Multivariate extension of matrix-based Renyi\u2019s \u03b1-order entropy functional","volume":"42","author":"Yu","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"066138","DOI":"10.1103\/PhysRevE.69.066138","article-title":"Estimating Mutual Information","volume":"69","author":"Kraskov","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_30","unstructured":"Kandasamy, K., Krishnamurthy, A., P\u00f3czos, B., Wasserman, L., and Robins, J.M. (2015). Nonparametric von Mises estimators for entropies, divergences and mutual informations. Advances in Neural Information Processing Systems, The MIT Press."},{"key":"ref_31","unstructured":"Belghazi, M.I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R.D. (2018, January 10\u201315). Mutual information neural estimation. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Elad, A., Haviv, D., Blau, Y., and Michaeli, T. (2019, January 27\u201328). Direct Validation of the Information Bottleneck Principle for Deep Nets. Proceedings of the 2019 International Conference on Computer Vision Workshops, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00099"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"J\u00f3nsson, H., Cherubini, G., and Eleftheriou, E. (2019). Convergence of DNNs with mutual-information-based regularization. Entropy, 22.","DOI":"10.3390\/e22070727"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1109\/TIT.1985.1057042","article-title":"The discrete memoryless multiple-access channel with cribbing encoders","volume":"31","author":"Willems","year":"1985","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_35","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley-Interscience. [2nd ed.]."},{"key":"ref_36","unstructured":"Moyer, D., Gao, S., Brekelmans, R., Steeg, G.V., and Galstyan, A. (2018, January 3\u20138). Invariant representations without adversarial training. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Moon, K.R., Sricharan, K., and Hero, A.O. (2017, January 25\u201330). Ensemble estimation of mutual information. In Proceedings of the 2017 IEEE International Symposium on Information Theory, Aachen, Germany.","DOI":"10.1109\/ISIT.2017.8007086"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Noshad, M., Zeng, Y., and Hero, A.O. (2019, January 12\u201317). Scalable mutual information estimation using dependence graphs. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8683351"},{"key":"ref_39","unstructured":"Singh, S., and P\u00f3czos, B. (2014). Exponential concentration of a density functional estimator. Advances in Neural Information Processing Systems, The MIT Press."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/8\/974\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:36:59Z","timestamp":1760164619000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/8\/974"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,29]]},"references-count":39,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["e23080974"],"URL":"https:\/\/doi.org\/10.3390\/e23080974","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,29]]}}}