{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T06:15:45Z","timestamp":1792822545752,"version":"4.5.0"},"reference-count":53,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2020,1,14]],"date-time":"2020-01-14T00:00:00Z","timestamp":1578960000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The information bottleneck (IB) problem tackles the issue of obtaining relevant compressed representations T of some random variable X for the task of predicting Y. It is defined as a constrained optimization problem that maximizes the information the representation has about the task,     I ( T ; Y )    , while ensuring that a certain level of compression r is achieved (i.e.,     I ( X ; T ) \u2264 r    ). For practical reasons, the problem is usually solved by maximizing the IB Lagrangian (i.e.,      L IB   ( T ; \u03b2 )  = I  ( T ; Y )  \u2212 \u03b2 I  ( X ; T )     ) for many values of     \u03b2 \u2208 [ 0 , 1 ]    . Then, the curve of maximal     I ( T ; Y )     for a given     I ( X ; T )     is drawn and a representation with the desired predictability and compression is selected. It is known when Y is a deterministic function of X, the IB curve cannot be explored and another Lagrangian has been proposed to tackle this problem: the squared IB Lagrangian:      L  sq \u2212 IB    ( T ;  \u03b2 sq  )  = I  ( T ; Y )  \u2212  \u03b2 sq  I   ( X ; T )  2     . In this paper, we (i) present a general family of Lagrangians which allow for the exploration of the IB curve in all scenarios; (ii) provide the exact one-to-one mapping between the Lagrange multiplier and the desired compression rate r for known IB curve shapes; and (iii) show we can approximately obtain a specific compression level with the convex IB Lagrangian for both known and unknown IB curve shapes. This eliminates the burden of solving the optimization problem for many values of the Lagrange multiplier. That is, we prove that we can solve the original constrained problem with a single optimization.<\/jats:p>","DOI":"10.3390\/e22010098","type":"journal-article","created":{"date-parts":[[2020,1,15]],"date-time":"2020-01-15T03:20:22Z","timestamp":1579058422000},"page":"98","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["The Convex Information Bottleneck Lagrangian"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0862-1333","authenticated-orcid":false,"given":"Borja","family":"Rodr\u00edguez G\u00e1lvez","sequence":"first","affiliation":[{"name":"Department of Intelligent Systems, Division of Information Science and Engineering (ISE), KTH Royal Institute of Technology, 11428 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9307-484X","authenticated-orcid":false,"given":"Ragnar","family":"Thobaben","sequence":"additional","affiliation":[{"name":"Department of Intelligent Systems, Division of Information Science and Engineering (ISE), KTH Royal Institute of Technology, 11428 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7926-5081","authenticated-orcid":false,"given":"Mikael","family":"Skoglund","sequence":"additional","affiliation":[{"name":"Department of Intelligent Systems, Division of Information Science and Engineering (ISE), KTH Royal Institute of Technology, 11428 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,14]]},"reference":[{"key":"ref_1","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (2000). The information bottleneck method. arXiv."},{"key":"ref_2","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2016). Deep variational information bottleneck. arXiv."},{"key":"ref_3","unstructured":"Peng, X.B., Kanazawa, A., Toyer, S., Abbeel, P., and Levine, S. (2019, January 6\u20139). Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2897","DOI":"10.1109\/TPAMI.2017.2784440","article-title":"Information dropout: Learning optimal representations through noisy computation","volume":"40","author":"Achille","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Slonim, N., and Tishby, N. (2000, January 24\u201328). Document clustering using word clusters via the information bottleneck method. Proceedings of the 23rd annual international ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece.","DOI":"10.1145\/345508.345578"},{"key":"ref_6","unstructured":"Slonim, N., and Tishby, N. (2000). Agglomerative information bottleneck. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"18297","DOI":"10.1073\/pnas.0507432102","article-title":"Information-based clustering","volume":"102","author":"Slonim","year":"2005","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_8","unstructured":"Teahan, W.J. (2000). Text classification and segmentation using minimum cross-entropy. Content-Based Multimedia Information Access, LE CENTRE DE HAUTES ETUDES INTERNATIONALES D\u2019INFORMATIQUE DOCUMENTAIRE."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1611","DOI":"10.1162\/NECO_a_00961","article-title":"The deterministic information bottleneck","volume":"29","author":"Strouse","year":"2017","journal-title":"Neur. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Nazer, B., Ordentlich, O., and Polyanskiy, Y. (2017, January 25\u201330). Information-distilling quantizers. Proceedings of the 2017 IEEE International Symposium on Information Theory (ISIT), Aachen, Germany.","DOI":"10.1109\/ISIT.2017.8006497"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hassanpour, S., W\u00fcbben, D., and Dekorsy, A. (2018, January 15\u201318). On the equivalence of double maxima and KL-means for information bottleneck-based source coding. Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC), Barcelona, Spain.","DOI":"10.1109\/WCNC.2018.8377199"},{"key":"ref_12","unstructured":"Goyal, A., Islam, R., Strouse, D., Ahmed, Z., Botvinick, M., Larochelle, H., Levine, S., and Bengio, Y. (2019). Infobot: Transfer and exploration via the information bottleneck. arXiv."},{"key":"ref_13","unstructured":"Yingjun, P., and Xinwen, H. (2019). Learning Representations in Reinforcement Learning:An Information Bottleneck Approach. arXiv."},{"key":"ref_14","unstructured":"Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. (2020, January 26\u201330). Dynamics-Aware Unsupervised Skill Discovery. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia."},{"key":"ref_15","unstructured":"Schulz, K., Sixt, L., Tombari, F., and Landgraf, T. (2020, January 26\u201330). Restricting the Flow: Information Bottlenecks for Attribution. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Li, X.L., and Eisner, J. (2019, January 3\u20137). Specializing Word Embeddings (for Parsing) by Information Bottleneck. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1276"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"7937","DOI":"10.1073\/pnas.1800521115","article-title":"Efficient compression in color naming and its evolution","volume":"115","author":"Zaslavsky","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1073\/pnas.1711114115","article-title":"Toward a unified theory of efficient, predictive, and sparse coding","volume":"115","author":"Chalk","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Gilad-Bachrach, R., Navot, A., and Tishby, N. (2003). An information theoretic tradeoff between complexity and accuracy. Learning Theory and Kernel Machines, Springer.","DOI":"10.1007\/978-3-540-45167-9_43"},{"key":"ref_20","unstructured":"Cover, T.M., and Thomas, J.A. (2012). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_21","unstructured":"Kolchinsky, A., Tracey, B.D., and Van Kuyk, S. (2019, January 6\u20139). Caveats for information bottleneck in deterministic scenarios. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Courcoubetis, C. (2003). Pricing Communication Networks Economics, Technology and Modelling, Wiley Online Library.","DOI":"10.1002\/0470867175"},{"key":"ref_23","unstructured":"Tishby, N., and Slonim, N. (2001). Data clustering by markovian relaxation and the information bottleneck method. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Slonim, N., Friedman, N., and Tishby, N. (2002, January 11\u201315). Unsupervised document classification using sequential information maximization. Proceedings of the 25th annual international ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland.","DOI":"10.1145\/564376.564401"},{"key":"ref_25","unstructured":"Chalk, M., Marre, O., and Tkacik, G. (2016). Relevant sparse codes with variational information bottleneck. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kolchinsky, A., Tracey, B.D., and Wolpert, D.H. (2019). Nonlinear information bottleneck. Entropy, 21.","DOI":"10.3390\/e21121181"},{"key":"ref_27","unstructured":"Wu, T., Fischer, I., Chuang, I., and Tegmark, M. (2019, January 6\u20139). Learnability for the Information Bottleneck. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA."},{"key":"ref_28","unstructured":"LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998, January 27\u201329). Gradient-based learning applied to document recognition. Proceedings of the 1998 IEEE International Frequency Control Symposium, Pasadena, CA, USA."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Li, X., and Roth, D. (2002). Learning question classifiers. Proceedings of the 19th international conference on Computational linguistics\u2014Volume 1, Association for Computational Linguistics.","DOI":"10.3115\/1072228.1072378"},{"key":"ref_30","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 9). Automatic differentiation in pytorch. Proceedings of the NIPS Autodiff Workshop, Long Beach, CA, USA."},{"key":"ref_31","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer Science+ Business Media."},{"key":"ref_32","unstructured":"Xu, A., and Raginsky, M. (2017). Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_33","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1109\/TPAMI.1982.4767189","article-title":"Minimum cross-entropy pattern classification and cluster analysis","volume":"1","author":"Shore","year":"1982","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_35","first-page":"472","article-title":"Properties of cross-entropy minimization","volume":"27","author":"Shore","year":"1981","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Vera, M., Piantanida, P., and Vega, L.R. (2018, January 17\u201322). The role of the information bottleneck in representation learning. Proceedings of the 2018 IEEE International Symposium on Information Theory (ISIT), Vail, CO, USA.","DOI":"10.1109\/ISIT.2018.8437679"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2696","DOI":"10.1016\/j.tcs.2010.04.006","article-title":"Learning and generalization with the information bottleneck","volume":"411","author":"Shamir","year":"2010","journal-title":"Theor. Comput. Sci."},{"key":"ref_38","first-page":"1947","article-title":"Emergence of invariance and disentanglement in deep representations","volume":"19","author":"Achille","year":"2018","journal-title":"J. Mach. Learn. Res"},{"key":"ref_39","first-page":"1879","article-title":"Strong data processing inequalities for input constrained additive noise channels","volume":"64","author":"Polyanskiy","year":"2017","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Kolchinsky, A., and Tracey, B. (2017). Estimating mixture entropy with pairwise distances. Entropy, 19.","DOI":"10.3390\/e19070361"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Amjad, R.A., and Geiger, B.C. (2019). Learning representations for neural network-based classification using the information bottleneck principle. IEEE Trans. Pattern Anal. Mach. Intell., 1.","DOI":"10.1109\/TPAMI.2019.2909031"},{"key":"ref_42","unstructured":"Alemi, A.A., Fischer, I., and Dillon, J.V. (2018). Uncertainty in the variational information bottleneck. arXiv."},{"key":"ref_43","unstructured":"Wu, T., and Fischer, I. (2020, January 26\u201330). Phase Transitions for the Information Bottleneck in Representation Learning. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia."},{"key":"ref_44","unstructured":"Ester, M., Kriegel, H.P., Sander, J., and Xu, X. (1996, January 2\u20134). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, Menlo Park, CA, USA."},{"key":"ref_45","first-page":"19","article-title":"DBSCAN revisited, revisited: Why and how you should (still) use DBSCAN","volume":"42","author":"Schubert","year":"2017","journal-title":"ACM Trans. Database Syst. TODS"},{"key":"ref_46","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_47","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy."},{"key":"ref_48","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_49","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/S0167-7152(96)00140-X","article-title":"Sparse spatial autoregressions","volume":"33","author":"Pace","year":"1997","journal-title":"Stat. Probab. Lett."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Voorhees, E.M., and Tice, D.M. (2000, January 24\u201328). Building a question answering test collection. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece.","DOI":"10.1145\/345508.345577"},{"key":"ref_52","unstructured":"(2020, January 14). Trevett, Ben. Tutorial on Sentiment Analysis: 5\u2014Multi-class Sentiment Analysis. April 2019. Available online: https:\/\/github.com\/bentrevett\/pytorch-sentiment-analysis."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C. (2014, January 25\u201329). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/1\/98\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:05:06Z","timestamp":1760364306000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/1\/98"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,14]]},"references-count":53,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,1]]}},"alternative-id":["e22010098"],"URL":"https:\/\/doi.org\/10.3390\/e22010098","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,14]]}}}