{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T06:11:13Z","timestamp":1781763073449,"version":"3.54.5"},"reference-count":58,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2019,11,30]],"date-time":"2019-11-30T00:00:00Z","timestamp":1575072000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CHE-1648973"],"award-info":[{"award-number":["CHE-1648973"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009566","name":"Foundational Questions Institute","doi-asserted-by":"publisher","award":["FQXi-RFP-1622"],"award-info":[{"award-number":["FQXi-RFP-1622"]}],"id":[{"id":"10.13039\/100009566","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"publisher","award":["A9550-15-1-0038"],"award-info":[{"award-number":["A9550-15-1-0038"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed \u201cbottleneck\u201d random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete X and Y with small state spaces, and continuous X and Y with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and\/or continuous X and Y, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed \u201cvariational IB\u201d method on several real-world datasets.<\/jats:p>","DOI":"10.3390\/e21121181","type":"journal-article","created":{"date-parts":[[2019,12,2]],"date-time":"2019-12-02T10:50:45Z","timestamp":1575283845000},"page":"1181","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":103,"title":["Nonlinear Information Bottleneck"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3518-9208","authenticated-orcid":false,"given":"Artemy","family":"Kolchinsky","sequence":"first","affiliation":[{"name":"Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brendan D.","family":"Tracey","sequence":"additional","affiliation":[{"name":"Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA"},{"name":"Department of Aeronautics &amp; Astronautics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3105-2869","authenticated-orcid":false,"given":"David H.","family":"Wolpert","sequence":"additional","affiliation":[{"name":"Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA"},{"name":"Complexity Science Hub, 1080 Vienna, Austria"},{"name":"Center for Bio-Social Complex Systems, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2019,11,30]]},"reference":[{"key":"ref_1","unstructured":"Tishby, N., Pereira, F., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1080\/net.12.4.441.472","article-title":"Neural coding and decoding: Communication channels and quantization","volume":"12","author":"Dimitrov","year":"2001","journal-title":"Netw. Comput. Neural Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"771","DOI":"10.1162\/089976602317318947","article-title":"Information loss in an optimal maximum likelihood decoding","volume":"14","author":"Samengo","year":"2002","journal-title":"Neural Comput."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1109\/TIT.1975.1055437","article-title":"A conditional entropy bound for a pair of discrete random variables","volume":"21","author":"Witsenhausen","year":"1975","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/TIT.1975.1055469","article-title":"Source Coding with Side Information and a Converse for Degraded Broadcast Channels","volume":"21","author":"Ahlswede","year":"1975","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1007\/978-3-540-45167-9_43","article-title":"An Information Theoretic Tradeoff between Complexity and Accuracy","volume":"Volume 2777","author":"Goos","year":"2003","journal-title":"Learning Theory and Kernel Machines"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Slonim, N., and Tishby, N. (2000, January 24\u201328). Document clustering using word clusters via the information bottleneck method. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece.","DOI":"10.1145\/345508.345578"},{"key":"ref_8","unstructured":"Tishby, N., and Slonim, N. (2001). Data clustering by markovian relaxation and the information bottleneck method. Advances in Neural Information Processing Systems 13 (NIPS 2000), MIT Press."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Cardinal, J. (2003, January 6\u20139). Compression of side information. Proceedings of the 2003 International Conference on Multimedia and Expo, Baltimore, MD, USA.","DOI":"10.1109\/ICME.2003.1221680"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zeitler, G., Koetter, R., Bauch, G., and Widmer, J. (2008, January 1\u20135). Design of network coding functions in multihop relay networks. Proceedings of the 2008 5th International Symposium on Turbo Codes and Related Topics, Lausanne, Switzerland.","DOI":"10.1109\/TURBOCODING.2008.4658706"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Courtade, T.A., and Wesel, R.D. (August, January 31). Multiterminal source coding with an entropy-based distortion measure. Proceedings of the 2011 IEEE International Symposium on Information Theory, St. Petersburg, Russia.","DOI":"10.1109\/ISIT.2011.6033913"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1294","DOI":"10.1109\/TPAMI.2008.138","article-title":"Supervised learning of quantizer codebooks by information loss minimization","volume":"31","author":"Lazebnik","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Winn, J., Criminisi, A., and Minka, T. (2005, January 17\u201321). Object categorization by learned universal visual dictionary. Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV\u201905) Volume 1, Beijing, China.","DOI":"10.1109\/ICCV.2005.171"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hecht, R.M., Noor, E., and Tishby, N. (2009, January 6\u201310). Speaker recognition by Gaussian information bottleneck. Proceedings of the 10th Annual Conference of the International Speech Communication Association, Brighton, UK.","DOI":"10.21437\/Interspeech.2009-387"},{"key":"ref_15","unstructured":"Yaman, S., Pelecanos, J., and Sarikaya, R. (2012, January 25\u201328). Bottleneck features for speaker recognition. Proceedings of the Speaker and Language Recognition Workshop, Singapore."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Van Kuyk, S., Kleijn, W.B., and Hendriks, R.C. (2017, January 5\u20139). On the information rate of speech communication. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.","DOI":"10.1109\/ICASSP.2017.7953233"},{"key":"ref_17","unstructured":"Van Kuyk, S. (2019). Speech Communication from an Information Theoretical Perspective. [Ph.D. Thesis, Victoria University of Wellington]."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"7937","DOI":"10.1073\/pnas.1800521115","article-title":"Efficient compression in color naming and its evolution","volume":"115","author":"Zaslavsky","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_19","unstructured":"Rodr\u00edguez G\u00e1lvez, B. (2019). The Information Bottleneck: Connections to Other Problems, Learning and Exploration of the IB Curve. [Master\u2019s Thesis, KTH Royal Institute of Technology]."},{"key":"ref_20","unstructured":"Hafez-Kolahi, H., and Kasaei, S. (2019). Information Bottleneck and its Applications in Deep Learning. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tishby, N., and Zaslavsky, N. (May, January 26). Deep learning and the information bottleneck principle. Proceedings of the 2015 IEEE Information Theory Workshop (ITW), Jerusalem, Israel.","DOI":"10.1109\/ITW.2015.7133169"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2696","DOI":"10.1016\/j.tcs.2010.04.006","article-title":"Learning and generalization with the information bottleneck","volume":"411","author":"Shamir","year":"2010","journal-title":"Theor. Comput. Sci."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Vera, M., Piantanida, P., and Vega, L.R. (2018, January 17\u201322). The Role of the Information Bottleneck in Representation Learning. Proceedings of the 2018 IEEE International Symposium on Information Theory (ISIT), Vail, CO, USA.","DOI":"10.1109\/ISIT.2018.8437679"},{"key":"ref_24","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2017, January 24\u201326). Deep Variational Information Bottleneck. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_25","unstructured":"Alemi, A.A., Fischer, I., and Dillon, J.V. (2018). Uncertainty in the variational information bottleneck. arXiv."},{"key":"ref_26","unstructured":"Amjad, R.A., and Geiger, B.C. (2018). Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle. arXiv."},{"key":"ref_27","unstructured":"Shwartz-Ziv, R., and Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. arXiv."},{"key":"ref_28","unstructured":"Saxe, A., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B., and Cox, D. (May, January 30). On the information bottleneck theory of deep learning. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lemar\u00e9chal, C. (2001). Lagrangian relaxation. Computational Combinatorial Optimization, Springer.","DOI":"10.1007\/3-540-45586-8_4"},{"key":"ref_30","unstructured":"Cover, T.M., and Thomas, J.A. (2012). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_31","first-page":"165","article-title":"Information bottleneck for Gaussian variables","volume":"6","author":"Chechik","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_32","unstructured":"Kolchinsky, A., Tracey, B.D., and Van Kuyk, S. (May, January 30). Caveats for information bottleneck in deterministic scenarios. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Miettinen, K. (1998). Nonlinear Multiobjective Optimization, Springer.","DOI":"10.1007\/978-1-4615-5563-6"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Rodr\u00edguez G\u00e1lvez, B., Thobaben, R., and Skoglund, M. (2019). The Convex Information Bottleneck Lagrangian. arXiv.","DOI":"10.3390\/e22010098"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kolchinsky, A., and Tracey, B.D. (2017). Estimating Mixture Entropy with Pairwise Distances. Entropy, 19.","DOI":"10.3390\/e19070361"},{"key":"ref_36","unstructured":"Chalk, M., Marre, O., and Tkacik, G. (2016, January 5\u201310). Relevant sparse codes with variational information bottleneck. Proceedings of the 2016 Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain."},{"key":"ref_37","unstructured":"Achille, A., and Soatto, S. (2016). Information Dropout: Learning optimal representations through noise. arXiv."},{"key":"ref_38","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Silverman, B.W. (2018). Density Estimation for Statistics and Data Analysis, Routledge.","DOI":"10.1201\/9781315140919"},{"key":"ref_40","unstructured":"Kolchinsky, A., and Wolpert, D.H. (2016, January 5\u201310). Supervised learning with information penalties. Proceedings of the 2016 Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kolchinsky, A., Tracey, B.D., and Wolpert, D.H. (2019). Nonlinear Information Bottleneck. Entropy, 21.","DOI":"10.3390\/e21121181"},{"key":"ref_42","unstructured":"Schraudolph, N.N. (1995). Optimization of entropy with neural networks. [Ph.D. Thesis, University of California]."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1109\/TNN.2004.828766","article-title":"Gradient-based manipulation of nonparametric entropy estimates","volume":"15","author":"Schraudolph","year":"2004","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1016\/j.sigpro.2004.11.022","article-title":"Fast kernel entropy estimation and optimization","volume":"85","author":"Shwartz","year":"2005","journal-title":"Signal Process."},{"key":"ref_45","first-page":"1415","article-title":"Feature extraction by non-parametric mutual information maximization","volume":"3","author":"Torkkola","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.physrep.2006.12.004","article-title":"Causality detection based on information-theoretic approaches in time series analysis","volume":"441","author":"Palus","year":"2007","journal-title":"Phys. Rep."},{"key":"ref_47","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8\u201313). Generative adversarial nets. Proceedings of the 2014 Conference on Neural Information Processing Systems (NIPS 2014), Montreal, QC, Canada."},{"key":"ref_48","unstructured":"Hinton, G.E., and Zemel, R.S. (1994). Autoencoders, minimum description length, and Helmholtz free energy. Advances in Neural Information Processing Systems 7 (NIPS 1994), MIT Press."},{"key":"ref_49","unstructured":"Hinton, G.E., and Zemel, R.S. (2019, November 30). Minimizing Description Length in an Unsupervised Neural Network. Available online: https:\/\/www.cs.toronto.edu\/~fritz\/absps\/mdlnn.pdf."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Gielen, S., and Kappen, B. (1993). Elimination of Overtraining by a Mutual Information Network. ICANN\u201993, Springer.","DOI":"10.1007\/978-1-4471-2063-6"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A. (2008, January 5\u20139). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland.","DOI":"10.1145\/1390156.1390294"},{"key":"ref_52","unstructured":"Kingma, D.P., and Welling, M. (2014, January 14\u201316). Auto-encoding variational bayes. Proceedings of the International Conference on Learning Representations (ICLR 2014), Banff, AB, Canada."},{"key":"ref_53","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017, January 24\u201326). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. Proceedings of the International Conference on Learning Representations (ICLR 2017), Toulon, France."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R.A., and Murphy, K. (2018, January 10\u201315). Fixing a Broken ELBO. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.","DOI":"10.1007\/978-3-319-67401-8_2"},{"key":"ref_55","unstructured":"Kingma, D., and Ba, J. (2015, January 7\u20139). Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference for Learning Representations (ICLR 2015), San Diego, CA, USA."},{"key":"ref_56","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/S0167-7152(96)00140-X","article-title":"Sparse spatial autoregressions","volume":"33","author":"Pace","year":"1997","journal-title":"Stat. Probab. Lett."},{"key":"ref_58","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/12\/1181\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:38:56Z","timestamp":1760189936000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/12\/1181"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,30]]},"references-count":58,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["e21121181"],"URL":"https:\/\/doi.org\/10.3390\/e21121181","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,30]]}}}