{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T13:40:06Z","timestamp":1760708406316,"version":"build-2065373602"},"reference-count":23,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2012,3,1]],"date-time":"2012-03-01T00:00:00Z","timestamp":1330560000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Information Bottleneck-based methods use mutual information as a distortion function in order to extract relevant details about the structure of a complex system by compression. One of the approaches used to generate optimal compressed representations is by annealing a parameter. In this manuscript we present a common framework for the study of annealing in information distortion problems. We identify features that should be common to any annealing optimization problem. The main mathematical tools that we use come from the analysis of dynamical systems in the presence of symmetry (equivariant bifurcation theory). Through the compression problem, we make connections to the world of combinatorial optimization and pattern recognition. The two approaches use very different vocabularies and consider different problems to be \u201cinteresting\u201d. We provide an initial link, through the Normalized Cut Problem, where the two disciplines can exchange tools and ideas.<\/jats:p>","DOI":"10.3390\/e14030456","type":"journal-article","created":{"date-parts":[[2012,3,1]],"date-time":"2012-03-01T11:01:53Z","timestamp":1330599713000},"page":"456-479","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["The Mathematical Structure of Information Bottleneck Methods"],"prefix":"10.3390","volume":"14","author":[{"given":"Tom\u00e1\u0161","family":"Gedeon","sequence":"first","affiliation":[{"name":"Department of Mathematical Sciences, Montana State University, Bozeman, MT 59717, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Albert E.","family":"Parker","sequence":"additional","affiliation":[{"name":"Center for Biofilm Engineering, Montana State University, Bozeman, MT 59717, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander G.","family":"Dimitrov","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Science Programs, Washington State University Vancouver, Vancouver, WA 98686, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2012,3,1]]},"reference":[{"key":"ref_1","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (1999, January September). The information bottleneck method. Proceedings of the 37th annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1080\/net.12.4.441.472","article-title":"Neural coding and decoding: Communication channels and quantization","volume":"12","author":"Dimitrov","year":"2001","journal-title":"Netw. Comput. Neural Syst."},{"key":"ref_3","first-page":"33","article-title":"Information distortion and neural coding","volume":"10","author":"Gedeon","year":"2003","journal-title":"Can. Appl. Math. Q."},{"key":"ref_4","unstructured":"Cover, T., and Thomas, J. (1991). Elements of Information Theory, Wiley Series in Communication."},{"key":"ref_5","first-page":"617","article-title":"Agglomerative information bottleneck","volume":"Volume 12","author":"Solla","year":"2000","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_6","unstructured":"Slonim, N. (2002). The information bottleneck: Theory and applications. [Ph.D. Thesis, Hebrew University]."},{"key":"ref_7","unstructured":"Pereira, F., Tishby, N.Z., and Lee, L. (July, January 28). Distributional clustering of english words. Proceedings of the 30th Annual Meeting of the Association for Computational Linguistics, Newark, DE, USA."},{"key":"ref_8","first-page":"33","article-title":"Distributional word clusters vs. words for text categorization","volume":"3","author":"Bekkerman","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_9","unstructured":"Mumey, B., Gedeon, T., Taubmann, J., and Hall, K. (,  2000). Network dynamics discovery in genetic and neural systems. Proceedings of the ISMB 2000, La Jolla, CA, USA."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Bialek, W., de Ruyter van Steveninck, R.R., and Tishby, N. (2006, January 9\u201314). Efficient representation as a design principle for neural coding and computation. Proceedings of the 2006 IEEE International Symposium on Information Theory, Seattle, WA, USA.","DOI":"10.1109\/ISIT.2006.261867"},{"key":"ref_11","unstructured":"Schneidman, E., Slonim, N., Tishby, N., de Ruyter van Steveninck, R.R., and Bialek, W. (2003). Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1046\/j.1365-8711.2001.04125.x","article-title":"Objective classification of galaxy spectra using the information bottleneck method","volume":"323","author":"Slonim","year":"2001","journal-title":"Mon. Not. R. Astron. Soc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1109\/TGRS.2006.890557","article-title":"Image time-series data mining based on the information-bottleneck principle","volume":"45","author":"Gueguen","year":"2007","journal-title":"IEEE Trans. Geosci. Rem. Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2210","DOI":"10.1109\/5.726788","article-title":"Deterministic annealing for clustering, compression, classification, regression, and related optimization problems","volume":"86","author":"Rose","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1109\/TIT.2009.2037045","article-title":"Symmetry breaking clusters in soft clustering decoding of neural codes","volume":"56","author":"Parker","year":"2010","journal-title":"IEEE Trans. Inform. Theor."},{"key":"ref_16","first-page":"969","article-title":"Annealing and the rate distortion problem","volume":"Volume 15","author":"Becker","year":"2003","journal-title":"Advances in Neural Information Processing Systems 15"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1007\/s10884-004-6112-0","article-title":"Bifurcation structure of a class of SN-invariant constrained optimization problems","volume":"16","author":"Parker","year":"2004","journal-title":"J. Dynam. Differ. Equat."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Nocedal, J., and Wright, S.J. (2000). Numerical Optimization, Springer.","DOI":"10.1007\/b98874"},{"key":"ref_19","unstructured":"Parker, A.E. (2003). Symmetry Breaking Bifurcations of the Information Distortion. [Ph.D. Thesis, Montana State University]."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Golubitsky, M., and Schaeffer, D.G. (1985). Singularities and Groups in Bifurcation Theory I, Springer Verlag.","DOI":"10.1007\/978-1-4612-5034-0"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"888","DOI":"10.1109\/34.868688","article-title":"Normalized cuts and image segmentation","volume":"22","author":"Shi","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1016\/j.patcog.2007.06.014","article-title":"Annealing an information type cost function computes the normalized cut","volume":"41","author":"Gedeon","year":"2008","journal-title":"Pattern Recogn."},{"key":"ref_23","unstructured":"Mumey, B., and Gedeon, T. (, January March). Optimal mutual information quantization is NP-complete. Proceedings of the Neural Information Coding (NIC) workshop, Snowbird, UT, USA."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/14\/3\/456\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:49:09Z","timestamp":1760219349000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/14\/3\/456"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,3,1]]},"references-count":23,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2012,3]]}},"alternative-id":["e14030456"],"URL":"https:\/\/doi.org\/10.3390\/e14030456","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2012,3,1]]}}}