{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T11:38:28Z","timestamp":1780054708668,"version":"3.54.0"},"reference-count":112,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T00:00:00Z","timestamp":1580083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020","doi-asserted-by":"publisher","award":["694630"],"award-info":[{"award-number":["694630"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate connections of this setting to remote source-coding under logarithmic loss distortion measure, information combining, common reconstruction, the Wyner\u2013Ahlswede\u2013Korner problem, the efficiency of investment information, as well as, generalization, variational inference, representation learning, autoencoders, and others are highlighted. We discuss its extension to the distributed information bottleneck problem with emphasis on the Gaussian model and highlight the basic connections to the uplink Cloud Radio Access Networks (CRAN) with oblivious processing. For this model, the optimal trade-offs between relevance (i.e., information) and complexity (i.e., rates) in the discrete and vector Gaussian frameworks is determined. In the concluding outlook, some interesting problems are mentioned such as the characterization of the optimal inputs (\u201cfeatures\u201d) distributions under power limitations maximizing the \u201crelevance\u201d for the Gaussian information bottleneck, under \u201ccomplexity\u201d constraints.<\/jats:p>","DOI":"10.3390\/e22020151","type":"journal-article","created":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T11:41:57Z","timestamp":1580125317000},"page":"151","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":86,"title":["On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2023-9476","authenticated-orcid":false,"given":"Abdellatif","family":"Zaidi","sequence":"first","affiliation":[{"name":"Institut d\u2019\u00c9lectronique et d\u2019Informatique Gaspard-Monge, Universit\u00e9 Paris-Est, 77454 Champs-sur-Marne, France"},{"name":"Mathematics and Algorithmic Sciences Lab, Paris Research Center, Huawei Technologies France, 92100 Boulogne-Billancourt, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"I\u00f1aki","family":"Estella-Aguerri","sequence":"additional","affiliation":[{"name":"Mathematics and Algorithmic Sciences Lab, Paris Research Center, Huawei Technologies France, 92100 Boulogne-Billancourt, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shlomo","family":"Shamai (Shitz)","sequence":"additional","affiliation":[{"name":"Technion Institute of Technology, Technion City, Haifa 32000, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,27]]},"reference":[{"key":"ref_1","unstructured":"Tishby, N., Pereira, F., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the Thirty-Seventh Annual Allerton Conference on Communication, Control, and Computing, Allerton House, Monticello, IL, USA."},{"key":"ref_2","unstructured":"Pratt, W.K. (1991). Digital Image Processing, John Willey & Sons Inc."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Yu, S., and Principe, J.C. (2018). Understanding Autoencoders with Information Theoretic Concepts. arXiv.","DOI":"10.1016\/j.neunet.2019.05.003"},{"key":"ref_4","unstructured":"Yu, S., Jenssen, R., and Principe, J.C. (2018). Understanding Convolutional Neural Network Training with Information Theory. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kong, Y., and Schoenebeck, G. (2018). Water from Two Rocks: Maximizing the Mutual Information. arXiv.","DOI":"10.1145\/3219166.3219194"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Ugur, Y., Aguerri, I.E., and Zaidi, A. (2017, January 6\u201310). A generalization of Blahut-Arimoto algorithm to computing rate-distortion regions of multiterminal source coding under logarithmic loss. Proceedings of the IEEE Information Theory Workshop, ITW, Kaohsiung, Taiwan.","DOI":"10.1109\/ITW.2017.8277967"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1109\/TIT.1962.1057738","article-title":"Information transmission with additional noise","volume":"85","author":"Dobrushin","year":"1962","journal-title":"IRE Trans. Inf. Theory"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1109\/TIT.1975.1055437","article-title":"A conditional Entropy Bound for a Pair of Discrete Random Variables","volume":"IT-21","author":"Witsenhausen","year":"1975","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1109\/TIT.1980.1056251","article-title":"Indirect Rate Distortion Problems","volume":"IT-26","author":"Witsenhausen","year":"1980","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_10","unstructured":"Shwartz-Ziv, R., and Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Achille, A., and Soatto, S. (2017). Emergence of Invariance and Disentangling in Deep Representations. arXiv.","DOI":"10.1109\/ITA.2018.8503149"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"McAllester, D.A. (2013). A PAC-Bayesian Tutorial with a Dropout Bound. arXiv.","DOI":"10.1007\/978-3-642-41136-6_10"},{"key":"ref_13","unstructured":"Alemi, A.A. (2019). Variational Predictive Information Bottleneck. arXiv."},{"key":"ref_14","unstructured":"Mukherjee, S. (2019). Machine Learning using the Variational Predictive Information Bottleneck with a Validation Set. arXiv."},{"key":"ref_15","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv."},{"key":"ref_16","unstructured":"Mukherjee, S. (2019). General Information Bottleneck Objectives and their Applications to Machine Learning. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1162\/neco_a_01136","article-title":"The information bottleneck and geometric clustering","volume":"31","author":"Strouse","year":"2019","journal-title":"Neural Comput."},{"key":"ref_18","first-page":"7908","article-title":"Gaussian Lower Bound for the Information Bottleneck Limit","volume":"18","author":"Painsky","year":"2018","journal-title":"J. Mach. Learn. Res. (JMLR)"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kittichokechai, K., and Caire, G. (2016, January 10\u201315). Privacy-constrained remote source coding. Proceedings of the 2016 IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain.","DOI":"10.1109\/ISIT.2016.7541465"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4666","DOI":"10.1109\/TIT.2008.928951","article-title":"Successive Refinement for Hypothesis Testing and Lossless One-Helper Problem","volume":"54","author":"Tian","year":"2008","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sreekumar, S., G\u00fcnd\u00fcz, D., and Cohen, A. (2018). Distributed Hypothesis Testing Under Privacy Constraints. arXiv.","DOI":"10.1109\/ITW.2018.8613433"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Aguerri, I.E., Zaidi, A., Caire, G., and Shamai (Shitz), S. (2017, January 25\u201330). On the Capacity of Cloud Radio Access Networks with Oblivious Relaying. Proceedings of the 2017 IEEE International Symposium on Information Theory (ISIT), Aachen, Germany.","DOI":"10.1109\/ISIT.2017.8006893"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4575","DOI":"10.1109\/TIT.2019.2897564","article-title":"On the capacity of uplink cloud radio access networks with oblivious relaying","volume":"65","author":"Aguerri","year":"2019","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Stark, M., Bauch, G., Lewandowsky, J., and Saha, S. (2019, January 20\u201324). Decoding of Non-Binary LDPC Codes Using the Information Bottleneck Method. Proceedings of the ICC 2019-2019 IEEE International Conference on Communications (ICC), Shanghai, China.","DOI":"10.1109\/ICC.2019.8761712"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","unstructured":"Erdogmus, D. (2002). Information Theoretic Learning: Renyi\u2019s Entropy and Its Applications to Adaptive System Training. [Ph.D. Thesis, University of Florida Gainesville]."},{"key":"ref_27","unstructured":"Principe, J.C., Euliano, N.R., and Lefebvre, W.C. (2000). Neural and Adaptive Systems: Fundamentals Through Simulations, Wiley."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Fisher, J.W. (1997). Nonlinear Extensions to the Minumum Average Correlation Energy Filter, University of Florida.","DOI":"10.1117\/1.601518"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"5357","DOI":"10.1109\/TIT.2015.2462848","article-title":"Justification of logarithmic loss via the benefit of side information","volume":"61","author":"Jiao","year":"2015","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Painsky, A., and Wornell, G.W. (2018). On the Universality of the Logistic Loss Function. arXiv.","DOI":"10.1109\/ISIT.2018.8437786"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1109\/2.36","article-title":"Self-organization in a perceptual network","volume":"21","author":"Linsker","year":"1988","journal-title":"Computer"},{"key":"ref_32","unstructured":"Quinlan, J.R. (2014). C4. 5: Programs for Machine Learning, Elsevier."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1109\/TIT.1968.1054142","article-title":"Approximating discrete probability distributions with dependence trees","volume":"14","author":"Chow","year":"1968","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_34","first-page":"308959","article-title":"On the impact of entropy estimation on transcriptional regulatory network inference based on mutual information","volume":"2009","author":"Olsen","year":"2008","journal-title":"EURASIP J. Bioinf. Syst. Biol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"986","DOI":"10.1109\/TMI.2003.815867","article-title":"Mutual-information-based registration of medical images: A survey","volume":"22","author":"Pluim","year":"2003","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/A:1007958904918","article-title":"Alignment by maximization of mutual information","volume":"24","author":"Viola","year":"1997","journal-title":"Int. J. Comput. Vis."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Cesa-Bianchi, N., and Lugosi, G. (2006). Prediction, Learning and Games, Cambridge University Press.","DOI":"10.1017\/CBO9780511546921"},{"key":"ref_38","unstructured":"Lehmann, E.L., and Casella, G. (2006). Theory of Point Estimation, Springer Science & Business Media."},{"key":"ref_39","first-page":"499","article-title":"Stability and generalization","volume":"2","author":"Bousquet","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"ref_40","first-page":"2635","article-title":"Learnability, stability and uniform convergence","volume":"11","author":"Shamir","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_41","unstructured":"Xu, A., and Raginsky, M. (2017, January 4\u20139). Information-theoretic analysis of generalization capability of learning algorithms. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_42","unstructured":"Russo, D., and Zou, J. (2015). How much does your data exploration overfit? Controlling bias via information usage. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Amjad, R.A., and Geiger, B.C. (2019). Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle. IEEE Trans. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2019.2909031"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"2835","DOI":"10.1109\/TIT.2015.2412945","article-title":"Minimax estimation of functionals of discrete distributions","volume":"61","author":"Jiao","year":"2015","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_46","unstructured":"Valiant, P., and Valiant, G. (2013, January 5\u201310). Estimating the unseen: improved estimators for entropy and other properties. Proceedings of the Advances in Neural Information Processing Systems 26 (NIPS 2013), Lake Tahoe, NV, USA."},{"key":"ref_47","unstructured":"Chalk, M., Marre, O., and Tkacik, G. (2016). Relevant sparse codes with variational information bottleneck. arXiv."},{"key":"ref_48","unstructured":"Alemi, A., Fischer, I., Dillon, J., and Murphy, K. (2017, January 24\u201326). Deep Variational Information Bottleneck. Proceedings of the International Conference on Learning Representations, ICLR 2017, Toulon, France."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"2897","DOI":"10.1109\/TPAMI.2017.2784440","article-title":"Information Dropout: Learning Optimal Representations Through Noisy Computation","volume":"40","author":"Achille","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Harremoes, P., and Tishby, N. (2007, January 24\u201329). The information bottleneck revisited or how to choose a good distortion measure. Proceedings of the 2007 IEEE International Symposium on Information Theory, Nice, France.","DOI":"10.1109\/ISIT.2007.4557285"},{"key":"ref_51","unstructured":"Gamal, A.E., and Kim, Y.H. (2011). Network Information Theory, Cambridge University Press."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1037\/h0058165","article-title":"The most predictable criterion","volume":"26","author":"Hotelling","year":"1935","journal-title":"J. Educ. Psycol."},{"key":"ref_53","unstructured":"Globerson, A., and Tishby, N. (2004). On the Optimality of the Gaussian Information Bottleneck Curve, Hebrew University. Technical Report."},{"key":"ref_54","first-page":"165","article-title":"Information Bottleneck for Gaussian Variables","volume":"6","author":"Chechik","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Wieczorek, A., and Roth, V. (2019). On the Difference Between the Information Bottleneck and the Deep Information Bottleneck. arXiv.","DOI":"10.3390\/e22020131"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood from incomplete data via the EM algorithm","volume":"39","author":"Dempster","year":"1977","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1109\/TIT.1972.1054855","article-title":"Computation of channel capacity and rate-distortion functions","volume":"18","author":"Blahut","year":"1972","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_58","first-page":"12","article-title":"An algorithm for computing the capacity of arbitrary discrete memoryless channels","volume":"IT-18","author":"Arimoto","year":"1972","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Winkelbauer, A., and Matz, G. (2014, January 19\u201321). Rate-information-optimal gaussian channel output compression. Proceedings of the 48th Annual Conference on Information Sciences and Systems (CISS), Princeton, NJ, USA.","DOI":"10.1109\/CISS.2014.6814120"},{"key":"ref_60","unstructured":"G\u00e1lvez, B.R., Thobaben, R., and Skoglund, M. (2020). The Convex Information Bottleneck Lagrangian. Entropy, 20."},{"key":"ref_61","unstructured":"Jang, E., Gu, S., and Poole, B. (2017). Categorical Reparameterization with Gumbel-Softmax. arXiv."},{"key":"ref_62","unstructured":"Maddison, C.J., Mnih, A., and Teh, Y.W. (2016). The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. arXiv."},{"key":"ref_63","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"3132","DOI":"10.1109\/TIT.2011.2119930","article-title":"Noisy Network Coding","volume":"57","author":"Lim","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1109\/TIT.2013.2288257","article-title":"Multiterminal source coding under logarithmic loss","volume":"60","author":"Courtade","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_66","unstructured":"Csisz\u00e1r, I., and K\u00f6rner, J. (1981). Information Theory: Coding Theorems for Discrete Memoryless Systems, Academic Press."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TIT.1976.1055508","article-title":"The rate-distortion function for source coding with side information at the decoder","volume":"22","author":"Wyner","year":"1976","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"4995","DOI":"10.1109\/TIT.2009.2030487","article-title":"Coding and Common Reconstruction","volume":"IT-11","author":"Steinberg","year":"2009","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_69","unstructured":"Benammar, M., and Zaidi, A. (2016, January 1\u20136). Rate-Distortion of a Heegard-Berger Problem with Common Reconstruction Constraint. Proceedings of the International Zurich Seminar on Information and Communication, Cambridge, MA, USA."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"5080","DOI":"10.1109\/TIT.2016.2586919","article-title":"Rate-distortion function for a heegard-berger problem with two sources and degraded reconstruction sets","volume":"62","author":"Benammar","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"1313","DOI":"10.1109\/TIT.2005.844077","article-title":"Extremes of Information Combining","volume":"51","author":"Sutskover","year":"2005","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1561\/0100000013","article-title":"Information Combining","volume":"3","author":"Land","year":"2006","journal-title":"Found. Trends Commun. Inf. Theory"},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1109\/TIT.1975.1055374","article-title":"On source coding with side information at the decoder","volume":"21","author":"Wyner","year":"1975","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/TIT.1975.1055469","article-title":"Source coding with side information and a converse for degraded broadcast channels","volume":"21","author":"Ahlswede","year":"1975","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Makhdoumi, A., Salamatian, S., Fawaz, N., and M\u00e9dard, M. (2014, January 2\u20135). From the information bottleneck to the privacy funnel. Proceedings of the IEEE Information Theory Workshop, ITW, Hobart, Tasamania, Australia.","DOI":"10.1109\/ITW.2014.6970882"},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"1512","DOI":"10.1109\/TIT.2018.2865558","article-title":"Information Extraction Under Privacy Constraints","volume":"65","author":"Asoodeh","year":"2019","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"1026","DOI":"10.1109\/18.669153","article-title":"The efficiency of investment information","volume":"44","author":"Erkip","year":"1998","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_78","doi-asserted-by":"crossref","unstructured":"Hinton, G.E., and van Camp, D. (1993). Keeping the Neural Networks Simple by Minimizing the Description Length of the Weights. Proceedings of the Sixth Annual Conference on Computational Learning Theory, ACM.","DOI":"10.1145\/168304.168306"},{"key":"ref_79","doi-asserted-by":"crossref","unstructured":"Gilad-Bachrach, R., Navot, A., and Tishby, N. (2003). An Information Theoretic Tradeoff between Complexity and Accuracy. Learning Theory and Kernel Machines, Springer. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-540-45167-9_43"},{"key":"ref_80","unstructured":"Vera, M., Piantanida, P., and Vega, L.R. (2018). The Role of Information Complexity and Randomization in Representation Learning. arXiv."},{"key":"ref_81","unstructured":"Huang, S.L., Makur, A., Wornell, G.W., and Zheng, L. (2019). On Universal Features for High-Dimensional Learning and Inference. arXiv."},{"key":"ref_82","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017, January 24\u201326). \u03b2-VAE: Learning basic visual concepts with a constrained variational framework. Proceedings of the International Conference on Learning Representations, Toulon, France."},{"key":"ref_83","unstructured":"Kurakin, A., Goodfellow, I., and Bengio, S. (2016). Adversarial Machine Learning at Scale. arXiv."},{"key":"ref_84","unstructured":"Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2017). Towards Deep Learning Models Resistant to Adversarial Attacks. arXiv."},{"key":"ref_85","unstructured":"Engstrom, L., Tran, B., Tsipras, D., Schmidt, L., and Madry, A. (2017). A rotation and a translation suffice: Fooling cnns with simple transformations. arXiv."},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Pensia, A., Jog, V., and Loh, P.L. (2019). Extracting robust and accurate features via a robust information bottleneck. arXiv.","DOI":"10.1109\/JSAIT.2020.2991005"},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1109\/TIT.2005.844072","article-title":"Mutual Information and Minimum Mean-Square Error in Gaussian Channels","volume":"51","author":"Guo","year":"2005","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"1501","DOI":"10.1109\/18.104312","article-title":"Information theoretic inequalities","volume":"37","author":"Dembo","year":"1991","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"6870","DOI":"10.1109\/TIT.2014.2358692","article-title":"An Outer Bound for the Vector Gaussian CEO Problem","volume":"60","author":"Ekrem","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_90","unstructured":"Cover, T.M., and Thomas, J.A. (1991). Elements of Information Theory, John Willey & Sons Inc."},{"key":"ref_91","unstructured":"Aguerri, I.E., and Zaidi, A. (2018, January 21\u201323). Distributed information bottleneck method for discrete and Gaussian sources. Proceedings of the International Zurich Seminar on Information and Communication, IZS, Zurich, Switzerland."},{"key":"ref_92","unstructured":"Aguerri, I.E., and Zaidi, A. (2018). Distributed Variational Representation Learning. arXiv."},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"Winkelbauer, A., Farthofer, S., and Matz, G. (July, January 29). The rate-information trade-off for Gaussian vector channels. Proceedings of the The 2014 IEEE International Symposium on Information Theory, Honolulu, HI, USA.","DOI":"10.1109\/ISIT.2014.6875354"},{"key":"ref_94","doi-asserted-by":"crossref","unstructured":"Ugur, Y., Aguerri, I.E., and Zaidi, A. (2018, January 25\u201329). Rate region of the vector Gaussian CEO problem under logarithmic loss. Proceedings of the 2018 IEEE Information Theory Workshop (ITW), Guangzhou, China.","DOI":"10.1109\/ITW.2018.8613480"},{"key":"ref_95","doi-asserted-by":"crossref","unstructured":"Ugur, Y., Aguerri, I.E., and Zaidi, A. (2018). Vector Gaussian CEO Problem under Logarithmic Loss. arXiv.","DOI":"10.1109\/ITW.2018.8613480"},{"key":"ref_96","doi-asserted-by":"crossref","first-page":"2880","DOI":"10.1109\/TIT.2011.2120030","article-title":"On Codebook Information for Interference Relay Channels With Out-of-Band Relaying","volume":"57","author":"Simeone","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_97","doi-asserted-by":"crossref","first-page":"3008","DOI":"10.1109\/TIT.2008.924659","article-title":"Communication Via Decentralized Processing","volume":"54","author":"Sanderovich","year":"2008","journal-title":"IEEE Tran. Inf. Theory"},{"key":"ref_98","doi-asserted-by":"crossref","first-page":"2148","DOI":"10.1109\/18.720535","article-title":"Reliable communication under channel uncertainty","volume":"44","author":"Lapidoth","year":"1998","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_99","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1109\/TIT.1979.1056084","article-title":"Capacity Theorems for the Relay Channel","volume":"25","author":"Cover","year":"1979","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_100","unstructured":"Xu, C., Tao, D., and Xu, C. (2013). A survey on multi-view learning. arXiv."},{"key":"ref_101","doi-asserted-by":"crossref","unstructured":"Blum, A., and Mitchell, T. (1998, January 24\u201326). Combining labeled and unlabeled data with co-training. Proceedings of the Eleventh Annual Conference on Computational Learning Theory, Madison, WI, USA.","DOI":"10.1145\/279943.279962"},{"key":"ref_102","unstructured":"Dhillon, P., Foster, D.P., and Ungar, L.H. (2011, January 12\u201317). Multi-view learning of word embeddings via CCA. Proceedings of the 2011 Advances in Neural Information Processing Systems, Granada, Spain."},{"key":"ref_103","unstructured":"Kumar, A., and Daum\u00e9, H. (July, January 28). A co-training approach for multi-view spectral clustering. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA."},{"key":"ref_104","first-page":"2211","article-title":"Multiple kernel learning algorithms","volume":"12","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_105","unstructured":"Jia, Y., Salzmann, M., and Darrell, T. (2010, January 24\u201325). Factorized latent spaces with structured sparsity. Proceedings of the Advances in Neural Information Processing Systems, Brno, Czech."},{"key":"ref_106","unstructured":"Vapnik, V. (2013). The Nature of Statistical Learning Theory, Springer Science & Business Media."},{"key":"ref_107","doi-asserted-by":"crossref","first-page":"1611","DOI":"10.1162\/NECO_a_00961","article-title":"The deterministic information bottleneck","volume":"26","author":"Strouse","year":"2017","journal-title":"Mass. Inst. Tech. Neural Comput."},{"key":"ref_108","doi-asserted-by":"crossref","first-page":"5112","DOI":"10.1109\/TCOMM.2018.2846248","article-title":"Oblivious Fronthaul-Constrained Relay for a Gaussian Channel","volume":"66","author":"Homri","year":"2018","journal-title":"IEEE Trans. Commun."},{"key":"ref_109","doi-asserted-by":"crossref","unstructured":"du Pin Calmon, F., and Fawaz, N. (2012, January 1\u20135). Privacy against statistical inference. Proceedings of the 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA.","DOI":"10.1109\/Allerton.2012.6483382"},{"key":"ref_110","first-page":"5788","article-title":"Robust Uplink Communications over Fading Channels with Variable Backhaul Connectivity","volume":"12","author":"Karasik","year":"2013","journal-title":"IEEE Trans. Commun."},{"key":"ref_111","doi-asserted-by":"crossref","first-page":"4739","DOI":"10.1109\/TIT.2014.2323406","article-title":"Channel capacity under sub-Nyquist nonuniform sampling","volume":"60","author":"Chen","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1109\/MSP.2017.2774249","article-title":"Analog-to-Digital Compression: A New Paradigm for Converting Signals to Bits","volume":"35","author":"Kipnis","year":"2018","journal-title":"IEEE Signal Process. Mag."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/151\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:44:45Z","timestamp":1760363085000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/151"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,27]]},"references-count":112,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["e22020151"],"URL":"https:\/\/doi.org\/10.3390\/e22020151","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,27]]}}}