{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T20:40:10Z","timestamp":1778704810916,"version":"3.51.4"},"reference-count":42,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T00:00:00Z","timestamp":1697846400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Swiss National Science Foundation (SNSF) Sinergia","doi-asserted-by":"publisher","award":["CRSII5_193716"],"award-info":[{"award-number":["CRSII5_193716"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001711","name":"Swiss National Science Foundation (SNSF) Sinergia","doi-asserted-by":"publisher","award":["200021_182063"],"award-info":[{"award-number":["200021_182063"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001711","name":"SNSF","doi-asserted-by":"publisher","award":["CRSII5_193716"],"award-info":[{"award-number":["CRSII5_193716"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001711","name":"SNSF","doi-asserted-by":"publisher","award":["200021_182063"],"award-info":[{"award-number":["200021_182063"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We present a novel information-theoretic framework, termed as TURBO, designed to systematically analyse and generalise auto-encoding methods. We start by examining the principles of information bottleneck and bottleneck-based networks in the auto-encoding setting and identifying their inherent limitations, which become more prominent for data with multiple relevant, physics-related representations. The TURBO framework is then introduced, providing a comprehensive derivation of its core concept consisting of the maximisation of mutual information between various data representations expressed in two directions reflecting the information flows. We illustrate that numerous prevalent neural network models are encompassed within this framework. The paper underscores the insufficiency of the information bottleneck concept in elucidating all such models, thereby establishing TURBO as a preferable theoretical reference. The introduction of TURBO contributes to a richer understanding of data representation and the structure of neural network models, enabling more efficient and versatile applications.<\/jats:p>","DOI":"10.3390\/e25101471","type":"journal-article","created":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T12:59:01Z","timestamp":1697893141000},"page":"1471","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["TURBO: The Swiss Knife of Auto-Encoders"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2957-3449","authenticated-orcid":false,"given":"Guillaume","family":"Qu\u00e9tant","sequence":"first","affiliation":[{"name":"Centre Universitaire d\u2019Informatique, Universit\u00e9 de Gen\u00e8ve, Route de Drize 7, CH-1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6461-734X","authenticated-orcid":false,"given":"Yury","family":"Belousov","sequence":"additional","affiliation":[{"name":"Centre Universitaire d\u2019Informatique, Universit\u00e9 de Gen\u00e8ve, Route de Drize 7, CH-1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5301-9141","authenticated-orcid":false,"given":"Vitaliy","family":"Kinakh","sequence":"additional","affiliation":[{"name":"Centre Universitaire d\u2019Informatique, Universit\u00e9 de Gen\u00e8ve, Route de Drize 7, CH-1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0416-9674","authenticated-orcid":false,"given":"Slava","family":"Voloshynovskiy","sequence":"additional","affiliation":[{"name":"Centre Universitaire d\u2019Informatique, Universit\u00e9 de Gen\u00e8ve, Route de Drize 7, CH-1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,21]]},"reference":[{"key":"ref_1","unstructured":"Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. arXiv."},{"key":"ref_2","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv."},{"key":"ref_3","unstructured":"Rezende, D.J., Mohamed, S., and Wierstra, D. (2014, January 21\u201326). Stochastic backpropagation and approximate inference in deep generative models. Proceedings of the International Conference on Machine Learning, PMLR, Beijing, China."},{"key":"ref_4","unstructured":"Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B. (2015). Adversarial Autoencoders. arXiv."},{"key":"ref_5","unstructured":"Tishby, N., and Zaslavsky, N. (May, January 26). Deep learning and the information bottleneck principle. Proceedings of the IEEE Information Theory Workshop, Jerusalem, Israel."},{"key":"ref_6","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2017, January 24\u201326). Deep Variational Information Bottleneck. Proceedings of the International Conference on Learning Representations, Toulon, France."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Voloshynovskiy, S., Taran, O., Kondah, M., Holotyak, T., and Rezende, D. (2020). Variational Information Bottleneck for Semi-Supervised Classification. Entropy, 22.","DOI":"10.3390\/e22090943"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2225","DOI":"10.1109\/TPAMI.2019.2909031","article-title":"Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle","volume":"42","author":"Amjad","year":"2019","journal-title":"IEEE Trans. Pattern. Anal. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"U\u011fur, Y., Arvanitakis, G., and Zaidi, A. (2020). Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding. Entropy, 22.","DOI":"10.3390\/e22020213"},{"key":"ref_10","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the Thirty-Seventh Annual Allerton Conference on Communication, Control and Computing, Monticello, IL, USA."},{"key":"ref_11","unstructured":"Cover, T.M. (1999). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_12","unstructured":"Voloshynovskiy, S., Kondah, M., Rezaeifar, S., Taran, O., Hotolyak, T., and Rezende, D.J. (2019, January 13). Information bottleneck through variational glasses. Proceedings of the Workshop on Bayesian Deep Learning, NeurIPS, Vancouver, Canada."},{"key":"ref_13","unstructured":"Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. (2021, January 18\u201324). Barlow twins: Self-supervised learning via redundancy reduction. Proceedings of the International Conference on Machine Learning, PMLR, Virtually."},{"key":"ref_14","unstructured":"Shwartz-Ziv, R., and LeCun, Y. (2023). To Compress or Not to Compress\u2013Self-Supervised Learning and Information Theory: A Review. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21\u201326). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.632"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22\u201329). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.244"},{"key":"ref_17","unstructured":"Rezende, D., and Mohamed, S. (2015, January 6\u201311). Variational inference with normalizing flows. Proceedings of the International Conference on Machine Learning, PMLR, Lille, France."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Pidhorskyi, S., Adjeroh, D.A., and Doretto, G. (2020, January 14\u201319). Adversarial latent autoencoders. Proceedings of the Conference on Computer Vision and Pattern Recognition, IEEE\/CVF, Virtually.","DOI":"10.1109\/CVPR42600.2020.01411"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2897","DOI":"10.1109\/TPAMI.2017.2784440","article-title":"Information Dropout: Learning Optimal Representations Through Noisy Computation","volume":"40","author":"Achille","year":"2018","journal-title":"IEEE Trans. Pattern. Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2060","DOI":"10.1109\/TIFS.2023.3262112","article-title":"Bottlenecks CLUB: Unifying Information-Theoretic Trade-Offs Among Complexity, Leakage, and Utility","volume":"18","author":"Razeghi","year":"2023","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tian, Y., Pang, G., Liu, Y., Wang, C., Chen, Y., Liu, F., Singh, R., Verjans, J.W., Wang, M., and Carneiro, G. (2022). Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder. arXiv.","DOI":"10.1007\/978-3-031-45676-3_2"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"172","DOI":"10.59275\/j.melba.2023-18c1","article-title":"Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection","volume":"2","author":"Patel","year":"2023","journal-title":"J. Mach. Learn. Biomed. Imaging"},{"key":"ref_23","unstructured":"Golling, T., Nobe, T., Proios, D., Raine, J.A., Sengupta, D., Voloshynovskiy, S., Arguin, J.F., Martin, J.L., Pilette, J., and Gupta, D.B. (2023). The Mass-ive Issue: Anomaly Detection in Jet Physics. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1007\/s41781-021-00056-0","article-title":"Getting high: High fidelity simulation of high granularity calorimeters with high speed","volume":"5","author":"Buhmann","year":"2021","journal-title":"Comput. Softw. Big Sci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"025014","DOI":"10.1088\/2632-2153\/ac7848","article-title":"Hadrons, better, faster, stronger","volume":"3","author":"Buhmann","year":"2022","journal-title":"Mach. Learn. Sci. Technol."},{"key":"ref_26","unstructured":"Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017, January 24\u201326). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. Proceedings of the International Conference on Learning Representations, Toulon, France."},{"key":"ref_27","unstructured":"Zhao, S., Song, J., and Ermon, S. (2017). InfoVAE: Information Maximizing Variational Autoencoders. arXiv."},{"key":"ref_28","unstructured":"Mohamed, S., and Lakshminarayanan, B. (2016). Learning in Implicit Generative Models. arXiv."},{"key":"ref_29","unstructured":"Larsen, A.B.L., S\u00f8nderby, S.K., Larochelle, H., and Winther, O. (2016, January 19\u201324). Autoencoding beyond pixels using a learned similarity metric. Proceedings of the International Conference on Machine Learning, PMLR, New York, NY, USA."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"7567","DOI":"10.1038\/s41598-022-10966-7","article-title":"Learning to simulate high energy particle collisions from unlabeled data","volume":"12","author":"Howard","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_31","unstructured":"Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6\u201311). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ledig, C., Theis, L., Husz\u00e1r, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21\u201326). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.19"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Karras, T., Laine, S., and Aila, T. (2019, January 15\u201320). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the Conference on Computer Vision and Pattern Recognition, IEEE\/CVF, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00453"},{"key":"ref_34","unstructured":"Brock, A., Donahue, J., and Simonyan, K. (2018). Large Scale GAN Training for High Fidelity Natural Image Synthesis. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Sauer, A., Schwarz, K., and Geiger, A. (2022, January 8\u201311). StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets. Proceedings of the SIGGRAPH Conference. ACM, Vancouver, BC, Canada.","DOI":"10.1145\/3528233.3530738"},{"key":"ref_36","unstructured":"(2023, August 29). Image Generation on ImageNet 256 \u00d7 256. Available online: https:\/\/paperswithcode.com\/sota\/image-generation-on-imagenet-256x256."},{"key":"ref_37","unstructured":"(2023, August 29). Image Generation on FFHQ 256 \u00d7 256. Available online: https:\/\/paperswithcode.com\/sota\/image-generation-on-ffhq-256-x-256."},{"key":"ref_38","first-page":"1","article-title":"Normalizing Flows for Probabilistic Modeling and Inference","volume":"22","author":"Papamakarios","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref_39","unstructured":"Qu\u00e9tant, G., Drozdova, M., Kinakh, V., Golling, T., and Voloshynovskiy, S. (2021, January 13). Turbo-Sim: A generalised generative model with a physical latent space. Proceedings of the Workshop on Machine Learning and the Physical Sciences, NeurIPS, Virtually."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"074","DOI":"10.21468\/SciPostPhys.9.5.074","article-title":"Invertible networks or partons to detector and back again","volume":"9","author":"Bellagente","year":"2020","journal-title":"SciPost Phys."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Belousov, Y., Pulfer, B., Chaban, R., Tutt, J., Taran, O., Holotyak, T., and Voloshynovskiy, S. (2022, January 12\u201316). Digital twins of physical printing-imaging channel. Proceedings of the IEEE International Workshop on Information Forensics and Security, Virtually.","DOI":"10.1109\/WIFS55849.2022.9975439"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"861","DOI":"10.21105\/joss.00861","article-title":"UMAP: Uniform Manifold Approximation and Projection","volume":"3","author":"McInnes","year":"2018","journal-title":"J. Open Source Softw."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1471\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:09:30Z","timestamp":1760130570000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1471"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,21]]},"references-count":42,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["e25101471"],"URL":"https:\/\/doi.org\/10.3390\/e25101471","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,21]]}}}