{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:20:38Z","timestamp":1760239238222,"version":"build-2065373602"},"reference-count":51,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2020,10,30]],"date-time":"2020-10-30T00:00:00Z","timestamp":1604016000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["407540_167158","200021_182063"],"award-info":[{"award-number":["407540_167158","200021_182063"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We present a new decentralized classification system based on a distributed architecture. This system consists of distributed nodes, each possessing their own datasets and computing modules, along with a centralized server, which provides probes to classification and aggregates the responses of nodes for a final decision. Each node, with access to its own training dataset of a given class, is trained based on an auto-encoder system consisting of a fixed data-independent encoder, a pre-trained quantizer and a class-dependent decoder. Hence, these auto-encoders are highly dependent on the class probability distribution for which the reconstruction distortion is minimized. Alternatively, when an encoding\u2013quantizing\u2013decoding node observes data from different distributions, unseen at training, there is a mismatch, and such a decoding is not optimal, leading to a significant increase of the reconstruction distortion. The final classification is performed at the centralized classifier that votes for the class with the minimum reconstruction distortion. In addition to the system applicability for applications facing big-data communication problems and or requiring private classification, the above distributed scheme creates a theoretical bridge to the information bottleneck principle. The proposed system demonstrates a very promising performance on basic datasets such as MNIST and FasionMNIST.<\/jats:p>","DOI":"10.3390\/e22111237","type":"journal-article","created":{"date-parts":[[2020,10,30]],"date-time":"2020-10-30T21:34:47Z","timestamp":1604093687000},"page":"1237","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Information Bottleneck Classification in Extremely Distributed Systems"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7179-005X","authenticated-orcid":false,"given":"Denis","family":"Ullmann","sequence":"first","affiliation":[{"name":"SIP\u2014Stochastic Information Processing Group, Computer Science Department CUI, University of Geneva, Route de Drize 7, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8103-3722","authenticated-orcid":false,"given":"Shideh","family":"Rezaeifar","sequence":"additional","affiliation":[{"name":"SIP\u2014Stochastic Information Processing Group, Computer Science Department CUI, University of Geneva, Route de Drize 7, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8537-5204","authenticated-orcid":false,"given":"Olga","family":"Taran","sequence":"additional","affiliation":[{"name":"SIP\u2014Stochastic Information Processing Group, Computer Science Department CUI, University of Geneva, Route de Drize 7, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taras","family":"Holotyak","sequence":"additional","affiliation":[{"name":"SIP\u2014Stochastic Information Processing Group, Computer Science Department CUI, University of Geneva, Route de Drize 7, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7096-7941","authenticated-orcid":false,"given":"Brandon","family":"Panos","sequence":"additional","affiliation":[{"name":"SIP\u2014Stochastic Information Processing Group, Computer Science Department CUI, University of Geneva, Route de Drize 7, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0416-9674","authenticated-orcid":false,"given":"Slava","family":"Voloshynovskiy","sequence":"additional","affiliation":[{"name":"SIP\u2014Stochastic Information Processing Group, Computer Science Department CUI, University of Geneva, Route de Drize 7, 1227 Carouge, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,10,30]]},"reference":[{"key":"ref_1","unstructured":"Delalleau, O., and Bengio, Y. (2007). Parallel Stochastic Gradient Descent, CIAR Summer School."},{"key":"ref_2","unstructured":"Tian, L., Jayaraman, B., Gu, Q., and Evans, D. (2016, January 8). Aggregating Private Sparse Learning Models Using Multi-Party Computation. Proceedings of the Private MultiParty Machine Learning (NIPS 2016 Workshop), Barcelona, Spain."},{"key":"ref_3","unstructured":"McMahan, H.B., Moore, E., Ramage, D., and y Arcas, B.A. (2016). Federated Learning of Deep Networks using Model Averaging. arXiv."},{"key":"ref_4","unstructured":"McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20\u201322). Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, Fort Lauderdale, FL, USA."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Oyallon, E., Belilovsky, E., and Zagoruyko, S. (2017). Scaling the Scattering Transform: Deep Hybrid Networks. arXiv.","DOI":"10.1109\/ICCV.2017.599"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tishby, N., and Zaslavsky, N. (May, January 26). Deep Learning and the Information Bottleneck Principle. Proceedings of the 2015 IEEE Information Theory Workshop (ITW), Jerusalem, Israel.","DOI":"10.1109\/ITW.2015.7133169"},{"key":"ref_7","unstructured":"Voloshynovskiy, S., Kondah, M., Rezaeifar, S., Taran, O., Holotyak, T., and Rezende, D.J. (2019, January 13). Information bottleneck through variational glasses. Proceedings of the Bayesian Deep Learning (NeurIPS 2019 Workshop), Vancouver, BC, Canada."},{"key":"ref_8","unstructured":"Gibiansky, A. (2020, October 28). Bringing HPC Techniques to Deep Learning. Available online: https:\/\/andrew.gibiansky.com\/blog\/machine-learning\/baidu-allreduce\/."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"You, S., Xu, C., Xu, C., and Tao, D. (2017, January 13\u201317). Learning from multiple teacher networks. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada.","DOI":"10.1145\/3097983.3098135"},{"key":"ref_10","unstructured":"Lin, T., Kong, L., Stich, S.U., and Jaggi, M. (2020). Ensemble Distillation for Robust Model Fusion in Federated Learning. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Asad, M., Moustafa, A., Ito, T., and Aslam, M. (2020). Evaluating the Communication Efficiency in Federated Learning Algorithms. arXiv.","DOI":"10.1109\/CSCWD49262.2021.9437738"},{"key":"ref_12","unstructured":"Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated Learning with Non-IID Data. arXiv."},{"key":"ref_13","unstructured":"Hsieh, K., Phanishayee, A., Mutlu, O., and Gibbons, P.B. (2019). The Non-IID Data Quagmire of Decentralized Machine Learning. arXiv."},{"key":"ref_14","unstructured":"Fung, C., Yoon, C.J.M., and Beschastnikh, I. (2018). Mitigating Sybils in Federated Learning Poisoning. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","article-title":"The MNIST Database of Handwritten Digit Images for Machine Learning Research","volume":"29","author":"Deng","year":"2012","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_16","unstructured":"Kingma, D.P., Rezende, D.J., Mohamed, S., and Welling, M. (2014). Semi-Supervised Learning with Deep Generative Models. arXiv."},{"key":"ref_17","unstructured":"Gordon, J., and Hern\u00e1ndez-Lobato, J.M. (2017). Bayesian Semisupervised Learning with Deep Generative Models. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1023\/B:MACH.0000008084.60811.49","article-title":"Support vector data description","volume":"54","author":"Tax","year":"2004","journal-title":"Mach. Learn."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sabokrou, M., Khalooei, M., Fathy, M., and Adeli, E. (2018). Adversarially Learned One-Class Classifier for Novelty Detection. arXiv.","DOI":"10.1109\/CVPR.2018.00356"},{"key":"ref_20","unstructured":"Pidhorskyi, S., Almohsen, R., Adjeroh, D.A., and Doretto, G. (2018). Generative Probabilistic Novelty Detection with Adversarial Autoencoders. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Perera, P., Nallapati, R., and Xiang, B. (2019). OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations. arXiv.","DOI":"10.1109\/CVPR.2019.00301"},{"key":"ref_22","unstructured":"Dewdney, P., Turner, W., Braun, R., Santander-Vela, J., Waterson, M., and Tan, G.H. (2015). SKA1 System Baselinev2 Description, SKA Organisation."},{"key":"ref_23","unstructured":"Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2016). Deep variational information bottleneck. arXiv."},{"key":"ref_24","unstructured":"Estella-Aguerri, I., and Zaidi, A. (2019). Distributed variational representation learning. IEEE Trans. Pattern Anal. Mach. Intell., 1."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Razeghi, B., Stanko, T., \u0160koric\u00b4, B., and Voloshynovskiy, S. (2019, January 9\u201312). Single-Component Privacy Guarantees in Helper Data Systems and Sparse Coding with Ambiguation. Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS), Delft, The Netherlands.","DOI":"10.1109\/WIFS47025.2019.9035095"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Chen, Y., Sun, X., and Jin, Y. (2019). Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation. arXiv.","DOI":"10.1109\/TNNLS.2019.2953131"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the Dimensionality of Data with Neural Networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1872","DOI":"10.1109\/TPAMI.2012.230","article-title":"Invariant scattering convolution networks","volume":"35","author":"Bruna","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Rezaeifar, S., Taran, O., and Voloshynovskiy, S. (2018). Classification by Re-generation: Towards Classification Based on Variational Inference. arXiv.","DOI":"10.23919\/EUSIPCO.2018.8553530"},{"key":"ref_30","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing), Wiley-Interscience."},{"key":"ref_31","unstructured":"Zhang, Y., Ozay, M., Sun, Z., and Okatani, T. (2017). Information Potential Auto-Encoders. arXiv."},{"key":"ref_32","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1331","DOI":"10.1002\/cpa.21413","article-title":"Group invariant scattering","volume":"65","author":"Mallat","year":"2012","journal-title":"Commun. Pure Appl. Math."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Bernstein, S., Bouchot, J.L., Reinhardt, M., and Heise, B. (2013). Generalized analytic signals in image processing: Comparison, theory and applications. Quaternion and Clifford Fourier Transforms and Wavelets, Birkh\u00e4user.","DOI":"10.1007\/978-3-0348-0603-9_11"},{"key":"ref_35","unstructured":"Zhao, H., Gallo, O., Frosio, I., and Kautz, J. (2015). Is L2 a Good Loss Function for Neural Networks for Image Processing. arXiv."},{"key":"ref_36","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ledig, C., Theis, L., Husz\u00e1r, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2016). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. arXiv.","DOI":"10.1109\/CVPR.2017.19"},{"key":"ref_38","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv."},{"key":"ref_39","unstructured":"Byerly, A., Kalganova, T., and Dear, I. (2020). A Branching and Merging Convolutional Network with Homogeneous Filter Capsules. arXiv."},{"key":"ref_40","unstructured":"Hirata, D., and Takahashi, N. (2020). Ensemble learning in CNN augmented with fully connected subnetworks. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kowsari, K., Heidarysafa, M., Brown, D.E., Meimandi, K.J., and Barnes, L.E. (2018). RMDL: Random Multimodel Deep Learning for Classification. arXiv.","DOI":"10.1109\/ICMLA.2017.0-134"},{"key":"ref_42","unstructured":"Harris, E., Marcu, A., Painter, M., Niranjan, M., Pr\u00fcgel-Bennett, A., and Hare, J. (2020). FMix: Enhancing Mixed Sample Data Augmentation. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Bhatnagar, S., Ghosal, D., and Kolekar, M.H. (2017, January 21\u201323). Classification of fashion article images using convolutional neural networks. Proceedings of the 2017 Fourth International Conference on Image Information Processing (ICIIP), Shimla, India.","DOI":"10.1109\/ICIIP.2017.8313740"},{"key":"ref_44","unstructured":"Hao, W., Mehta, N., Liang, K.J., Cheng, P., El-Khamy, M., and Carin, L. (2020). WAFFLe: Weight Anonymized Factorization for Federated Learning. arXiv."},{"key":"ref_45","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1038\/35016072","article-title":"Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit","volume":"405","author":"Hahnloser","year":"2000","journal-title":"Nature"},{"key":"ref_47","unstructured":"HasanPour, S.H., Rouhani, M., Fayyaz, M., and Sabokrou, M. (2016). Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures. arXiv."},{"key":"ref_48","unstructured":"Belghazi, M.I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R.D. (2018). MINE: Mutual Information Neural Estimation. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Quattoni, A., and Torralba, A. (2009, January 20\u201325). Recognizing indoor scenes. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206537"},{"key":"ref_51","unstructured":"Huang, G.B., Ramesh, M., Berg, T., and Learned-Miller, E. (2007). Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments Technical Report 07-49, University of Massachusetts."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/11\/1237\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:27:16Z","timestamp":1760178436000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/11\/1237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,30]]},"references-count":51,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2020,11]]}},"alternative-id":["e22111237"],"URL":"https:\/\/doi.org\/10.3390\/e22111237","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2020,10,30]]}}}