{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,11]],"date-time":"2026-01-11T00:33:09Z","timestamp":1768091589847,"version":"3.49.0"},"reference-count":37,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,6,3]],"date-time":"2021-06-03T00:00:00Z","timestamp":1622678400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Information plane analysis, describing the mutual information between the input and a hidden layer and between a hidden layer and the target over time, has recently been proposed to analyze the training of neural networks. Since the activations of a hidden layer are typically continuous-valued, this mutual information cannot be computed analytically and must thus be estimated, resulting in apparently inconsistent or even contradicting results in the literature. The goal of this paper is to demonstrate how information plane analysis can still be a valuable tool for analyzing neural network training. To this end, we complement the prevailing binning estimator for mutual information with a geometric interpretation. With this geometric interpretation in mind, we evaluate the impact of regularization and interpret phenomena such as underfitting and overfitting. In addition, we investigate neural network learning in the presence of noisy data and noisy labels.<\/jats:p>","DOI":"10.3390\/e23060711","type":"journal-article","created":{"date-parts":[[2021,6,3]],"date-time":"2021-06-03T21:03:32Z","timestamp":1622754212000},"page":"711","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A Geometric Perspective on Information Plane Analysis"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9206-2588","authenticated-orcid":false,"given":"Mina","family":"Basirat","sequence":"first","affiliation":[{"name":"Institute of Computer Graphics and Vision, Graz University of Technology, Inffeldgasse 16\/II, 8010 Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3257-743X","authenticated-orcid":false,"given":"Bernhard C.","family":"Geiger","sequence":"additional","affiliation":[{"name":"Know-Center GmbH, Inffeldgasse 13, 8010 Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9566-1298","authenticated-orcid":false,"given":"Peter M.","family":"Roth","sequence":"additional","affiliation":[{"name":"International AI Future Lab, Technical University of Munich (TUM), Willy-Messerschmitt-Stra\u00dfe 1, 85521 Taufkirchen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,3]]},"reference":[{"key":"ref_1","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_3","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). Imagenet Classification with Deep Convolutional Neural Networks. Proceedings of the Neural Information Processing Systems 2012, Lake Tahoe, CA, USA."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep learning in neural networks: An overview","volume":"61","author":"Schmidhuber","year":"2015","journal-title":"Neural Netw."},{"key":"ref_5","unstructured":"Erhan, D., Bengio, Y., Courville, A., and Vincent, P. (2009). Visualizing Higher-Layer Features of a Deep Network, University of Montreal. Technical Report."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2013). Visualizing and Understanding Convolutional Networks. arXiv.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_7","unstructured":"Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv."},{"key":"ref_8","unstructured":"Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H. (2015). Understanding Neural Networks Through Deep Visualization. arXiv."},{"key":"ref_9","unstructured":"Nguyen, A., Yosinski, J., and Clune, J. (2016). Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. arXiv.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27\u201330). Learning Deep Features for Discriminative Localization. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.319"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_13","unstructured":"Zintgraf, L.M., Cohen, T.S., Adel, T., and Welling, M. (2017). Visualizing Deep Neural Network Decisions: Prediction Difference Analysis. arXiv."},{"key":"ref_14","unstructured":"Shwartz-Ziv, R., and Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. arXiv."},{"key":"ref_15","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (1999, January 22\u201324). The Information Bottleneck Method. Proceedings of the Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA."},{"key":"ref_16","unstructured":"Saxe, A.M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B.D., and Cox, D.D. (2000). On the Information Bottleneck Theory of Deep Learning. arXiv."},{"key":"ref_17","unstructured":"Geiger, B.C. (2020). On Information Plane Analyses of Neural Network Classifiers\u2014A Review. arXiv."},{"key":"ref_18","unstructured":"Goldfeld, Z., Van Den Berg, E., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., and Polyanskiy, Y. (2018). Estimating Information Flow in Deep Neural Networks. arXiv."},{"key":"ref_19","unstructured":"Schiemer, M., and Ye, J. (2021, May 30). Revisiting the Information Plane. Available online: https:\/\/openreview.net\/forum?id=Hyljn1SFwr."},{"key":"ref_20","unstructured":"Chelombiev, I., Houghton, C.J., and O\u2019Donnel, C. (2019). Adaptive Estimators show Information Compression in Deep Neural Networks. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Fang, H., Wang, V., and Yamaguchi, M. (2018). Dissecting Deep Learning Networks\u2014Visualizing Mutual Information. Entropy, 20.","DOI":"10.3390\/e20110823"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Elad, A., Haviv, D., Blau, Y., and Michaeli, T. (2019, January 27\u201328). Direct Validation of the Information Bottleneck Principle for Deep Nets. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00099"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Noshad, M., Zeng, Y., and Hero, A.O. (2019, January 12\u201317). Scalable Mutual Information Estimation Using Dependence Graphs. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8683351"},{"key":"ref_24","unstructured":"Wickstr\u00f8m, K., L\u00f8kse, S., Kampffmeyer, M., Yu, S., Principe, J., and Jenssen, R. (2019). Information Plane Analysis of Deep Neural Networks via Matrix-Based R\u00e9nyi\u2019s Entropy and Tensor Kernels. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2225","DOI":"10.1109\/TPAMI.2019.2909031","article-title":"Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle","volume":"42","author":"Amjad","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","unstructured":"Cover, T.M., and Thomas, J.A. (1991). Elements of Information Theory, Wiley Interscience."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_28","unstructured":"Mu, N., and Gilmer, J. (2019). MNIST-C: A Robustness Benchmark for Computer Vision. arXiv."},{"key":"ref_29","unstructured":"Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Maharaj, T., Fischer, A., Courville, A.C., and Bengio, Y. (2017, January 6\u201311). A Closer Look at Memorization in Deep Networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_30","unstructured":"Northcutt, C.G., Athalye, A., and Mueller, J. (2021). Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"845","DOI":"10.1109\/TNNLS.2013.2292894","article-title":"Classification in the Presence of Label Noise: A Survey","volume":"25","author":"Frenay","year":"2014","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"ref_32","unstructured":"Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. arXiv."},{"key":"ref_33","unstructured":"Jiang, L., Zhou, Z., Leung, T., Li, L.J., and Fei-Fei, L. (2018, January 10\u201315). MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Patrini, G., Rozza, A., Menon, A., Nock, R., and Qu, L. (2017, January 21\u201326). Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.240"},{"key":"ref_35","unstructured":"Natarajan, N., Dhillon, I.S., Ravikumar, P.K., and Tewari, A. (2013, January 5\u201310). Learning with Noisy Labels. Proceedings of the Neural Information Processing Systems, Lake Tahoe, CA, USA."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Zhang, H., Arik, S.O., Lee, H., and Pfister, T. (2020, January 13\u201319). Distilling Effective Supervision From Severe Label Noise. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00931"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ma, X., Wang, Y., Houle, M.E., Zhou, S., Erfani, S.M., Xia, S., Wijewickrema, S.N.R., and Bailey, J. (2018, January 10\u201315). Dimensionality-Driven Learning with Noisy Labels. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.","DOI":"10.1109\/CVPR.2018.00906"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/6\/711\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:10:41Z","timestamp":1760163041000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/6\/711"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,3]]},"references-count":37,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["e23060711"],"URL":"https:\/\/doi.org\/10.3390\/e23060711","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,3]]}}}