{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T19:24:30Z","timestamp":1775935470794,"version":"3.50.1"},"reference-count":45,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2023,6,3]],"date-time":"2023-06-03T00:00:00Z","timestamp":1685750400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005416","name":"The Research Council of Norway (RCN)","doi-asserted-by":"publisher","award":["309439"],"award-info":[{"award-number":["309439"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"The Research Council of Norway (RCN)","doi-asserted-by":"publisher","award":["315029"],"award-info":[{"award-number":["315029"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"The Research Council of Norway (RCN)","doi-asserted-by":"publisher","award":["303514"],"award-info":[{"award-number":["303514"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"RCN FRIPRO","doi-asserted-by":"publisher","award":["309439"],"award-info":[{"award-number":["309439"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"RCN FRIPRO","doi-asserted-by":"publisher","award":["315029"],"award-info":[{"award-number":["315029"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"RCN FRIPRO","doi-asserted-by":"publisher","award":["303514"],"award-info":[{"award-number":["303514"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"RCN IKTPLUSS","doi-asserted-by":"publisher","award":["309439"],"award-info":[{"award-number":["309439"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"RCN IKTPLUSS","doi-asserted-by":"publisher","award":["315029"],"award-info":[{"award-number":["315029"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"RCN IKTPLUSS","doi-asserted-by":"publisher","award":["303514"],"award-info":[{"award-number":["303514"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]},{"name":"UiT Thematic Initiative","award":["309439"],"award-info":[{"award-number":["309439"]}]},{"name":"UiT Thematic Initiative","award":["315029"],"award-info":[{"award-number":["315029"]}]},{"name":"UiT Thematic Initiative","award":["303514"],"award-info":[{"award-number":["303514"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs\u2019 generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input\/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based R\u00e9nyi\u2019s entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.<\/jats:p>","DOI":"10.3390\/e25060899","type":"journal-article","created":{"date-parts":[[2023,6,5]],"date-time":"2023-06-05T02:57:47Z","timestamp":1685933867000},"page":"899","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1395-7154","authenticated-orcid":false,"given":"Kristoffer K.","family":"Wickstr\u00f8m","sequence":"first","affiliation":[{"name":"Machine Learning Group, Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Troms\u00f8, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1953-4315","authenticated-orcid":false,"given":"Sigurd","family":"L\u00f8kse","sequence":"additional","affiliation":[{"name":"Machine Learning Group, Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Troms\u00f8, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7699-0405","authenticated-orcid":false,"given":"Michael C.","family":"Kampffmeyer","sequence":"additional","affiliation":[{"name":"Machine Learning Group, Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Troms\u00f8, Norway"},{"name":"Norwegian Computing Center, Department of Statistical Analysis and Machine Learning, 114 Blindern, NO-0314 Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6385-1705","authenticated-orcid":false,"given":"Shujian","family":"Yu","sequence":"additional","affiliation":[{"name":"Machine Learning Group, Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Troms\u00f8, Norway"},{"name":"Computational NeuroEngineering Laboratory, Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, USA"},{"name":"Department of Computer Science, Vrije Universiteit Amsterdam, 1081 HV Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3449-3531","authenticated-orcid":false,"given":"Jos\u00e9 C.","family":"Pr\u00edncipe","sequence":"additional","affiliation":[{"name":"Computational NeuroEngineering Laboratory, Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7496-8474","authenticated-orcid":false,"given":"Robert","family":"Jenssen","sequence":"additional","affiliation":[{"name":"Machine Learning Group, Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Troms\u00f8, Norway"},{"name":"Norwegian Computing Center, Department of Statistical Analysis and Machine Learning, 114 Blindern, NO-0314 Oslo, Norway"},{"name":"Department of Computer Science, University of Copenhagen, Universitetsparken 1, 2100 Copenhagen, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,3]]},"reference":[{"key":"ref_1","unstructured":"Shwartz-Ziv, R., and Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"7039","DOI":"10.1109\/TNNLS.2021.3089037","article-title":"On Information Plane Analyses of Neural Network Classifiers\u2014A Review","volume":"33","author":"Geiger","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Cheng, H., Lian, D., Gao, S., and Geng, Y. (2018, January 8\u201314). Evaluating Capability of Deep Neural Networks for Image Classification via Information Plane. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01252-6_11"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Noshad, M., Zeng, Y., and Hero, A.O. (2019, January 12\u201317). Scalable Mutual Information Estimation Using Dependence Graphs. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8683351"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"124020","DOI":"10.1088\/1742-5468\/ab3985","article-title":"On the information bottleneck theory of deep learning","volume":"2019","author":"Saxe","year":"2019","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1109\/TNNLS.2020.2968509","article-title":"Understanding Convolutional Neural Network Training with Information Theory","volume":"32","author":"Yu","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.neunet.2019.05.003","article-title":"Understanding autoencoders with information theoretic concepts","volume":"117","author":"Yu","year":"2019","journal-title":"Neural Netw."},{"key":"ref_8","unstructured":"Lorenzen, S.S., Igel, C., and Nielsen, M. (2022, January 25\u201329). Information Bottleneck: Exact Analysis of (Quantized) Neural Networks. Proceedings of the International Conference on Learning Representations, Virtual."},{"key":"ref_9","unstructured":"Goldfeld, Z., Van Den Berg, E., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., and Polyanskiy, Y. (2019, January 9\u201315). Estimating Information Flow in Deep Neural Networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_10","unstructured":"Chelombiev, I., Houghton, C., and O\u2019Donnell, C. (2019). Adaptive Estimators Show Information Compression in Deep Neural Networks. arXiv."},{"key":"ref_11","unstructured":"Zhouyin, Z., and Liu, D. (2021). Understanding Neural Networks with Logarithm Determinant Entropy Estimator. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1109\/TIT.2014.2370058","article-title":"Measures of Entropy From Data Using Infinitely Divisible Kernels","volume":"61","author":"Rao","year":"2015","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tishby, N., and Zaslavsky, N. (May, January 26). Deep learning and the information bottleneck principle. Proceedings of the 2015 IEEE Information Theory Workshop (ITW), Jerusalem, Israel.","DOI":"10.1109\/ITW.2015.7133169"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kolchinsky, A., Tracey, B.D., and Kuyk, S.V. (2019). Caveats for information bottleneck in deterministic scenarios. arXiv.","DOI":"10.3390\/e21121181"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2225","DOI":"10.1109\/TPAMI.2019.2909031","article-title":"Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle","volume":"42","author":"Amjad","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","unstructured":"Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019, January 10\u201315). Similarity of Neural Network Representations Revisited. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"J\u00f3nsson, H., Cherubini, G., and Eleftheriou, E. (2020). Convergence Behavior of DNNs with Mutual-Information-Based Regularization. Entropy, 22.","DOI":"10.3390\/e22070727"},{"key":"ref_18","unstructured":"Dy, J., and Krause, A. (2018, January 10\u201315). Mutual Information Neural Estimation. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Geiger, B.C., and Kubin, G. (2020). Information Bottleneck: Theory and Applications in Deep Learning. Entropy, 22.","DOI":"10.3390\/e22121408"},{"key":"ref_20","first-page":"2960","article-title":"Multivariate Extension of Matrix-based Renyi\u2019s \u03b1-order Entropy Functional","volume":"42","author":"Yu","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Landsverk, M.C., and Riemer-S\u00f8rensen, S. (2022, January 10\u201311). Mutual information estimation for graph convolutional neural networks. Proceedings of the 3rd Northern Lights Deep Learning Workshop, Tromso, Norway.","DOI":"10.7557\/18.6257"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1080\/00029890.2006.11920300","article-title":"Infinitely Divisible Matrices","volume":"113","author":"Bhatia","year":"2006","journal-title":"Am. Math. Mon."},{"key":"ref_23","unstructured":"Renyi, A. On Measures of Entropy and Information. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Nielsen, M.A., and Chuang, I.L. (2011). Quantum Computation and Quantum Information, Cambridge University Press. [10th ed.].","DOI":"10.1017\/CBO9780511976667"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2474","DOI":"10.1109\/TIT.2011.2110050","article-title":"On the Quantum R\u00e9nyi Relative Entropies and Related Capacity Formulas","volume":"57","author":"Mosonyi","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1667","DOI":"10.1109\/TPAMI.2002.1114861","article-title":"Input feature selection by mutual information based on Parzen window","volume":"24","author":"Kwak","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","first-page":"723","article-title":"A Kernel Two-sample Test","volume":"13","author":"Gretton","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_28","unstructured":"Muandet, K., Fukumizu, K., Sriperumbudur, B., and Sch\u00f6lkopf, B. (2017). Foundations and Trends\u00ae in Machine Learning, Now Foundations and Trends."},{"key":"ref_29","unstructured":"Fukumizu, K., Gretton, A., Sun, X., and Sch\u00f6lkopf, B. (2008, January 3\u20136). Kernel Measures of Conditional Dependence. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Hutter, M., Servedio, R.A., and Takimoto, E. (2007, January 1\u20134). A Hilbert Space Embedding for Distributions. Proceedings of the Algorithmic Learning Theory, Sendai, Japan.","DOI":"10.1007\/978-3-540-75225-7"},{"key":"ref_31","first-page":"1203","article-title":"A Computationally Efficient Estimator for Mutual Information","volume":"464","author":"Evans","year":"2008","journal-title":"Proc. Math. Phys. Eng. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.neunet.2011.05.011","article-title":"A kernel-based framework to tensorial data analysis","volume":"34","author":"Signoretto","year":"2011","journal-title":"Neural Netw."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"888","DOI":"10.1109\/34.868688","article-title":"Normalized Cuts and Image Segmentation","volume":"22","author":"Shi","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"3960","DOI":"10.1214\/09-AOS700","article-title":"Data spectroscopy: Eigenspaces of convolution operators and clustering","volume":"37","author":"Shi","year":"2009","journal-title":"Ann. Stat."},{"key":"ref_35","unstructured":"Silverman, B.W. (1986). Density Estimation for Statistics and Data Analysis, CRC Press."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Cristianini, N., Shawe-Taylor, J., Elisseeff, A., and Kandola, J.S. (2002, January 3\u20136). On kernel-target alignment. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.","DOI":"10.7551\/mitpress\/1120.003.0052"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"15849","DOI":"10.1073\/pnas.1903070116","article-title":"Reconciling modern machine-learning practice and the classical bias\u2013variance trade-off","volume":"116","author":"Belkin","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2020). Deep Double Descent: Where Bigger Models and More Data Hurt. arXiv.","DOI":"10.1088\/1742-5468\/ac3a74"},{"key":"ref_40","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley-Interscience."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Yu, X., Yu, S., and Pr\u00edncipe, J.C. (2021, January 6\u201311). Deep Deterministic Information Bottleneck with Matrix-Based Entropy Functional. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, ON, Canada.","DOI":"10.1109\/ICASSP39728.2021.9414151"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7\u201313). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.123"},{"key":"ref_43","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the International Conference on Artificial Intelligence and Statistics, Sardinia, Italy."},{"key":"ref_44","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2023, January 01). Automatic Differentiation in PyTorch. Available online: https:\/\/pytorch.org."},{"key":"ref_45","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 6\u201311). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the International Conference on Machine Learning, Lille, France."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/6\/899\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:47:53Z","timestamp":1760125673000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/6\/899"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,3]]},"references-count":45,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["e25060899"],"URL":"https:\/\/doi.org\/10.3390\/e25060899","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,3]]}}}