{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T10:47:03Z","timestamp":1776077223116,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T00:00:00Z","timestamp":1640822400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/T000783\/1"],"award-info":[{"award-number":["EP\/T000783\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Deep learning has proven to be an important element of modern data processing technology, which has found its application in many areas such as multimodal sensor data processing and understanding, data generation and anomaly detection. While the use of deep learning is booming in many real-world tasks, the internal processes of how it draws results is still uncertain. Understanding the data processing pathways within a deep neural network is important for transparency and better resource utilisation. In this paper, a method utilising information theoretic measures is used to reveal the typical learning patterns of convolutional neural networks, which are commonly used for image processing tasks. For this purpose, training samples, true labels and estimated labels are considered to be random variables. The mutual information and conditional entropy between these variables are then studied using information theoretical measures. This paper shows that more convolutional layers in the network improve its learning and unnecessarily higher numbers of convolutional layers do not improve the learning any further. The number of convolutional layers that need to be added to a neural network to gain the desired learning level can be determined with the help of theoretic information quantities including entropy, inequality and mutual information among the inputs to the network. The kernel size of convolutional layers only affects the learning speed of the network. This study also shows that where the dropout layer is applied to has no significant effects on the learning of networks with a lower dropout rate, and it is better placed immediately after the last convolutional layer with higher dropout rates.<\/jats:p>","DOI":"10.3390\/e24010067","type":"journal-article","created":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T21:41:21Z","timestamp":1640900481000},"page":"67","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Evaluating the Learning Procedure of CNNs through a Sequence of Prognostic Tests Utilising Information Theoretical Measures"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6174-3383","authenticated-orcid":false,"given":"Xiyu","family":"Shi","sequence":"first","affiliation":[{"name":"Institute for Digital Technologies, Loughborough University London, Queen Elizabeth Olympic Park, Here East, London E20 3BS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7535-141X","authenticated-orcid":false,"given":"Varuna","family":"De-Silva","sequence":"additional","affiliation":[{"name":"Institute for Digital Technologies, Loughborough University London, Queen Elizabeth Olympic Park, Here East, London E20 3BS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3340-2036","authenticated-orcid":false,"given":"Yusuf","family":"Aslan","sequence":"additional","affiliation":[{"name":"Institute for Digital Technologies, Loughborough University London, Queen Elizabeth Olympic Park, Here East, London E20 3BS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3759-4629","authenticated-orcid":false,"given":"Erhan","family":"Ekmekcioglu","sequence":"additional","affiliation":[{"name":"Institute for Digital Technologies, Loughborough University London, Queen Elizabeth Olympic Park, Here East, London E20 3BS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmet","family":"Kondoz","sequence":"additional","affiliation":[{"name":"Institute for Digital Technologies, Loughborough University London, Queen Elizabeth Olympic Park, Here East, London E20 3BS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Graves, A., Mohamed, A., and Hinton, G. (2013, January 26\u201331). Speech recognition with deep recurrent neural networks. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.","DOI":"10.1109\/ICASSP.2013.6638947"},{"key":"ref_2","unstructured":"Zhang, X., and LeCun, Y. (2015). Text Understanding from Scratch. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1109\/MSP.2012.2205597","article-title":"Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups","volume":"29","author":"Hinton","year":"2012","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Singh, A., Sengupta, S., and Lakshminarayanan, V. (2020). Explainable Deep Learning Models in Medical Image Analysis. J. Imaging, 6.","DOI":"10.3390\/jimaging6060052"},{"key":"ref_7","unstructured":"Alain, G., and Bengio, Y. (2020, September 01). Understanding Intermediate Layers Using Linear Classifier Probes. Available online: https:\/\/arxiv.org\/abs\/1610.01644v4."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Cheng, H., Lian, D., Gao, S., and Geng, Y. (2019). Utilizing Information Bottleneck to Evaluate the Capability of Deep Neural Networks for Image Classification. Entropy, 21.","DOI":"10.3390\/e21050456"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1109\/TNNLS.2020.2968509","article-title":"Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration","volume":"32","author":"Yu","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_10","unstructured":"Shwartz-Ziv, R., and Tishby, N. (2021, December 28). Opening the Black Box of Deep Neural Networks via Information. Available online: https:\/\/arxiv.org\/pdf\/1703.00810v3.pdf."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Tishby, N., and Zaslavsky, N. (May, January 26). Deep learning and the information bottleneck principle. Proceedings of the 2015 IEEE Information Theory Workshop (ITW), Jerusalem, Israel.","DOI":"10.1109\/ITW.2015.7133169"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Balda, E.R., Behboodi, A., and Mathar, R. (2018, January 17\u201319). An Information Theoretic View on Learning of Artificial Neural Networks. Proceedings of the 2018 12th International Conference on Signal Processing and Communication Systems (ICSPCS), Cairns, Australia.","DOI":"10.1109\/ICSPCS.2018.8631758"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1038\/538020a","article-title":"Can we open the black box of AI?","volume":"538","author":"Castelvecchi","year":"2016","journal-title":"Nature"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"42200","DOI":"10.1109\/ACCESS.2020.2976199","article-title":"Explainable Machine Learning for Scientific Insights and Discoveries","volume":"8","author":"Roscher","year":"2020","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Arrieta","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_16","first-page":"818","article-title":"Visualizing and Understanding Convolutional Networks","volume":"Volume 8689","author":"Fleet","year":"2014","journal-title":"Proceedings of the European Conference on Computer Vision 2014"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"157","DOI":"10.22266\/ijies2019.1231.15","article-title":"Driving Preference Metric-Aware Control for Self-Driving Vehicles","volume":"12","author":"Bae","year":"2019","journal-title":"IJIES"},{"key":"ref_18","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (2000). The Information Bottleneck Method. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"124014","DOI":"10.1088\/1742-5468\/ab3430","article-title":"Entropy and mutual information in models of deep neural networks","volume":"2019","author":"Manoel","year":"2019","journal-title":"J. Stat. Mech."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1088\/1742-5468\/ab3985","article-title":"On the information bottleneck theory of deep learning","volume":"2019","author":"Saxe","year":"2019","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.neunet.2019.05.003","article-title":"Understanding autoencoders with information theoretic concepts","volume":"117","author":"Yu","year":"2019","journal-title":"Neural Netw."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1\u20133). Explaining Explanations: An Overview of Interpretability of Machine Learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy.","DOI":"10.1109\/DSAA.2018.00018"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014, January 25\u201329). Learning Phrase Representations using RNN Encoder\u2013Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"ref_25","first-page":"44","article-title":"DARPA\u2019s Explainable Artificial Intelligence Program","volume":"40","author":"Gunning","year":"2019","journal-title":"AI Mag."},{"key":"ref_26","unstructured":"Doshi-Velez, F., and Kim, B. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv."},{"key":"ref_27","unstructured":"Preece, A., Harborne, D., Braines, D., Tomsett, R., and Chakraborty, S. (2018). Stakeholders in Explainable AI. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"22071","DOI":"10.1073\/pnas.1900654116","article-title":"Definitions, methods, and applications in interpretable machine learning","volume":"116","author":"Murdoch","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery","volume":"16","author":"Lipton","year":"2018","journal-title":"ACM Queue"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1002\/j.1538-7305.1948.tb00917.x","article-title":"A Mathematical Theory of Communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1002\/j.1538-7305.1949.tb00928.x","article-title":"Communication Theory of Secrecy Systems","volume":"28","author":"Shannon","year":"1949","journal-title":"Bell Syst. Tech. J."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley. [2nd ed.].","DOI":"10.1002\/047174882X"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"Lecun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_35","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv."},{"key":"ref_36","unstructured":"Chollet, F. (2021, September 17). Keras. Available online: https:\/\/keras.io\/about\/."},{"key":"ref_37","unstructured":"Google Brain Team (2020, June 01). TensorFlow. Google. Available online: https:\/\/github.com\/tensorflow\/tensorflow\/tree\/r2.3\/tensorflow\/python."},{"key":"ref_38","unstructured":"Shi, X., and De-Silva, V. (2021, December 26). Evaluate CNN with Info Theory. Available online: https:\/\/github.com\/shiluldn\/Evaluate_CNN_with_Info_Theory."},{"key":"ref_39","unstructured":"Wu, Z. (2020, September 06). Deep Learning-Deterministic Neural Networks, Presented at the MIT Techchnology Review 2013, 23 April 2013. Available online: http:\/\/3dvision.princeton.edu\/courses\/COS598\/2014sp\/slides\/lecture05_cnn\/lecture05_cnn.pdf."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/1\/67\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:55:56Z","timestamp":1760169356000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/1\/67"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,30]]},"references-count":39,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["e24010067"],"URL":"https:\/\/doi.org\/10.3390\/e24010067","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,30]]}}}