{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T12:46:08Z","timestamp":1791377168677,"version":"4.3.2"},"reference-count":28,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2020,1,16]],"date-time":"2020-01-16T00:00:00Z","timestamp":1579132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Current neural networks architectures are many times harder to train because of the increasing size and complexity of the used datasets. Our objective is to design more efficient training algorithms utilizing causal relationships inferred from neural networks. The transfer entropy (TE) was initially introduced as an information transfer measure used to quantify the statistical coherence between events (time series). Later, it was related to causality, even if they are not the same. There are only few papers reporting applications of causality or TE in neural networks. Our contribution is an information-theoretical method for analyzing information transfer between the nodes of feedforward neural networks. The information transfer is measured by the TE of feedback neural connections. Intuitively, TE measures the relevance of a connection in the network and the feedback amplifies this connection. We introduce a backpropagation type training algorithm that uses TE feedback connections to improve its performance.<\/jats:p>","DOI":"10.3390\/e22010102","type":"journal-article","created":{"date-parts":[[2020,1,17]],"date-time":"2020-01-17T04:14:41Z","timestamp":1579234481000},"page":"102","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Learning in Feedforward Neural Networks Accelerated by Transfer Entropy"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1393-2723","authenticated-orcid":false,"given":"Adrian","family":"Moldovan","sequence":"first","affiliation":[{"name":"Department of Electronics and Computers, Transilvania University, 500024 Bra\u015fov, Romania"},{"name":"Corporate Technology, Siemens SRL, 500007 Bra\u015fov, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3986-5437","authenticated-orcid":false,"given":"Angel","family":"Ca\u0163aron","sequence":"additional","affiliation":[{"name":"Department of Electronics and Computers, Transilvania University, 500024 Bra\u015fov, Romania"},{"name":"Corporate Technology, Siemens SRL, 500007 Bra\u015fov, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6015-3151","authenticated-orcid":false,"given":"R\u0103zvan","family":"Andonie","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Central Washington University, Ellensburg, WA 98926, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Marwala, T. (2015). Causality, Correlation and Artificial Intelligence for Rational Decision Making, World Scientific.","DOI":"10.1142\/9356"},{"key":"ref_2","unstructured":"Shadish, W., Cook, T., and Campbell, D. (2001). Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Houghton Mifflin."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2309","DOI":"10.3390\/e16042309","article-title":"Measures of Causality in Complex Datasets with Application to Financial Data","volume":"16","author":"Zaremba","year":"2014","journal-title":"Entropy"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Pearl, J. (2009). Causality: Models, Reasoning and Inference, Cambridge University Press. [2nd ed.].","DOI":"10.1017\/CBO9780511803161"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1103\/PhysRevLett.85.461","article-title":"Measuring Information Transfer","volume":"85","author":"Schreiber","year":"2000","journal-title":"Phys. Rev. Lett."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"605","DOI":"10.1140\/epjb\/e2010-00034-5","article-title":"Differentiating information transfer and causal effect","volume":"73","author":"Lizier","year":"2010","journal-title":"Eur. Phys. J. B"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Cataron, A., and Andonie, R. (2017, January 11\u201314). Transfer Information Energy: A Quantitative Causality Indicator Between Time Series. Proceedings of the Artificial Neural Networks and Machine Learning\u2014ICANN 2017\u201426th International Conference on Artificial Neural Networks, Alghero, Italy.","DOI":"10.1007\/978-3-319-68612-7_58"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ca\u0163aron, A., and Andonie, R. (2018). Transfer Information Energy: A Quantitative Indicator of Information Transfer between Time Series. Entropy, 20.","DOI":"10.3390\/e20050323"},{"key":"ref_9","unstructured":"Massey, J.L. (1990, January 27\u201330). Causality, feedback and directed information. Proceedings of the 1990 International Symp. on Information Theory and Its Applications, Hawaii, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1007\/s10827-010-0271-2","article-title":"Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity","volume":"30","author":"Lizier","year":"2011","journal-title":"J. Comput. Neurosci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1007\/s10827-010-0262-3","article-title":"Transfer entropy\u2014A model-free measure of effective connectivity for the neurosciences","volume":"30","author":"Vicente","year":"2011","journal-title":"J. Comput. Neurosci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3743","DOI":"10.1093\/cercor\/bhu252","article-title":"Functional Clusters, Hubs, and Communities in the Cortical Microconnectome","volume":"25","author":"Shimono","year":"2015","journal-title":"Cerebral Cortex"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Fang, H., Wang, V., and Yamaguchi, M. (2018). Dissecting Deep Learning Networks\u2014Visualizing Mutual Information. Entropy, 20.","DOI":"10.3390\/e20110823"},{"key":"ref_14","first-page":"193","article-title":"Improving Recurrent Neural Network Performance Using Transfer Entropy","volume":"Volume II","author":"Obst","year":"2010","journal-title":"Proceedings of the 17th International Conference on Neural Information Processing: Models and Applications (ICONIP 2010)"},{"key":"ref_15","unstructured":"Herzog, S., Tetzlaff, C., and W\u00f6rg\u00f6tter, F. (2017). Transfer entropy-based feedback improves performance in artificial neural networks. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1016\/S0893-6080(01)00127-7","article-title":"A methodology to explain neural network classification","volume":"15","year":"2002","journal-title":"Neur. Netw."},{"key":"ref_17","unstructured":"Patterson, J., and Gibson, A. (2017). Deep Learning: A Practitioner\u2019s Approach, O\u2019Reilly Media, Inc.. [1st ed.]."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Bossomaier, T., Barnett, L., Harr\u00e9, M., and Lizier, J.T. (2016). An Introduction to Transfer Entropy. Information Flow in Complex Systems, Springer.","DOI":"10.1007\/978-3-319-43222-9"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.physrep.2006.12.004","article-title":"Causality detection based on information-theoretic approaches in time series analysis","volume":"441","author":"Vejmelka","year":"2007","journal-title":"Phys. Rep."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/S0167-2789(02)00432-3","article-title":"Information transfer in continuous processes","volume":"166","author":"Kaiser","year":"2002","journal-title":"Phys. D"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"438","DOI":"10.3390\/e17010438","article-title":"A Recipe for the Estimation of Information Flow in a Dynamical System","volume":"17","author":"Gencaga","year":"2015","journal-title":"Entropy"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Emmert-Streib, F., and Dehmer, M. (2009). Causality in Time Series: Its Detection and Quantification by Means of Information Theory. Information Theory and Statistical Learning, Springer.","DOI":"10.1007\/978-0-387-84816-7"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4173","DOI":"10.3390\/e17064173","article-title":"Contribution to Transfer Entropy Estimation via the k-Nearest-Neighbors Approach","volume":"17","author":"Zhu","year":"2015","journal-title":"Entropy"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"68003","DOI":"10.1209\/0295-5075\/82\/68003","article-title":"Information flow between stock indices","volume":"82","author":"Kwon","year":"2008","journal-title":"EPL (Europhys. Lett.)"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"4443","DOI":"10.3390\/e16084443","article-title":"Structure of a Global Network of Financial Companies Based on Transfer Entropy","volume":"16","author":"Sandoval","year":"2014","journal-title":"Entropy"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"524","DOI":"10.3390\/e15020524","article-title":"On Thermodynamic Interpretation of Transfer Entropy","volume":"15","author":"Prokopenko","year":"2013","journal-title":"Entropy"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition, MIT Press.","DOI":"10.7551\/mitpress\/5236.001.0001"},{"key":"ref_28","unstructured":"Dua, D., and Graff, C. (2019, November 25). UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/index.php."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/1\/102\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:43:09Z","timestamp":1760362989000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/1\/102"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,16]]},"references-count":28,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,1]]}},"alternative-id":["e22010102"],"URL":"https:\/\/doi.org\/10.3390\/e22010102","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,16]]}}}