{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T06:15:52Z","timestamp":1790662552866,"version":"4.1.0"},"reference-count":43,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,2,11]],"date-time":"2020-02-11T00:00:00Z","timestamp":1581379200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100007479","name":"Fondazione Cassa di Risparmio di Padova e Rovigo","doi-asserted-by":"publisher","award":["Excellence Project 2018"],"award-info":[{"award-number":["Excellence Project 2018"]}],"id":[{"id":"10.13039\/100007479","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Deepmath","award":["2018 STARS Grant"],"award-info":[{"award-number":["2018 STARS Grant"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article, we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Simulations show that the final network topology is shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.<\/jats:p>","DOI":"10.3390\/e22020204","type":"journal-article","created":{"date-parts":[[2020,2,11]],"date-time":"2020-02-11T09:25:21Z","timestamp":1581413121000},"page":"204","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Emergence of Network Motifs in Deep Neural Networks"],"prefix":"10.3390","volume":"22","author":[{"given":"Matteo","family":"Zambra","sequence":"first","affiliation":[{"name":"Department of Civil, Environmental and Architectural Engineering, University of Padova, Via Marzolo 9, 35131 Padova, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amos","family":"Maritan","sequence":"additional","affiliation":[{"name":"Department of Physics and Astronomy, University of Padova; Istituto Nazionale di Fisica Nucleare\u2014Sezione di Padova, Via Marzolo 8, 35131 Padova, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7062-4861","authenticated-orcid":false,"given":"Alberto","family":"Testolin","sequence":"additional","affiliation":[{"name":"Department of General Psychology, University of Padova, Via Venezia 8, 35131 Padova, Italy"},{"name":"Department of Information Engineering, University of Padova, Via Gradenigo 6\/b, 35131 Padova, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,11]]},"reference":[{"key":"ref_1","unstructured":"Newman, M. (2010). Networks: An Introduction, Oxford University Press, Inc."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1038\/35065725","article-title":"Exploring complex networks","volume":"410","author":"Strogatz","year":"2001","journal-title":"Nature"},{"key":"ref_3","unstructured":"Caldarelli, G. (2010). Complex Networks, EOLSS Publications."},{"key":"ref_4","unstructured":"Newman, M.E., Barabasi, A.L., and Watts, D.J. (2006). The Structure and Dynamics of Networks: (Princeton Studies in Complexity), Princeton University Press."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Latora, V., Nicosia, V., and Russo, G. (2017). Complex Networks: Principles, Methods and Applications, Cambridge University Press.","DOI":"10.1017\/9781316216002"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1126\/science.298.5594.824","article-title":"Network Motifs: Simple Building Blocks of Complex Networks","volume":"298","author":"Milo","year":"2002","journal-title":"Science"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1038\/nature01568","article-title":"The evolutionary origin of complex features","volume":"423","author":"Lenski","year":"2003","journal-title":"Nature"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1038\/ng1003-118","article-title":"Evolution thinks modular","volume":"35","author":"Vespignani","year":"2003","journal-title":"Nat. Genet."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Alon, U. (2006). An Introduction to Systems Biology: Design Principles of Biological Circuits, Chapman & Hall\/CRC Mathematical and Computational Biology, Taylor & Francis.","DOI":"10.1201\/9781420011432"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"LeCun, Y., Bengio, Y., and Hinton, G.E. (2015). Deep learning. Nature, 521.","DOI":"10.1038\/nature14539"},{"key":"ref_11","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press. Available online: http:\/\/www.deeplearningbook.org."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_13","first-page":"3104","article-title":"Sequence to sequence learning with neural networks","volume":"27","author":"Sutskever","year":"2014","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.dsp.2017.10.011","article-title":"Methods for interpreting and understanding deep neural networks","volume":"73","author":"Montavon","year":"2018","journal-title":"Digit. Signal Process."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"11537","DOI":"10.1073\/pnas.1820226116","article-title":"A mathematical theory of semantic development in deep neural networks","volume":"116","author":"Saxe","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Testolin, A., Piccolini, M., and Suweis, S. (2018). Deep learning systems as complex networks. J. Complex Netw., 521.","DOI":"10.1093\/comnet\/cnz018"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"73","DOI":"10.3389\/fncom.2016.00073","article-title":"Probabilistic models and generative neural networks: Towards an unified framework for modeling normal and impaired neurocognitive functions","volume":"10","author":"Testolin","year":"2016","journal-title":"Front. Comput. Neurosci."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1126\/science.220.4598.671","article-title":"Optimization by Simulated Annealing","volume":"220","author":"Kirkpatrick","year":"1983","journal-title":"Science"},{"key":"ref_20","unstructured":"Saxe, A., McClelland, J., and Ganguli, S. (2014, January 14\u201316). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. Proceedings of the International Conference on Learning Representations, Banff, AB, Canada."},{"key":"ref_21","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1152","DOI":"10.1093\/bioinformatics\/btl038","article-title":"FANMOD: A tool for fast network motif detection","volume":"22","author":"Wernicke","year":"2006","journal-title":"Bioinformatics"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1109\/TCBB.2006.51","article-title":"Efficient Detection of Network Motifs","volume":"3","author":"Wernicke","year":"2006","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1049\/iet-syb.2011.0011","article-title":"Building blocks of biological networks: A review on major network motif discovery algorithms","volume":"6","author":"Schreiber","year":"2012","journal-title":"IET Syst. Biol."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Alon, U. (2007). Network motifs: Theory and experimental approaches. Nat. Rev. Genet., 8.","DOI":"10.1038\/nrg2102"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1538","DOI":"10.1126\/science.1089167","article-title":"Superfamilies of Evolved and Designed Networks","volume":"303","author":"Milo","year":"2004","journal-title":"Science"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1038\/ng1242","article-title":"Evolutionary conservation of motif constituents in the yeast protein interaction network","volume":"35","author":"Wuchty","year":"2003","journal-title":"Nat. Genet."},{"key":"ref_29","unstructured":"van Dyk, D., and Welling, M. (2009). Deep boltzmann machines. Artificial Intelligence and Statistics, PMLR."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"515","DOI":"10.3389\/fpsyg.2013.00515","article-title":"Modeling language and cognition with deep unsupervised learning: A tutorial overview","volume":"4","author":"Zorzi","year":"2013","journal-title":"Front. Psychol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"579","DOI":"10.1111\/cogs.12258","article-title":"Learning orthographic structure with sequential generative neural networks","volume":"40","author":"Testolin","year":"2016","journal-title":"Cogn. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"7068349","DOI":"10.1155\/2018\/7068349","article-title":"Deep Learning for Computer Vision: A Brief Review","volume":"2018","author":"Voulodimos","year":"2018","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_35","first-page":"30:1","article-title":"Isomorphism Test for Digraphs with Weighted Edges","volume":"Volume 103","author":"Piperno","year":"2018","journal-title":"Proceedings of the 17th International Symposium on Experimental Algorithms (SEA 2018)"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.jsc.2013.09.003","article-title":"Practical graph isomorphism, II","volume":"60","author":"McKay","year":"2014","journal-title":"J. Symb. Comput."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Raina, R., Madhavan, A., and Ng, A.Y. (2009, January 14\u201318). Large-scale deep unsupervised learning using graphics processors. Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada.","DOI":"10.1145\/1553374.1553486"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"251","DOI":"10.3389\/fpsyg.2013.00251","article-title":"Deep unsupervised learning on a desktop PC: A primer for cognitive scientists","volume":"4","author":"Testolin","year":"2013","journal-title":"Front. Psychol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"8577","DOI":"10.1073\/pnas.0601602103","article-title":"Modularity and community structure in networks","volume":"103","author":"Newman","year":"2006","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1137\/S003614450342480","article-title":"The structure and function of complex networks","volume":"45","author":"Newman","year":"2003","journal-title":"SIAM Rev."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Choobdar, S., Ribeiro, P., and Silva, F. (2012, January 10). Motif Mining in Weighted Networks. Proceedings of the 12nd IEEE ICDM Workshop on Data Mining in Networks, Brussels, Belgium.","DOI":"10.1109\/ICDMW.2012.111"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Onnela, J.P., Saram\u00e4ki, J., Kert\u00e9sz, J., and Kaski, K. (2005). Intensity and coherence of motifs in weighted complex networks. Phys. Rev. E, 71.","DOI":"10.1103\/PhysRevE.71.065103"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"10687","DOI":"10.1073\/pnas.0802631105","article-title":"The discovery of structural form","volume":"105","author":"Kemp","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/204\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:56:47Z","timestamp":1760173007000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/204"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,11]]},"references-count":43,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["e22020204"],"URL":"https:\/\/doi.org\/10.3390\/e22020204","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,11]]}}}