{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T14:36:29Z","timestamp":1776868589430,"version":"3.51.2"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"13","license":[{"start":{"date-parts":[[2021,1,4]],"date-time":"2021-01-04T00:00:00Z","timestamp":1609718400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,1,4]],"date-time":"2021-01-04T00:00:00Z","timestamp":1609718400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100008982","name":"Qatar National Research Fund","doi-asserted-by":"publisher","award":["NPRP11S-0108-180228"],"award-info":[{"award-number":["NPRP11S-0108-180228"]}],"id":[{"id":"10.13039\/100008982","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) that are homogenous only with a linear neuron model. As a heterogenous network model, ONNs are based on a generalized neuron model that can encapsulate<jats:italic>any<\/jats:italic>set of non-linear operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. However, the default search method to find optimal operators in ONNs, the so-called Greedy Iterative Search (GIS) method, usually takes several training sessions to find a single operator set per layer. This is not only computationally demanding, also the network heterogeneity is limited since the same set of operators will then be used for all neurons in each layer. To address this deficiency and exploit a superior level of heterogeneity, in this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the \u201cSynaptic Plasticity\u201d paradigm that poses the essential learning theory in biological neurons. During training, each operator set in the library can be evaluated by their synaptic plasticity level, ranked from the worst to the best, and an \u201celite\u201d ONN can then be configured using the top-ranked operator sets found at each hidden layer. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs and as a result, the performance gap over the CNNs further widens.<\/jats:p>","DOI":"10.1007\/s00521-020-05543-w","type":"journal-article","created":{"date-parts":[[2021,1,4]],"date-time":"2021-01-04T01:02:22Z","timestamp":1609722142000},"page":"7997-8015","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Exploiting heterogeneity in operational neural networks by synaptic plasticity"],"prefix":"10.1007","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1551-3397","authenticated-orcid":false,"given":"Serkan","family":"Kiranyaz","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junaid","family":"Malik","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Habib Ben","family":"Abdallah","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Turker","family":"Ince","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexandros","family":"Iosifidis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Moncef","family":"Gabbouj","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,1,4]]},"reference":[{"key":"5543_CR1","volume-title":"Diversity in the neuronal machine: order and variability in interneuronal microcircuits","author":"I Soltesz","year":"2002","unstructured":"Soltesz I (2002) Diversity in the neuronal machine: order and variability in interneuronal microcircuits. Oxford University Press, USA"},{"key":"5543_CR2","isbn-type":"print","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-011-0840-9","volume-title":"Learning in natural and connectionist systems","author":"RH Phaf","year":"1994","unstructured":"Phaf RH (1994) Learning in natural and connectionist systems. Springer, Netherlands. ISBN 978-0-7923-2685-4","ISBN":"https:\/\/id.crossref.org\/isbn\/9780792326854"},{"key":"5543_CR3","unstructured":"Werbos P (1974) Beyond regression: new tools for prediction and analysis in the behavioral sciences. PhD thesis, Harvard University"},{"issue":"5885","key":"5543_CR4","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1126\/science.1149381","volume":"321","author":"T Klausberger","year":"2008","unstructured":"Klausberger T, Somogyi P (2008) Neuronal diversity and temporal dynamics: the unity of hippocampal circuit operations. Science 321(5885):53\u201357","journal-title":"Science"},{"key":"5543_CR5","doi-asserted-by":"crossref","unstructured":"Z. Nusser, \u201cVariability in the subcellular distribution of ion channels increases neuronal diversity\u201d, Trends Neuroscience, pp. 267\u2013274, May 2009","DOI":"10.1016\/j.tins.2009.01.003"},{"issue":"7","key":"5543_CR6","doi-asserted-by":"publisher","first-page":"563","DOI":"10.1038\/nrn1949","volume":"7","author":"E Marder","year":"2006","unstructured":"Marder E, Goaillard JM (2006) Variability, compensation and homeostasis in neuron and network function. Nat Rev Neurosci 7(7):563\u2013574","journal-title":"Nat Rev Neurosci"},{"issue":"4","key":"5543_CR7","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1016\/S0959-4388(00)00230-0","volume":"11","author":"RH Masland","year":"2001","unstructured":"Masland RH (2001) Neuronal diversity in the retina. Curr Opinion Neurobiol 11(4):431\u2013436","journal-title":"Curr Opinion Neurobiol"},{"key":"5543_CR8","doi-asserted-by":"crossref","unstructured":"C. I. Moore, M. Carlen, U. Knoblich, J. A. Cardin, \u201cNeocortical interneurons: from diversity, strength\u201d, Cell, pp. 189-193, 2010","DOI":"10.1016\/j.cell.2010.07.005"},{"issue":"6088","key":"5543_CR9","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1038\/323533a0","volume":"323","author":"D Rumelhart","year":"1986","unstructured":"Rumelhart D, Hinton G, Williams R (1986) Learning representations by back-propagating errors. Nature 323(6088):533\u2013536","journal-title":"Nature"},{"key":"5543_CR10","volume-title":"Neural networks for optimization and signal processing","author":"A Chichocki","year":"1993","unstructured":"Chichocki A, Unbehauen R (1993) Neural networks for optimization and signal processing, 1st edn. Wiley Inc., New York","edition":"1"},{"key":"5543_CR11","volume-title":"Neural networks and learning machines","author":"S Haykin","year":"2008","unstructured":"Haykin S (2008) Neural networks and learning machines, 3rd edn. Prentice Hall, New Jersey","edition":"3"},{"issue":"6","key":"5543_CR12","doi-asserted-by":"publisher","first-page":"989","DOI":"10.1109\/72.329697","volume":"5","author":"M Hagan","year":"1994","unstructured":"Hagan M, Menhaj M (1994) Training feedforward networks with the Marquardt algorithm. IEEE Trans Neur Net 5(6):989\u2013993","journal-title":"IEEE Trans Neur Net"},{"issue":"2","key":"5543_CR13","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1109\/TNNLS.2011.2178124","volume":"23","author":"R Zhang","year":"2012","unstructured":"Zhang R, Lan Y, Huang G, Zu Z (2012) Universal approximation of extreme learning machine with adaptive growth of hidden nodes. IEEE Trans Neur Net Learn Syst. 23(2):365\u2013371","journal-title":"IEEE Trans Neur Net Learn Syst."},{"key":"5543_CR14","doi-asserted-by":"crossref","unstructured":"Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layerwise training of deep networks. NIPS","DOI":"10.7551\/mitpress\/7503.003.0024"},{"key":"5543_CR15","unstructured":"Kulkarni M, Karande S (2017) Layer-wise training of deep networks using kernel similarity. arXiv preprint arXiv:1703.07115"},{"key":"5543_CR16","doi-asserted-by":"crossref","unstructured":"Zoph B, et al. (2018) Learning transferable architectures for scalable image recognition. In: IEEE CVPR","DOI":"10.1109\/CVPR.2018.00907"},{"key":"5543_CR17","unstructured":"Pham H et al. (2018) Efficient neural architecture search via parameters sharing. ICML"},{"key":"5543_CR18","doi-asserted-by":"publisher","first-page":"1448","DOI":"10.1016\/j.neunet.2009.05.013","volume":"22","author":"S Kiranyaz","year":"2009","unstructured":"Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2009) Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw 22:1448\u20131462","journal-title":"Neural Netw"},{"issue":"5","key":"5543_CR19","doi-asserted-by":"publisher","first-page":"1415","DOI":"10.1109\/TBME.2009.2013934","volume":"56","author":"T Ince","year":"2009","unstructured":"Ince T, Kiranyaz S, Gabbouj M (2009) A generic and robust system for automated patient-specific classification of electrocardiogram signals. IEEE Trans Biomed Eng 56(5):1415\u20131426","journal-title":"IEEE Trans Biomed Eng"},{"key":"5543_CR20","doi-asserted-by":"publisher","first-page":"383","DOI":"10.1007\/978-3-642-37846-1","volume-title":"Multi-dimensional particle swarm optimization for machine learning and pattern recognition","author":"S Kiranyaz","year":"2014","unstructured":"Kiranyaz S, Ince T, Gabbouj M (2014) Multi-dimensional particle swarm optimization for machine learning and pattern recognition. Springer, Berlin, p 383"},{"key":"5543_CR21","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1007\/BF02478259","volume":"5","author":"W McCulloch","year":"1943","unstructured":"McCulloch W, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophysics 5:115\u2013133","journal-title":"Bull Math Biophysics"},{"key":"5543_CR22","doi-asserted-by":"crossref","unstructured":"Rauber TW, Berns K (2011) Kernel multilayer perceptron. In: Conference on graphics, patterns and images, 2011","DOI":"10.1109\/SIBGRAPI.2011.21"},{"issue":"6","key":"5543_CR23","doi-asserted-by":"publisher","first-page":"771","DOI":"10.1080\/00207720050030815","volume":"31","author":"MY Mashor","year":"2000","unstructured":"Mashor MY (2000) Hybrid multilayered perceptron networks. Int J Syst Sci 31(6):771\u2013785","journal-title":"Int J Syst Sci"},{"issue":"1","key":"5543_CR24","doi-asserted-by":"publisher","first-page":"1457","DOI":"10.1016\/j.asoc.2010.04.017","volume":"11","author":"NAM Isa","year":"2011","unstructured":"Isa NAM, Mamat WMFW (2011) Clustered-hybrid multilayer perceptron network for pattern recognition application. Appl Soft Comput 11(1):1457\u20131466","journal-title":"Appl Soft Comput"},{"issue":"4","key":"5543_CR25","doi-asserted-by":"publisher","first-page":"835","DOI":"10.1109\/TNN.2003.813841","volume":"14","author":"NB Karayiannis","year":"2003","unstructured":"Karayiannis NB, Randolph-Gips MM (2003) On the construction and training of reformulated radial basis function neural networks. IEEE Trans Neural Netw 14(4):835\u2013846","journal-title":"IEEE Trans Neural Netw"},{"key":"5543_CR26","unstructured":"Ruiwang H, Binwang H (2002) A new algorithm of selection the radial basis function networks center. In: ICMLC, 2002"},{"key":"5543_CR27","doi-asserted-by":"crossref","unstructured":"Kiranyaz S, Ince T, Iosifidis A, Gabbouj M (2017) Generalized model of biological neural networks: progressive operational perceptrons. In: IJCNN, 2017","DOI":"10.1109\/IJCNN.2017.7966157"},{"key":"5543_CR28","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1016\/j.neucom.2016.10.044","volume":"224","author":"S Kiranyaz","year":"2017","unstructured":"Kiranyaz S, Ince T, Iosifidis A, Gabbouj M (2017) Progressive operational perceptrons. Neurocomputing 224:142\u2013154","journal-title":"Neurocomputing"},{"key":"5543_CR29","doi-asserted-by":"publisher","DOI":"10.1109\/tnnls.2019.2914082","author":"DT Tran","year":"2019","unstructured":"Tran DT, Kiranyaz S, Gabbouj M, Iosifidis A (2019) Heterogeneous multilayer generalized operational perceptron. IEEE Trans Neural Netw Learn Syst. https:\/\/doi.org\/10.1109\/tnnls.2019.2914082","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"5543_CR30","unstructured":"Tran DT, Kanniainen J, Gabbouj M, Iosifidis A (2019) Data-driven neural architecture learning for financial time-series forecasting. Digital Image and Signal Processing, Oxford, U.K. 2019"},{"key":"5543_CR31","doi-asserted-by":"crossref","unstructured":"Tran DT, Iosifidis A (2019) Learning to rank: a progressive neural network learning approach. In: IEEE int. conf. on acoustics, speech, and signal processing, Brighton, U.K., 2019","DOI":"10.1109\/ICASSP.2019.8683711"},{"key":"5543_CR32","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1016\/j.neucom.2019.10.079","volume":"379","author":"DT Tran","year":"2020","unstructured":"Tran DT, Kiranyaz S, Gabbouj M, Iosifidis A (2020) Progressive operational perceptron with memory. Neurocomputing 379:172\u2013181","journal-title":"Neurocomputing"},{"key":"5543_CR33","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2019.06.009","author":"DT Tran","year":"2019","unstructured":"Tran DT, Kiranyaz S, Gabbouj M, Iosifidis A (2019) PyGOP: a python library for generalized operational perceptron. Knowl Based Syst. https:\/\/doi.org\/10.1016\/j.knosys.2019.06.009","journal-title":"Knowl Based Syst"},{"key":"5543_CR34","doi-asserted-by":"crossref","unstructured":"Tran DT, Kiranyaz S, Gabbouj M, Iosifidis A (2019) Knowledge transfer for face verification using heterogeneous generalized operational perceptrons. In: IEEE int. conf. on image processing (ICIP\u201919), Taiwan, Sep. 2019","DOI":"10.1109\/ICIP.2019.8804296"},{"key":"5543_CR35","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-020-04780-3","author":"S Kiranyaz","year":"2020","unstructured":"Kiranyaz S, Ince T, Iosifidis A, Gabbouj M (2020) Operational neural networks. Neural Comput Appl. https:\/\/doi.org\/10.1007\/s00521-020-04780-3","journal-title":"Neural Comput Appl"},{"key":"5543_CR36","unstructured":"Jain V, Seung S (2009) Natural image denoising with convolutional networks. In: Proc. adv. neural inf. process. syst., pp. 769\u2013776"},{"key":"5543_CR37","doi-asserted-by":"crossref","unstructured":"Lefkimmiatis S (2017) Non-local color image denoising with convolutional neural networks. Proc. IEEE conf. comput. vis. pattern recognit., pp 3587\u20133596","DOI":"10.1109\/CVPR.2017.623"},{"issue":"1","key":"5543_CR38","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1109\/LSP.2017.2768660","volume":"25","author":"D Yang","year":"2018","unstructured":"Yang D, Sun J (2018) BM3D-Net: a convolutional neural network for transform-domain collaborative filtering. IEEE Signal Process Lett 25(1):55\u201359. https:\/\/doi.org\/10.1109\/LSP.2017.2768660","journal-title":"IEEE Signal Process Lett"},{"key":"5543_CR39","doi-asserted-by":"crossref","unstructured":"Chen Y, Yu W, Pock T (2015) On learning optimized reaction diffusion processes for effective image restoration. In: Proc. IEEE conf. comput. vis. pattern recognit., pp 5261\u20135269","DOI":"10.1109\/CVPR.2015.7299163"},{"key":"5543_CR40","first-page":"3371","volume":"11","author":"P Vincent","year":"2010","unstructured":"Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11:3371\u20133408","journal-title":"J Mach Learn Res"},{"key":"5543_CR41","doi-asserted-by":"crossref","unstructured":"Kiranyaz S, Waris MA, Ahmad I, Hamila R, Gabbouj M (2016) Face segmentation in thumbnail images by data-adaptive convolutional segmentation networks. In: Int. conf. on image processing, ICIP\u201916, Phoenix, Arizona, USA, September 25-28, 2016","DOI":"10.1109\/ICIP.2016.7532770"},{"key":"5543_CR42","doi-asserted-by":"crossref","unstructured":"Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proc. of the IEEE conf.on computer vision and pattern recognition (CVPR), 2017","DOI":"10.1109\/CVPR.2017.632"},{"key":"5543_CR43","doi-asserted-by":"crossref","unstructured":"Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proc. of the IEEE conf.on computer vision and pattern recognition (cVPR), 2017","DOI":"10.1109\/ICCV.2017.244"},{"key":"5543_CR44","unstructured":"Jain V, Learned-Miller E (2010) FDDB: A benchmark for face detection in unconstrained settings. Technical Report UM-CS2010-009, University of Massachusetts, Amherst"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-020-05543-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-020-05543-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-020-05543-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,16]],"date-time":"2023-10-16T18:53:12Z","timestamp":1697482392000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-020-05543-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,4]]},"references-count":44,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["5543"],"URL":"https:\/\/doi.org\/10.1007\/s00521-020-05543-w","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,4]]},"assertion":[{"value":"21 April 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 November 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 January 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors declare that they have no conflict of interest or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}