{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T15:38:36Z","timestamp":1790523516589,"version":"4.1.0"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2020,3,6]],"date-time":"2020-03-06T00:00:00Z","timestamp":1583452800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,3,6]],"date-time":"2020-03-06T00:00:00Z","timestamp":1583452800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2020,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Feed-forward, fully connected artificial neural networks or the so-called multi-layer perceptrons are well-known universal approximators. However, their learning performance varies significantly depending on the function or the solution space that they attempt to approximate. This is mainly because of their homogenous configuration based solely on the linear neuron model. Therefore, while they learn very well those problems with a monotonous, relatively simple and linearly separable solution space, they may entirely fail to do so when the solution space is highly nonlinear and complex. Sharing the same linear neuron model with two additional constraints (local connections and weight sharing), this is also true for the conventional convolutional neural networks (CNNs) and it is, therefore, not surprising that in many challenging problems only the deep CNNs with a massive complexity and depth can achieve the required diversity and the learning performance. In order to address this drawback and also to accomplish a more generalized model over the convolutional neurons, this study proposes a novel network model, called operational neural networks (ONNs), which can be heterogeneous and encapsulate neurons with any set of operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. Finally, the training method to back-propagate the error through the operational layers of ONNs is formulated. Experimental results over highly challenging problems demonstrate the superior learning capabilities of ONNs even with few neurons and hidden layers.<\/jats:p>","DOI":"10.1007\/s00521-020-04780-3","type":"journal-article","created":{"date-parts":[[2020,3,6]],"date-time":"2020-03-06T13:02:52Z","timestamp":1583499772000},"page":"6645-6668","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":104,"title":["Operational neural networks"],"prefix":"10.1007","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1551-3397","authenticated-orcid":false,"given":"Serkan","family":"Kiranyaz","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Turker","family":"Ince","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandros","family":"Iosifidis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Moncef","family":"Gabbouj","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,3,6]]},"reference":[{"issue":"6088","key":"4780_CR1","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","volume":"323","author":"D Rumelhart","year":"1986","unstructured":"Rumelhart D, Hinton G, Williams R (1986) Learning representations by back-propagating errors. Nature 323(6088):533\u2013536","journal-title":"Nature"},{"issue":"6","key":"4780_CR2","doi-asserted-by":"crossref","first-page":"989","DOI":"10.1109\/72.329697","volume":"5","author":"M Hagan","year":"1994","unstructured":"Hagan M, Menhaj M (1994) Training feedforward networks with the Marquardt algorithm. IEEE Trans Neural Netw 5(6):989\u2013993","journal-title":"IEEE Trans Neural Netw"},{"issue":"2","key":"4780_CR3","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1109\/TNNLS.2011.2178124","volume":"23","author":"R Zhang","year":"2012","unstructured":"Zhang R, Lan Y, Huang G, Zu Z (2012) Universal approximation of extreme learning machine with adaptive growth of hidden nodes. IEEE Trans Neural Netw Learn Syst 23(2):365\u2013371","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"4780_CR4","doi-asserted-by":"crossref","unstructured":"Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layerwise training of deep networks. In: NIPS","DOI":"10.7551\/mitpress\/7503.003.0024"},{"key":"4780_CR5","unstructured":"Kulkarni M, Karande S (2017) Layer-wise training of deep networks using kernel similarity. arXiv preprint arXiv:1703.07115"},{"key":"4780_CR6","doi-asserted-by":"crossref","unstructured":"Zoph B et al. (2018) Learning transferable architectures for scalable image recognition. In: IEEE CVPR","DOI":"10.1109\/CVPR.2018.00907"},{"key":"4780_CR7","unstructured":"Pham H et al. (2018) Efficient neural architecture search via parameters sharing. In: ICML"},{"key":"4780_CR8","doi-asserted-by":"crossref","first-page":"1448","DOI":"10.1016\/j.neunet.2009.05.013","volume":"22","author":"S Kiranyaz","year":"2009","unstructured":"Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2009) Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw 22:1448\u20131462","journal-title":"Neural Netw"},{"issue":"5","key":"4780_CR9","doi-asserted-by":"crossref","first-page":"1415","DOI":"10.1109\/TBME.2009.2013934","volume":"56","author":"T Ince","year":"2009","unstructured":"Ince T, Kiranyaz S, Gabbouj M (2009) A generic and robust system for automated patient-specific classification of electrocardiogram signals. IEEE Trans Biomed Eng 56(5):1415\u20131426","journal-title":"IEEE Trans Biomed Eng"},{"key":"4780_CR10","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/978-3-642-37846-1","volume-title":"Multi-dimensional particle swarm optimization for machine learning and pattern recognition","author":"S Kiranyaz","year":"2014","unstructured":"Kiranyaz S, Ince T, Gabbouj M (2014) Multi-dimensional particle swarm optimization for machine learning and pattern recognition. Springer, Berlin, p 383"},{"key":"4780_CR11","unstructured":"Werbos P (1974) Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph. D thesis, Harvard University"},{"issue":"5885","key":"4780_CR12","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1126\/science.1149381","volume":"321","author":"T Klausberger","year":"2008","unstructured":"Klausberger T, Somogyi P (2008) Neuronal diversity and temporal dynamics: the unity of hippocampal circuit operations. Science 321(5885):53\u201357","journal-title":"Science"},{"issue":"5","key":"4780_CR13","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/j.tins.2009.01.003","volume":"32","author":"Z Nusser","year":"2009","unstructured":"Nusser Z (2009) Variability in the subcellular distribution of ion channels increases neuronal diversity. Trends Neurosci 32(5):267\u2013274","journal-title":"Trends Neurosci"},{"key":"4780_CR14","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1038\/nrn1949","volume":"7","author":"E Marder","year":"2006","unstructured":"Marder E, Goaillard JM (2006) Variability, compensation and homeostasis in neuron and network function. Nat Rev Neurosci 7:563\u2013574","journal-title":"Nat Rev Neurosci"},{"key":"4780_CR15","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1016\/S0959-4388(00)00230-0","volume":"11","author":"RH Masland","year":"2001","unstructured":"Masland RH (2001) Neuronal diversity in the retina. Current Opin Neurobiol 11:431\u2013436","journal-title":"Current Opin Neurobiol"},{"key":"4780_CR16","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/j.cell.2010.07.004","volume":"142","author":"CI Moore","year":"2010","unstructured":"Moore CI, Carlen M, Knoblich U, Cardin JA (2010) Neocortical interneurons: from diversity, strength. Cell 142:189\u2013193","journal-title":"Cell"},{"key":"4780_CR17","volume-title":"Diversity in the neuronal machine: order and variability in interneuronal microcircuits","author":"I Soltesz","year":"2002","unstructured":"Soltesz I (2002) Diversity in the neuronal machine: order and variability in interneuronal microcircuits. Oxford University Press, Oxford"},{"key":"4780_CR18","doi-asserted-by":"crossref","unstructured":"Rauber TW, Berns K (2011) Kernel multilayer perceptron. In: Conference on graphics, patterns and images","DOI":"10.1109\/SIBGRAPI.2011.21"},{"issue":"6","key":"4780_CR19","doi-asserted-by":"crossref","first-page":"771","DOI":"10.1080\/00207720050030815","volume":"31","author":"MY Mashor","year":"2000","unstructured":"Mashor MY (2000) Hybrid multilayered perceptron networks. Int J Syst Sci 31(6):771\u2013785","journal-title":"Int J Syst Sci"},{"issue":"1","key":"4780_CR20","doi-asserted-by":"crossref","first-page":"1457","DOI":"10.1016\/j.asoc.2010.04.017","volume":"11","author":"NAM Isa","year":"2011","unstructured":"Isa NAM, Mamat WMFW (2011) Clustered-hybrid multilayer perceptron network for pattern recognition application. Appl Soft Comput 11(1):1457\u20131466","journal-title":"Appl Soft Comput"},{"issue":"4","key":"4780_CR21","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1109\/TNN.2003.813841","volume":"14","author":"NB Karayiannis","year":"2003","unstructured":"Karayiannis NB, Randolph-Gips MM (2003) The construction and training of reformulated radial basis function. IEEE Trans Neur Net 14(4):835\u2013846","journal-title":"IEEE Trans Neur Net"},{"key":"4780_CR22","unstructured":"Ruiwang H, Binwang H (2002) A new algorithm of selection the radial basis function networks center. In: ICMLC"},{"key":"4780_CR23","volume-title":"Neural networks for optimization and signal processing","author":"A Cochocki","year":"1993","unstructured":"Cochocki A, Unbehauen R (1993) Neural networks for optimization and signal processing, 1st edn. Wiley Inc., New York","edition":"1"},{"key":"4780_CR24","volume-title":"Neural networks and learning machines (3rd Edition)","author":"S Haykin","year":"2008","unstructured":"Haykin S (2008) Neural networks and learning machines (3rd Edition). Prentice Hall, Prentice"},{"key":"4780_CR25","doi-asserted-by":"crossref","unstructured":"S. Kiranyaz, T. Ince, A. Iosifidis and M. Gabbouj, \u201cGeneralized model of biological neural networks: progressive operational perceptrons,\u201d IJCNN, 2017","DOI":"10.1109\/IJCNN.2017.7966157"},{"key":"4780_CR26","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.neucom.2016.10.044","volume":"224","author":"S Kiranyaz","year":"2017","unstructured":"Kiranyaz S, Ince T, Iosifidis A, Gabbouj M (2017) Progressive operational perceptrons. Neurocomputing 224:142\u2013154","journal-title":"Neurocomputing"},{"key":"4780_CR27","unstructured":"Tran DT, Kiranyaz S, Gabbouj M, Iosifidis A (2018) Progressive operational perceptron with memory. arXiv:1808.06377"},{"issue":"3","key":"4780_CR28","doi-asserted-by":"publisher","first-page":"710","DOI":"10.1109\/TNNLS.2019.2914082","volume":"31","author":"Dat Thanh Tran","year":"2020","unstructured":"Tran DT, Kiranyaz S, Gabbouj M, Iosifidis A (2019) Heterogeneous multilayer generalized operational perceptron. IEEE Trans Neural Netw Learn Syst (2019). https:\/\/doi.org\/10.1109\/TNNLS.2019.2914082","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"4780_CR29","doi-asserted-by":"crossref","unstructured":"Tran DT, Kiranyaz S, Gabbouj M, Iosifidis A (2019) Knowledge transfer for face verification using heterogeneous generalized operational perceptrons. In: IEEE International conference on image processing, Taipei, Taiwan","DOI":"10.1109\/ICIP.2019.8804296"},{"key":"4780_CR30","unstructured":"Jain V, Seung S (2009) Natural image denoising with convolutional networks. In: Proceedings of advances in neural information processing system, pp 769\u2013776"},{"key":"4780_CR31","doi-asserted-by":"crossref","unstructured":"Lefkimmiatis S (2017) Non-local color image denoising with convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 3587\u20133596","DOI":"10.1109\/CVPR.2017.623"},{"issue":"1","key":"4780_CR32","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1109\/lsp.2017.2768660","volume":"25","author":"D Yang","year":"2018","unstructured":"Yang D, Sun J (2018) BM3D-Net: a convolutional neural network for transform-domain collaborative filtering. IEEE Signal Process Lett 25(1):55\u201359. https:\/\/doi.org\/10.1109\/lsp.2017.2768660","journal-title":"IEEE Signal Process Lett"},{"key":"4780_CR33","doi-asserted-by":"crossref","unstructured":"Chen Y, Yu W, Pock T (2015) On learning optimized reaction diffusion processes for effective image restoration. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 5261\u20135269","DOI":"10.1109\/CVPR.2015.7299163"},{"key":"4780_CR34","doi-asserted-by":"crossref","unstructured":"Kiranyaz S, Waris MA, Ahmad I, Hamila R, Gabbouj M (2016) Face segmentation in thumbnail images by data-adaptive convolutional segmentation networks. In: International conference on image processing, ICIP\u201916, Phoenix, Arizona, USA, 25\u201328 Sept 2016","DOI":"10.1109\/ICIP.2016.7532770"},{"key":"4780_CR35","doi-asserted-by":"crossref","unstructured":"Li Y, Qi H, Dai J, Ji X, Wei Y (2017) Fully convolutional instance aware semantic segmentation. In: Computer vision and pattern recognition (CVPR), pp. 4438\u20134446","DOI":"10.1109\/CVPR.2017.472"},{"key":"4780_CR36","doi-asserted-by":"crossref","unstructured":"Lin T, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollar\u00b4 P, Zitnick L (2014) Microsoft COCO: common objects in context. In: ECCV, pp. 740\u2013755","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"4780_CR37","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.jvcir.2015.10.012","volume":"34","author":"H Zhu","year":"2016","unstructured":"Zhu H, Meng F, Cai J, Lu S (2016) Beyond pixels: a comprehensive survey from bottom-up to semantic image segmentation and cosegmentation. J Vis Commun Image Represent 34:12\u201327","journal-title":"J Vis Commun Image Represent"},{"key":"4780_CR38","doi-asserted-by":"crossref","unstructured":"Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431\u20133440","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"4780_CR39","first-page":"234","volume-title":"Lecture Notes in Computer Science","author":"Olaf Ronneberger","year":"2015","unstructured":"Ronneberger O, Fischer P, Brox T (2015) UNet: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp. 234\u2013241"},{"key":"4780_CR40","first-page":"189","volume-title":"Advances in Face Detection and Facial Image Analysis","author":"Erik Learned-Miller","year":"2016","unstructured":"Learned-Miller E, Huang GB, RoyChowdhury A, Li H, Hua G (2016) Labeled faces in the wild: A survey. In: Advances in Face Detection and Facial Image Analysis. Springer, Berlin, pp 189\u2013248"},{"key":"4780_CR41","unstructured":"Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. CoRR, vol. abs\/1511.07122, 2015"},{"key":"4780_CR42","unstructured":"Luc P, Couprie C, Chintala S, Verbeek J (2016) Semantic segmentation using adversarial networks. CoRR, vol. abs\/1611.08408, 2016"},{"key":"4780_CR43","unstructured":"Jain V, Learned-Miller E (2010) FDDB: a benchmark for face detection in unconstrained settings. Technical Report UM-CS2010-009, University of Massachusetts, Amherst"},{"key":"4780_CR44","doi-asserted-by":"crossref","unstructured":"Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017","DOI":"10.1109\/CVPR.2017.632"},{"key":"4780_CR45","doi-asserted-by":"crossref","unstructured":"Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","DOI":"10.1109\/ICCV.2017.244"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-020-04780-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s00521-020-04780-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-020-04780-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T07:00:46Z","timestamp":1695884446000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s00521-020-04780-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,6]]},"references-count":45,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["4780"],"URL":"https:\/\/doi.org\/10.1007\/s00521-020-04780-3","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,6]]},"assertion":[{"value":"13 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 February 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}