{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T12:35:00Z","timestamp":1790598900489,"version":"4.1.0"},"reference-count":44,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,2,5]],"date-time":"2021-02-05T00:00:00Z","timestamp":1612483200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1831980"],"award-info":[{"award-number":["CNS-1831980"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["ACI-1548562"],"award-info":[{"award-number":["ACI-1548562"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The nonlinearity of activation functions used in deep learning models is crucial for the success of predictive models. Several simple nonlinear functions, including Rectified Linear Unit (ReLU) and Leaky-ReLU (L-ReLU) are commonly used in neural networks to impose the nonlinearity. In practice, these functions remarkably enhance the model accuracy. However, there is limited insight into the effects of nonlinearity in neural networks on their performance. Here, we investigate the performance of neural network models as a function of nonlinearity using ReLU and L-ReLU activation functions in the context of different model architectures and data domains. We use entropy as a measurement of the randomness, to quantify the effects of nonlinearity in different architecture shapes on the performance of neural networks. We show that the ReLU nonliearity is a better choice for activation function mostly when the network has sufficient number of parameters. However, we found that the image classification models with transfer learning seem to perform well with L-ReLU in fully connected layers. We show that the entropy of hidden layer outputs in neural networks can fairly represent the fluctuations in information loss as a function of nonlinearity. Furthermore, we investigate the entropy profile of shallow neural networks as a way of representing their hidden layer dynamics.<\/jats:p>","DOI":"10.3390\/a14020051","type":"journal-article","created":{"date-parts":[[2021,2,5]],"date-time":"2021-02-05T08:33:48Z","timestamp":1612514028000},"page":"51","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":38,"title":["Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0480-1142","authenticated-orcid":false,"given":"Nalinda","family":"Kulathunga","sequence":"first","affiliation":[{"name":"Department of Mathematics, Texas Southern University, 3100 Cleburne Street, Houston, TX 77004, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3115-1226","authenticated-orcid":false,"given":"Nishath Rajiv","family":"Ranasinghe","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Prairie View A &amp; M University, 700 University Drive, Prairie View, TX 77446, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8820-9073","authenticated-orcid":false,"given":"Daniel","family":"Vrinceanu","sequence":"additional","affiliation":[{"name":"Department of Physics, Texas Southern University, 3100 Cleburne Street, Houston, TX 77004, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zackary","family":"Kinsman","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Texas Southern University, 3100 Cleburne Street, Houston, TX 77004, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Prairie View A &amp; M University, 700 University Drive, Prairie View, TX 77446, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yunjiao","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Texas Southern University, 3100 Cleburne Street, Houston, TX 77004, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,5]]},"reference":[{"key":"ref_1","unstructured":"Pedamonti, D. (2018). Comparison of non-linear activation functions for deep neural networks on MNIST classification task. arXiv."},{"key":"ref_2","unstructured":"Nwankpa, C., Ijomah, W., Gachagan, A., and Marshall, S. (2018). Activation Functions: Comparison of trends in Practice and Research for Deep Learning. arXiv."},{"key":"ref_3","unstructured":"Xu, B., Wang, N., Chen, T., and Li, M. (2015). Empirical Evaluation of Rectified Activations in Convolutional Network. arXiv."},{"key":"ref_4","unstructured":"Nair, V., and Hinton, G.E. (2010, January 21\u201324). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on Machine Learning, ICML\u201910, Haifa, Israel."},{"key":"ref_5","unstructured":"Arora, R., Basu, A., Mianjy, P., and Mukherjee, A. (2016). Understanding Deep Neural Networks with Rectified Linear Units. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Javid, A.M., Das, S., Skoglund, M., and Chatterjee, S. (2020). A ReLU Dense Layer to Improve the Performance of Neural Networks. arXiv.","DOI":"10.1109\/ICASSP39728.2021.9414269"},{"key":"ref_7","unstructured":"Pascanu, R., Mikolov, T., and Bengio, Y. (2013, January 17\u201319). On the Difficulty of Training Recurrent Neural Networks. Proceedings of the 30th International Conference on Machine Learning, ICML, Atlanta, GA, USA."},{"key":"ref_8","unstructured":"Maas, A.L. (2013, January 17\u201319). Rectifier Nonlinearities Improve Neural Network Acoustic Models. Proceedings of the 30th International Conference on Machine Learning (ICML), Atlanta, GA, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1038\/s41598-020-57866-2","article-title":"Structural Analysis and Optimization of Convolutional Neural Networks with a Small Sample Size","volume":"10","author":"Huang","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_10","unstructured":"Glorot, X., Bordes, A., and Bengio, Y. (2011, January 11\u201313). Deep Sparse Rectifier Neural Networks. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep Learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_12","unstructured":"Panigrahi, A., Shetty, A., and Goyal, N. (2020). Effect of Activation Functions on the Training of Overparametrized Neural Nets. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1038\/35016072","article-title":"Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit","volume":"405","author":"Hahnloser","year":"2000","journal-title":"Nature"},{"key":"ref_14","first-page":"2962","article-title":"Efficient and Robust Automated Machine Learning","volume":"Volume 28","author":"Cortes","year":"2015","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_15","unstructured":"Hutter, F., Hoos, H., and Leyton-Brown, K. (2014, January 21\u201326). An Efficient Approach for Assessing Hyperparameter Importance. Proceedings of the 31st International Conference on Machine Learning ICML, Beijing, China."},{"key":"ref_16","unstructured":"Probst, P., Bischl, B., and Boulesteix, A.L. (2018). Tunability: Importance of Hyperparameters of Machine Learning Algorithms. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"van Rijn, J.N., and Hutter, F. (2018, January 19\u201323). Hyperparameter Importance Across Datasets. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3220058"},{"key":"ref_18","unstructured":"Chen, B., Zhang, K., Ou, L., Ba, C., Wang, H., and Wang, C. (2020). Automatic Hyper-Parameter Optimization Based on Mapping Discovery from Data to Hyper-Parameters. arXiv."},{"key":"ref_19","unstructured":"Lee, A., Xin, D., Lee, D., and Parameswaran, A. (2020). Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"5455","DOI":"10.1007\/s10462-020-09825-6","article-title":"A survey of the recent architectures of deep convolutional neural networks","volume":"53","author":"Khan","year":"2020","journal-title":"Artif. Intell. Rev."},{"key":"ref_21","unstructured":"Lu, L., Shin, Y., Su, Y., and Karniadakis, G.E. (2019). Dying ReLU and Initialization: Theory and Numerical Examples. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Sperl, P., and B\u00f6ttinger, K. (2020). Optimizing Information Loss Towards Robust Neural Networks. arXiv.","DOI":"10.1145\/3477997.3478016"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4073","DOI":"10.1109\/TNNLS.2019.2952029","article-title":"Information Losses in Neural Classifiers From Sampling","volume":"31","author":"Foggo","year":"2020","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"ref_24","unstructured":"LeCun, Y., and Cortes, C. (2020, October 10). MNIST Handwritten Digit Database. Available online: http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"ref_25","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Cohen, G., Afshar, S., Tapson, J., and Schaik, A.V. (2017, January 14\u201319). EMNIST: Extending MNIST to handwritten letters. Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA.","DOI":"10.1109\/IJCNN.2017.7966217"},{"key":"ref_27","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_28","unstructured":"Chollet, F. (2021, January 10). Keras: The Python Deep Learning API. Available online: https:\/\/github.com\/fchollet\/keras."},{"key":"ref_29","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Nystrom, N.A., Levine, M.J., Roskies, R.Z., and Scott, J.R. (2015, January 26\u201330). Bridges: A Uniquely Flexible HPC Resource for New Communities and Data Analytics. Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure, XSEDE \u201915, St. Louis, MO, USA.","DOI":"10.1145\/2792745.2792775"},{"key":"ref_31","unstructured":"Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mannor, S., Peleg, D., and Rubinstein, R. (2005, January 7\u201311). The cross entropy method for classification. Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany.","DOI":"10.1145\/1102351.1102422"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Singla, A., Yuan, L., and Ebrahimi, T. (2016, January 16). Food\/Non-food Image Classification and Food Categorization using Pre-Trained GoogLeNet Model. Proceedings of the 2nd International Workshop on MADiMa, Amsterdam, The Netherlands.","DOI":"10.1145\/2986035.2986039"},{"key":"ref_34","unstructured":"Krizhevsky, A. (2021, January 15). Learning Multiple Layers of Features from Tiny Images. Available online: https:\/\/www.cs.toronto.edu\/~kriz\/cifar.html."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liu, J., Kanazawa, A., Jacobs, D., and Belhumeur, P. (2012). Dog Breed Classification Using Part Localization. Computer Vision\u2014ECCV, Springer.","DOI":"10.1007\/978-3-642-33718-5_13"},{"key":"ref_36","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis. IJCV"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015). Rethinking the Inception Architecture for Computer Vision. arXiv.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017). Xception: Deep Learning with Depthwise Separable Convolutions. arXiv.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_41","unstructured":"(2020, December 10). The Negative Log Likelihood Loss. Available online: https:\/\/pytorch.org\/docs\/stable\/generated\/torch.nn.NLLLoss.html."},{"key":"ref_42","unstructured":"Kingma, D.P., and Ba, J.L. (2017). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On Information and Sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Statist."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Lovric, M. (2011). Kullback-Leibler Divergence. International Encyclopedia of Statistical Science, Springer.","DOI":"10.1007\/978-3-642-04898-2"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/2\/51\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:19:58Z","timestamp":1760159998000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/2\/51"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,5]]},"references-count":44,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["a14020051"],"URL":"https:\/\/doi.org\/10.3390\/a14020051","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,5]]}}}