{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T00:22:02Z","timestamp":1776126122125,"version":"3.50.1"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,2,11]],"date-time":"2023-02-11T00:00:00Z","timestamp":1676073600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,2,11]],"date-time":"2023-02-11T00:00:00Z","timestamp":1676073600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003708","name":"Korea Institute of Science and Technology Information","doi-asserted-by":"publisher","award":["K-22-L04-C07-S01"],"award-info":[{"award-number":["K-22-L04-C07-S01"]}],"id":[{"id":"10.13039\/501100003708","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Intell Syst"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Activation functions are essential in deep learning, and the rectified linear unit (ReLU) is the most widely used activation function to solve the vanishing gradient problem. However, owing to the dying ReLU problem and bias shift effect, deep learning models using ReLU cannot exploit the potential benefits of negative values. Numerous ReLU variants have been proposed to address this issue. In this study, we propose Dynamic Parametric ReLU (DPReLU), which can dynamically control the overall functional shape of ReLU with four learnable parameters. The parameters of DPReLU are determined by training rather than by humans, thereby making the formulation more suitable and flexible for each model and dataset. Furthermore, we propose an appropriate and robust weight initialization method for DPReLU. To evaluate DPReLU and its weight initialization method, we performed two experiments on various image datasets: one using an autoencoder for image generation and the other using the ResNet50 for image classification. The results show that DPReLU and our weight initialization method provide faster convergence and better accuracy than the original ReLU and the previous ReLU variants.<\/jats:p>","DOI":"10.1007\/s44196-023-00186-w","type":"journal-article","created":{"date-parts":[[2023,2,11]],"date-time":"2023-02-11T08:40:13Z","timestamp":1676104813000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["DPReLU: Dynamic Parametric Rectified Linear Unit and Its Proper Weight Initialization Method"],"prefix":"10.1007","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6931-7469","authenticated-orcid":false,"given":"Donghun","family":"Yang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kien Mai","family":"Ngoc","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Iksoo","family":"Shin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Myunggwon","family":"Hwang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,2,11]]},"reference":[{"issue":"6557","key":"186_CR1","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1126\/science.abj8754","volume":"373","author":"M Baek","year":"2021","unstructured":"Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G.R., Wang, J., Cong, Q., Kinch, L.N., Schaeffer, R.D., et al.: Accurate prediction of protein structures and interactions using a three-track neural network. Science 373(6557), 871\u2013876 (2021)","journal-title":"Science"},{"key":"186_CR2","doi-asserted-by":"crossref","unstructured":"Barba, E., Procopio, L., Navigli, R.: ConSec: Word sense disambiguation as continuous sense comprehension. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1492\u20131503 (2021)","DOI":"10.18653\/v1\/2021.emnlp-main.112"},{"key":"186_CR3","doi-asserted-by":"crossref","unstructured":"Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Advances in neural information processing systems, pp. 153\u2013160 (2007)","DOI":"10.7551\/mitpress\/7503.003.0024"},{"key":"186_CR4","unstructured":"Erhan, D., Manzagol, P.A., Bengio, Y., Bengio, S., Vincent, P.: The difficulty of training deep architectures and the effect of unsupervised pre-training. In: Artificial Intelligence and Statistics, pp. 153\u2013160 (2009)"},{"key":"186_CR5","unstructured":"Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. In: International Conference on Learning Representations (2020)"},{"key":"186_CR6","doi-asserted-by":"crossref","unstructured":"Fu, B., Zhang, W., Hu, G., Dai, X., Huang, S., Chen, J.: Dual side deep context-aware modulation for social recommendation. In: Proceedings of the Web Conference 2021, pp. 2524\u20132534 (2021)","DOI":"10.1145\/3442381.3449940"},{"key":"186_CR7","unstructured":"Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249\u2013256 (2010)"},{"key":"186_CR8","doi-asserted-by":"crossref","unstructured":"Han, S.C., Lim, T., Long, S., Burgstaller, B., Poon, J.: Glocal-K: Global and local kernels for recommender systems. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3063\u20133067 (2021)","DOI":"10.1145\/3459637.3482112"},{"key":"186_CR9","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026\u20131034 (2015)","DOI":"10.1109\/ICCV.2015.123"},{"key":"186_CR10","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778 (2016)","DOI":"10.1109\/CVPR.2016.90"},{"issue":"7","key":"186_CR11","doi-asserted-by":"publisher","first-page":"1527","DOI":"10.1162\/neco.2006.18.7.1527","volume":"18","author":"GE Hinton","year":"2006","unstructured":"Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527\u20131554 (2006)","journal-title":"Neural Comput."},{"issue":"7873","key":"186_CR12","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","volume":"596","author":"J Jumper","year":"2021","unstructured":"Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., \u017d\u00eddek, A., Potapenko, A., et al.: Highly accurate protein structure prediction with alphafold. Nature 596(7873), 583\u2013589 (2021)","journal-title":"Nature"},{"key":"186_CR13","first-page":"18661","volume":"33","author":"P Khosla","year":"2020","unstructured":"Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Adv. Neural Inform. Process. Syst. 33, 18661\u201318673 (2020)","journal-title":"Adv. Neural Inform. Process. Syst."},{"issue":"3","key":"186_CR14","doi-asserted-by":"publisher","first-page":"167","DOI":"10.3390\/bios12030167","volume":"12","author":"JK Kim","year":"2022","unstructured":"Kim, J.K., Bae, M.N., Lee, K., Kim, J.C., Hong, S.G.: Explainable artificial intelligence and wearable sensor-based gait analysis to identify patients with osteopenia and sarcopenia in daily life. Biosensors 12(3), 167 (2022)","journal-title":"Biosensors"},{"key":"186_CR15","unstructured":"Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Master\u2019s thesis, Department of Computer Science, University of Toronto (2009)"},{"key":"186_CR16","unstructured":"LeCun, Y.: The mnist database of handwritten digits. https:\/\/www.tensorflow.org\/datasets\/catalog\/mnist (1998)"},{"key":"186_CR17","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/978-3-642-35289-8_3","volume-title":"Neural networks: tricks of the trade","author":"YA LeCun","year":"2012","unstructured":"LeCun, Y.A., Bottou, L., Orr, G.B., M\u00fcller, K.R.: Efficient backprop. In: Neural networks: tricks of the trade, pp. 9\u201348. Springer (2012)"},{"key":"186_CR18","unstructured":"Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. ICML, vol. 30, p. 3 (2013)"},{"key":"186_CR19","doi-asserted-by":"crossref","unstructured":"Mai Ngoc, K., Yang, D., Shin, I., Kim, H., Hwang, M.: Dprelu: Dynamic parametric rectified linear unit. In: The 9th International Conference on Smart Media and Applications, pp. 121\u2013125 (2020)","DOI":"10.1145\/3426020.3426049"},{"key":"186_CR20","unstructured":"Mishkin, D., Matas, J.: All you need is a good init. arXiv preprint arXiv:1511.06422 (2015)"},{"key":"186_CR21","unstructured":"Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: ICML\u201910: Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 807\u2013814 (2010)"},{"key":"186_CR22","unstructured":"Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)"},{"key":"186_CR23","unstructured":"Nwankpa, C., Ijomah, W., Gachagan, A., Marshall, S.: Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378 (2018)"},{"key":"186_CR24","doi-asserted-by":"crossref","unstructured":"Qiu, S., Xu, X., Cai, B.: FReLU: Flexible rectified linear units for improving convolutional neural networks. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 1223\u20131228. IEEE (2018)","DOI":"10.1109\/ICPR.2018.8546022"},{"issue":"21","key":"186_CR25","doi-asserted-by":"publisher","first-page":"7557","DOI":"10.3390\/app10217557","volume":"10","author":"C Ronran","year":"2020","unstructured":"Ronran, C., Lee, S., Jang, H.J.: Delayed combination of feature embedding in bidirectional lstm crf for ner. Appl. Sci. 10(21), 7557 (2020)","journal-title":"Appl. Sci."},{"key":"186_CR26","doi-asserted-by":"crossref","unstructured":"Sharma, S.: Activation functions in neural networks. Towards Data Science 6(12), 310\u2013316 (2017)","DOI":"10.33564\/IJEAST.2020.v04i12.054"},{"key":"186_CR27","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)"},{"key":"186_CR28","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1\u20139 (2015)","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"186_CR29","first-page":"908","volume":"13","author":"YW Teh","year":"2000","unstructured":"Teh, Y.W., Hinton, G.E.: Rate-coded restricted Boltzmann machines for face recognition. Adv. Neural Inform. Process. Syst. 13, 908\u2013914 (2000)","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"186_CR30","doi-asserted-by":"crossref","unstructured":"Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops (2018)","DOI":"10.1007\/978-3-030-11021-5_5"},{"key":"186_CR31","unstructured":"Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)"},{"key":"186_CR32","unstructured":"Yang, D., Hwang, M.: ADADL: Automatic dementia identification model based on activities of daily living using smart home sensor data. In: The Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-2022), Workshop: Trustworthy AI for Healthcare (2022)"},{"issue":"5","key":"186_CR33","doi-asserted-by":"publisher","first-page":"567","DOI":"10.3390\/electronics10050567","volume":"10","author":"D Yang","year":"2021","unstructured":"Yang, D., Mai Ngoc, K., Shin, I., Lee, K.H., Hwang, M.: Ensemble-based out-of-distribution detection. Electronics 10(5), 567 (2021)","journal-title":"Electronics"},{"key":"186_CR34","doi-asserted-by":"crossref","unstructured":"Yang, D., Shin, I., Kien, M.N., Kim, H., Yu, C., Hwang, M.: Out-of-distribution detection based on distance metric learning. In: The 9th International Conference on Smart Media and Applications, pp. 214\u2013218 (2020)","DOI":"10.1145\/3426020.3426076"},{"key":"186_CR35","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Zhang, Z., Lew, L.: PokeBNN: A binary pursuit of lightweight accuracy. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (2022)","DOI":"10.1109\/CVPR52688.2022.01215"}],"container-title":["International Journal of Computational Intelligence Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-023-00186-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44196-023-00186-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-023-00186-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,6]],"date-time":"2023-12-06T18:24:23Z","timestamp":1701887063000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44196-023-00186-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,11]]},"references-count":35,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["186"],"URL":"https:\/\/doi.org\/10.1007\/s44196-023-00186-w","relation":{},"ISSN":["1875-6883"],"issn-type":[{"value":"1875-6883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,11]]},"assertion":[{"value":"5 August 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 January 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 February 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"11"}}