{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T15:44:09Z","timestamp":1774367049683,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,5,24]],"date-time":"2021-05-24T00:00:00Z","timestamp":1621814400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61803337"],"award-info":[{"award-number":["61803337"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The selection of the hyper-parameters plays a critical role in the task of prediction based on the recurrent neural networks (RNN). Traditionally, the hyper-parameters of the machine learning models are selected by simulations as well as human experiences. In recent years, multiple algorithms based on Bayesian optimization (BO) are developed to determine the optimal values of the hyper-parameters. In most of these methods, gradients are required to be calculated. In this work, the particle swarm optimization (PSO) is used under the BO framework to develop a new method for hyper-parameter optimization. The proposed algorithm (BO-PSO) is free of gradient calculation and the particles can be optimized in parallel naturally. So the computational complexity can be effectively reduced which means better hyper-parameters can be obtained under the same amount of calculation. Experiments are done on real world power load data, where the proposed method outperforms the existing state-of-the-art algorithms, BO with limit-BFGS-bound (BO-L-BFGS-B) and BO with truncated-newton (BO-TNC), in terms of the prediction accuracy. The errors of the prediction result in different models show that BO-PSO is an effective hyper-parameter optimization method.<\/jats:p>","DOI":"10.3390\/a14060163","type":"journal-article","created":{"date-parts":[[2021,5,24]],"date-time":"2021-05-24T12:38:01Z","timestamp":1621859881000},"page":"163","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["A New Hyper-Parameter Optimization Method for Power Load Forecast Based on Recurrent Neural Networks"],"prefix":"10.3390","volume":"14","author":[{"given":"Yaru","family":"Li","sequence":"first","affiliation":[{"name":"School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China"}]},{"given":"Yulai","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China"}]},{"given":"Yongping","family":"Cai","sequence":"additional","affiliation":[{"name":"School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"52528","DOI":"10.1109\/ACCESS.2020.2981141","article-title":"Hyper-parameter selection in convolutional neural networks using microcanonical optimization algorithm","volume":"8","year":"2020","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1925","DOI":"10.1109\/TASE.2020.2983061","article-title":"Data-driven approach for fault detection and diagnostic in semiconductor manufacturing","volume":"17","author":"Fan","year":"2020","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"101166","DOI":"10.1016\/j.aei.2020.101166","article-title":"Defective wafer detection using a denoising autoencoder for semiconductor manufacturing processes","volume":"46","author":"Fan","year":"2020","journal-title":"Adv. Eng. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Park, Y.J., Fan, S.K.S., and Hsu, C.Y. (2020). A Review on Fault Detection and Process Diagnostics in Industrial Processes. Processes, 8.","DOI":"10.3390\/pr8091123"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Pu, L., Zhang, X.L., Wei, S.J., Fan, X.T., and Xiong, Z.R. (2016, January 10\u201313). Target recognition of 3-d synthetic aperture radar images via deep belief network. Proceedings of the CIE International Conference, Guangzhou, China.","DOI":"10.1109\/RADAR.2016.8059199"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1007\/s11554-017-0717-0","article-title":"Polarimetric synthetic aperture radar image segmentation by convolutional neural network using graphical processing units","volume":"15","author":"Wang","year":"2016","journal-title":"J. Real-Time Image Process."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1109\/97.789608","article-title":"Maximum-likelihood DOA estimation by data-supported grid search","volume":"6","author":"Stoica","year":"1999","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_8","unstructured":"Bellman, R.E. (2015). Adaptive Control Processes: A Guided Tour, Princeton University Press."},{"key":"ref_9","first-page":"281","article-title":"Random search for hyper-parameter optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Frazier, P.I. (2018). A tutorial on Bayesian optimization. arXiv.","DOI":"10.1287\/educ.2018.0188"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1109\/JPROC.2015.2494218","article-title":"Taking the human out of the loop: A review of Bayesian optimization","volume":"104","author":"Shahriari","year":"2015","journal-title":"Proc. IEEE"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"52588","DOI":"10.1109\/ACCESS.2020.2981072","article-title":"Basic enhancement strategies when using Bayesian optimization for hyperparameter tuning of deep neural networks","volume":"8","author":"Cho","year":"2020","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1023\/A:1008306431147","article-title":"Efficient global optimization of expensive black-box functions","volume":"13","author":"Jones","year":"1998","journal-title":"J. Glob. Optim."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1007\/BF01589116","article-title":"On the limited memory BFGS method for large scale optimization","volume":"45","author":"Liu","year":"1989","journal-title":"Math. Program."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/S0377-0427(00)00426-X","article-title":"A survey of truncated-Newton methods","volume":"124","author":"Nash","year":"2000","journal-title":"J. Comput. Appl. Math."},{"key":"ref_16","first-page":"180","article-title":"Analysis of particle swarm optimization algorithm","volume":"3","author":"Bai","year":"2010","journal-title":"Comput. Inf. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1109\/TPWRD.2014.2301219","article-title":"Estimation of composite load model parameters using an improved particle swarm optimization method","volume":"30","author":"Regulski","year":"2014","journal-title":"IEEE Trans. Power Deliv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1542","DOI":"10.1016\/j.ces.2007.11.024","article-title":"Nonlinear parameter estimation through particle swarm optimization","volume":"63","author":"Schwaab","year":"2008","journal-title":"Chem. Eng. Sci."},{"key":"ref_19","unstructured":"Wenjing, Z. (2007, January 26\u201331). Parameter identification of LuGre friction model in servo system based on improved particle swarm optimization algorithm. Proceedings of the Chinese Control Conference, Zhangjiajie, China."},{"key":"ref_20","unstructured":"Bergstra, J., Bardenet, R., Bengio, Y., and K\u00e9gl, B. (2011, January 20). Algorithms for hyper-parameter optimization. Proceedings of the Neural Information Processing Systems Foundation, Granada, Spain."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Krajsek, K., and Mester, R. (2006, January 8\u201313). Marginalized Maximum a Posteriori Hyper-parameter Estimation for Global Optical Flow Techniques. Proceedings of the American Institute of Physics Conference, Paris, France.","DOI":"10.1063\/1.2423289"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1016\/j.rser.2015.03.035","article-title":"Regression analysis for prediction of residential energy consumption","volume":"47","author":"Fumo","year":"2015","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1278","DOI":"10.1109\/TSTE.2019.2922782","article-title":"Linear non-causal optimal control of an attenuator type wave energy converter m4","volume":"11","author":"Liao","year":"2019","journal-title":"IEEE Trans. Sustain. Energy"},{"key":"ref_24","first-page":"105","article-title":"Cross-scale recurrent neural network based on Zoneout and its application in short-term power load forecasting","volume":"47","author":"Zhuang","year":"2020","journal-title":"Comput. Sci."},{"key":"ref_25","unstructured":"Snoek, J., Larochelle, H., and Adams, R.P. (2012). Practical bayesian optimization of machine learning algorithms. arXiv."},{"key":"ref_26","unstructured":"Brochu, E., Cora, V.M., and De Freitas, N. (2010). A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Rasmussen, C.E. (2003). Gaussian processes in machine learning. Summer School on Machine Learning, Springer.","DOI":"10.1007\/978-3-540-28650-9_4"},{"key":"ref_28","unstructured":"Mahendran, N., Wang, Z., Hamze, F., and De Freitas, N. (2012, January 21\u201323). Adaptive MCMC with Bayesian optimization. Proceedings of the Artificial Intelligence and Statistics, La Palma, Canary Islands, Spain."},{"key":"ref_29","first-page":"1809","article-title":"Entropy Search for Information-Efficient Global Optimization","volume":"13","author":"Hennig","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Toscano-Palmerin, S., and Frazier, P.I. (2018). Bayesian optimization with expensive integrands. arXiv.","DOI":"10.1007\/978-3-319-91436-7_7"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1016\/j.neucom.2019.11.004","article-title":"Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes","volume":"380","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_32","unstructured":"Oh, C., Tomczak, J.M., and Gavves, E.M. (2019). Combinatorial bayesian optimization using the graph cartesian product. arXiv."},{"key":"ref_33","unstructured":"Dai, Z., Yu, H., Low, B.K.H., and Jaillet, P. (2019, January 9\u201315). Bayesian optimization meets Bayesian optimal stopping. Proceedings of the PMLR, Long Beach, CA, USA."},{"key":"ref_34","unstructured":"Gong, C., Peng, J., and Liu, Q. (2019, January 9\u201315). Quantile stein variational gradient descent for batch bayesian optimization. Proceedings of the PMLR, Long Beach, CA, USA."},{"key":"ref_35","unstructured":"Paria, B., Kandasamy, K., and P\u00f3czos, B. (2020, January 3\u20136). A flexible framework for multi-objective Bayesian optimization using random scalarizations. Proceedings of the PMLR, Toronto, AB, Canada."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Fan, S.K.S., and Jen, C.H. (2019). An enhanced partial search to particle swarm optimization for unconstrained optimization. Mathematics, 7.","DOI":"10.3390\/math7040357"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1016\/j.ejor.2006.06.034","article-title":"A hybrid simplex search and particle swarm optimization for unconstrained optimization","volume":"181","author":"Fan","year":"2007","journal-title":"Eur. J. Oper. Res."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Hung, C., and Wan, L. (April, January 30). Hybridization of particle swarm optimization with the k-means algorithm for image classification. Proceedings of the 2009 IEEE Symposium on Computational Intelligence for Image Processing, Nashville, TN, USA.","DOI":"10.1109\/CIIP.2009.4937881"},{"key":"ref_39","unstructured":"Srinivas, N., Krause, A., Kakade, S.M., and Seeger, M. (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.ipl.2006.10.005","article-title":"Stochastic convergence analysis and parameter selection of the standard particle swarm optimization algorithm","volume":"102","author":"Jiang","year":"2007","journal-title":"Inf. Process. Lett."},{"key":"ref_41","unstructured":"Zheng, Y., Ma, L., Zhang, L., and Qian, J. (2003, January 5). On the convergence analysis and parameter selection in particle swarm optimization. Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No. 03EX693), Xi\u2019an, China."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/6\/163\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:07:01Z","timestamp":1760162821000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/6\/163"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,24]]},"references-count":41,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["a14060163"],"URL":"https:\/\/doi.org\/10.3390\/a14060163","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,24]]}}}