{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T23:06:44Z","timestamp":1761174404915,"version":"build-2065373602"},"reference-count":50,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T00:00:00Z","timestamp":1761091200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Tianlei Wang","award":["2024ZDZX1009"],"award-info":[{"award-number":["2024ZDZX1009"]}]},{"name":"Xiaoxi Hao","award":["2022ZDZX3034"],"award-info":[{"award-number":["2022ZDZX3034"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>In response to the issues of premature convergence and insufficient parameter control in Particle Swarm Optimization (PSO) for high-dimensional complex optimization problems, this paper proposes a Multi-Strategy Topological Particle Swarm Optimization algorithm (MSTPSO). The method builds upon a reinforcement learning-driven topological switching framework, where Q-learning dynamically selects among fully informed topology, small-world topology, and exemplar-set topology to achieve an adaptive balance between global exploration and local exploitation. Furthermore, the algorithm integrates differential evolution perturbations and a global optimal restart strategy based on stagnation detection, together with a dual-layer experience replay mechanism to enhance population diversity at multiple levels and strengthen the ability to escape local optima. Experimental results on 29 CEC2017 benchmark functions, compared against various PSO variants and other advanced evolutionary algorithms, show that MSTPSO achieves superior fitness performance and exhibits stronger stability on high-dimensional and complex functions. Ablation studies further validate the critical contribution of the Q-learning-based multi-topology control and stagnation detection mechanisms to performance improvement. Overall, MSTPSO demonstrates significant advantages in convergence accuracy and global search capability.<\/jats:p>","DOI":"10.3390\/a18110672","type":"journal-article","created":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T07:03:51Z","timestamp":1761116631000},"page":"672","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Q-Learning-Based Multi-Strategy Topology Particle Swarm Optimization Algorithm"],"prefix":"10.3390","volume":"18","author":[{"given":"Xiaoxi","family":"Hao","sequence":"first","affiliation":[{"name":"School of Mechanical and Automation Engineering, Wuyi University, Jiangmen 529020, China"}]},{"given":"Shenwei","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Mechanical and Automation Engineering, Wuyi University, Jiangmen 529020, China"}]},{"given":"Xiaotong","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Mechanical and Automation Engineering, Wuyi University, Jiangmen 529020, China"}]},{"given":"Tianlei","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Mechanical and Automation Engineering, Wuyi University, Jiangmen 529020, China"}]},{"given":"Guangfan","family":"Qiu","sequence":"additional","affiliation":[{"name":"School of Mechanical and Automation Engineering, Wuyi University, Jiangmen 529020, China"}]},{"given":"Zhiqiang","family":"Zeng","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Dongguan University of Technology, Dongguan 523808, China"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Tanabe, R., and Fukunaga, A. (2013, January 20\u201323). Success-history based parameter adaptation for differential evolution. Proceedings of the 2013 IEEE Congress on Evolutionary Computation, Cancun, Mexico.","DOI":"10.1109\/CEC.2013.6557555"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1039","DOI":"10.1109\/TEVC.2023.3291697","article-title":"A surrogate-assisted differential evolution with knowledge transfer for expensive incremental optimization problems","volume":"28","author":"Liu","year":"2023","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Lambora, A., Gupta, K., and Chopra, K. (2019, January 14\u201316). Genetic algorithm\u2014A literature review. Proceedings of the 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), Faridabad, India.","DOI":"10.1109\/COMITCon.2019.8862255"},{"key":"ref_4","unstructured":"Ngo, L., Ha, H., Chan, J., Nguyen, V., and Zhang, H. (2024). High-dimensional Bayesian optimization via covariance matrix adaptation strategy. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"106955","DOI":"10.1016\/j.asoc.2020.106955","article-title":"Artificial bee colony algorithm based on adaptive neighborhood search and Gaussian perturbation","volume":"100","author":"Xiao","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_6","unstructured":"Shi, Y., and Eberhart, R. (1998, January 25\u201327). A modified particle swarm optimizer. Proceedings of the 1998 IEEE International Conference on Evolutionary Computation Proceedings, San Diego, CA, USA. IEEE World Congress on Computational Intelligence (Cat. No. 98TH8360)."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Eberhart, R., and Shi, Y. (2005, January 28\u201330). Particle swarm optimization and its applications to VLSI design and video technology. Proceedings of the 2005 IEEE International Workshop on VLSI Design and Video Technology, Suzhou, China.","DOI":"10.1109\/IWVDVT.2005.1504445"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1109\/TEVC.2004.826071","article-title":"Self-organizing hierarchical particle swarm optimizer with time-varying acceleration coefficients","volume":"8","author":"Ratnaweera","year":"2004","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Chen, Q., Li, C., and Guo, W. (2009, January 25\u201326). Railway passenger volume forecast based on IPSO-BP neural network. Proceedings of the 2009 International Conference on Information Technology and Computer Science, Kiev, Ukraine.","DOI":"10.1109\/ITCS.2009.187"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1016\/j.asoc.2019.01.004","article-title":"A hierarchical simple particle swarm optimization with mean dimensional information","volume":"76","author":"Liu","year":"2019","journal-title":"Appl. Soft Comput."},{"key":"ref_11","first-page":"831","article-title":"Hybrid gray wolf and particle swarm optimization for feature selection","volume":"16","author":"Eid","year":"2020","journal-title":"Int. J. Innov. Comput. Inf. Control"},{"key":"ref_12","first-page":"100108","article-title":"An hybrid particle swarm optimization with crow search algorithm for feature selection","volume":"6","author":"Adamu","year":"2021","journal-title":"Mach. Learn. Appl."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"104263","DOI":"10.1016\/j.engappai.2021.104263","article-title":"A novel hybrid gravitational search particle swarm optimization algorithm","volume":"102","author":"Khan","year":"2021","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"6905","DOI":"10.1007\/s40747-023-01082-8","article-title":"Heterogeneous differential evolution particle swarm optimization with local search","volume":"9","author":"Lin","year":"2023","journal-title":"Complex Intell. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1016\/j.eij.2022.04.003","article-title":"Mutual learning differential particle swarm optimization","volume":"23","author":"Lin","year":"2022","journal-title":"Egypt. Inform. J."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"129460","DOI":"10.1016\/j.neucom.2025.129460","article-title":"Multi Actors-Critic based particle swarm optimization algorithm","volume":"624","author":"Chen","year":"2025","journal-title":"Neurocomputing"},{"key":"ref_17","first-page":"140","article-title":"Quality monitoring of tractor hydraulic oil based on improved PSO-BPNN","volume":"45","author":"Zhongxing","year":"2024","journal-title":"J. Chin. Agric. Mech."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"106693","DOI":"10.1016\/j.asoc.2020.106693","article-title":"A fitness landscape ruggedness multiobjective differential evolution algorithm with a reinforcement learning strategy","volume":"96","author":"Huang","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2199","DOI":"10.1007\/s40747-021-00395-w","article-title":"Reinforcement learning-based particle swarm optimization for sewage treatment control","volume":"7","author":"Lu","year":"2021","journal-title":"Complex Intell. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1272","DOI":"10.1109\/TSMCB.2005.850530","article-title":"A hierarchical particle swarm optimizer and its adaptive variant","volume":"35","author":"Janson","year":"2005","journal-title":"IEEE Trans. Syst. Man Cybern. Part"},{"key":"ref_21","first-page":"919","article-title":"Particle swarm optimization algorithm with self-correcting and dimension by dimension learning capabilities","volume":"42","author":"Zhang","year":"2021","journal-title":"J. Chin. Comput. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.ins.2019.08.065","article-title":"An expanded particle swarm optimization based on multi-exemplar and forgetting ability","volume":"508","author":"Xia","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_23","first-page":"3209140","article-title":"A personalized recommendation algorithm based on weighted information entropy and particle swarm optimization","volume":"2021","author":"Jiang","year":"2021","journal-title":"Mob. Inf. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Engelbrecht, A.P. (2007). Computational Intelligence: An Introduction, Wiley Online Library.","DOI":"10.1002\/9780470512517"},{"key":"ref_25","unstructured":"Liu, Q., Van Wyk, B.J., and Sun, Y. (2015, January 15\u201317). Small world network based dynamic topology for particle swarm optimization. Proceedings of the 2015 11th International Conference on Natural Computation (ICNC), Zhangjiajie, China."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"9290","DOI":"10.1109\/TCYB.2020.3029748","article-title":"A dynamic neighborhood-based switching particle swarm optimization algorithm","volume":"52","author":"Zeng","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"101019","DOI":"10.1016\/j.swevo.2021.101019","article-title":"Neighborhood-based particle swarm optimization with discrete crossover for nonlinear equation systems","volume":"69","author":"Pan","year":"2022","journal-title":"Swarm Evol. Comput."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"108731","DOI":"10.1016\/j.asoc.2022.108731","article-title":"Pyramid particle swarm optimization with novel strategies of competition and cooperation","volume":"121","author":"Li","year":"2022","journal-title":"Appl. Soft Comput."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"101274","DOI":"10.1016\/j.swevo.2023.101274","article-title":"Reinforcement learning-based particle swarm optimization with neighborhood differential mutation strategy","volume":"78","author":"Li","year":"2023","journal-title":"Swarm Evol. Comput."},{"key":"ref_30","first-page":"102","article-title":"Time domain waveform synthesis method of shock response spectrum based on PSO-LSN algorithm","volume":"43","author":"Jiang","year":"2024","journal-title":"J. Vib. Shock"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1016\/j.asoc.2018.09.034","article-title":"Dynamic parameter adaptation in metaheuristics using gradient approximation and line search","volume":"74","author":"Tatsis","year":"2019","journal-title":"Appl. Soft Comput."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Liu, Y., Lu, H., Cheng, S., and Shi, Y. (2019, January 10\u201313). An adaptive online parameter control algorithm for particle swarm optimization based on reinforcement learning. Proceedings of the 2019 IEEE Congress on Evolutionary Computation (CEC), Wellington, New Zealand.","DOI":"10.1109\/CEC.2019.8790035"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"5585","DOI":"10.1007\/s40747-023-01012-8","article-title":"Reinforcement-learning-based parameter adaptation method for particle swarm optimization","volume":"9","author":"Yin","year":"2023","journal-title":"Complex Intell. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"116417","DOI":"10.1016\/j.eswa.2021.116417","article-title":"Q-learning embedded sine cosine algorithm (QLESCA)","volume":"193","author":"Hamad","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"18630","DOI":"10.1007\/s10489-022-04446-8","article-title":"Feature selection of pre-trained shallow CNN using the QLESCA optimizer: COVID-19 detection as a case study","volume":"53","author":"Hamad","year":"2023","journal-title":"Appl. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1016\/j.ins.2022.04.053","article-title":"A reinforcement learning level-based particle swarm optimization algorithm for large-scale optimization","volume":"602","author":"Wang","year":"2022","journal-title":"Inf. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_38","first-page":"279","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Qiang, W., and Zhongli, Z. (2011, January 19\u201322). Reinforcement learning model, algorithms and its application. Proceedings of the 2011 International Conference on Mechatronic Science, Electric Engineering and Computer (MEC), Jilin, China.","DOI":"10.1109\/MEC.2011.6025669"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Awad, N.H., Ali, M.Z., and Suganthan, P.N. (2017, January 5\u20138). Ensemble sinusoidal differential covariance matrix adaptation with Euclidean neighborhood for solving CEC2017 benchmark problems. Proceedings of the 2017 IEEE Congress on Evolutionary Computation (CEC), Donostia\/San Sebastian, Spain.","DOI":"10.1109\/CEC.2017.7969336"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1109\/TEVC.2004.826074","article-title":"The fully informed particle swarm: Simpler, maybe better","volume":"8","author":"Mendes","year":"2004","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"101222","DOI":"10.1016\/j.swevo.2022.101222","article-title":"Incorporating surprisingly popular algorithm and euclidean distance-based adaptive topology into PSO","volume":"76","author":"Wu","year":"2023","journal-title":"Swarm Evol. Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1709","DOI":"10.3390\/modelling5040089","article-title":"Deep Q-network-enhanced self-tuning control of particle swarm optimization","volume":"5","author":"Aoun","year":"2024","journal-title":"Modelling"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"von Eschwege, D., and Engelbrecht, A. (2024). Soft actor-critic approach to self-adaptive particle swarm optimisation. Mathematics, 12.","DOI":"10.3390\/math12223481"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"101257","DOI":"10.1016\/j.swevo.2023.101257","article-title":"Regional-modal optimization problems and corresponding normal search particle swarm optimization algorithm","volume":"78","author":"Yuan","year":"2023","journal-title":"Swarm Evol. Comput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"109660","DOI":"10.1016\/j.asoc.2022.109660","article-title":"A self-adaptive gradient-based particle swarm optimization algorithm with dynamic population topology","volume":"130","author":"Zhang","year":"2022","journal-title":"Appl. Soft Comput."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"101486","DOI":"10.1016\/j.swevo.2024.101486","article-title":"A multi-swarm optimizer with a reinforcement learning mechanism for large-scale optimization","volume":"86","author":"Wang","year":"2024","journal-title":"Swarm Evol. Comput."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/s10586-024-04954-x","article-title":"Divine Religions Algorithm: A novel social-inspired metaheuristic algorithm for engineering and continuous optimization problems","volume":"28","author":"Mozhdehi","year":"2025","journal-title":"Clust. Comput."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10586-024-04950-1","article-title":"R\u00fcppell\u2019s fox optimizer: A novel meta-heuristic approach for solving global optimization problems","volume":"28","author":"Braik","year":"2025","journal-title":"Clust. Comput."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"101935","DOI":"10.1016\/j.swevo.2025.101935","article-title":"Three stage based reinforcement learning for combining multiple metaheuristic algorithms","volume":"95","author":"Liu","year":"2025","journal-title":"Swarm Evol. Comput."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/11\/672\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T07:08:03Z","timestamp":1761116883000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/11\/672"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,22]]},"references-count":50,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["a18110672"],"URL":"https:\/\/doi.org\/10.3390\/a18110672","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,22]]}}}