{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T15:03:13Z","timestamp":1772550193435,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T00:00:00Z","timestamp":1692316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["61973203"],"award-info":[{"award-number":["61973203"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["72271048"],"award-info":[{"award-number":["72271048"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>In recent years, integration of production scheduling and machine maintenance has gained increasing attention in order to improve the stability and efficiency of flowshop manufacturing systems. This paper proposes a Q-learning-based aquila optimizer (QL-AO) for solving the integrated optimization problem of blocking flowshop scheduling and preventive maintenance since blocking in the jobs processing requires to be considered in the practice manufacturing environments. In the proposed algorithmic framework, a Q-learning algorithm is designed to adaptively adjust the selection probabilities of four key population update strategies in the classic aquila optimizer. In addition, five local search methods are employed to refine the quality of the individuals according to their fitness level. A series of numerical experiments are carried out according to two groups of flowshop scheduling benchmark. Experimental results show that QL-AO significantly outperforms six peer algorithms and two state-of-the-art hybrid algorithms based on Q-Learning on the investigated integrated scheduling problem. Additionally, the proposed Q-learning and local search strategies are effective in improving its performance.<\/jats:p>","DOI":"10.3390\/sym15081600","type":"journal-article","created":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T10:14:38Z","timestamp":1692353678000},"page":"1600","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Integrated Optimization of Blocking Flowshop Scheduling and Preventive Maintenance Using a Q-Learning-Based Aquila Optimizer"],"prefix":"10.3390","volume":"15","author":[{"given":"Zhenpeng","family":"Ge","sequence":"first","affiliation":[{"name":"College of Information Science and Engineering, Northeastern University, Shenyang 110819, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8954-0876","authenticated-orcid":false,"given":"Hongfeng","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Northeastern University, Shenyang 110819, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"publisher","first-page":"108097","DOI":"10.1016\/j.ress.2021.108097","article-title":"Joint Planning of Production and Inspection of Parallel Machines with Two-phase of Failure","volume":"217","author":"Azimpoor","year":"2022","journal-title":"Rellab. Eng. Syst. Saf."},{"key":"ref_2","doi-asserted-by":"publisher","first-page":"3891","DOI":"10.1080\/00207543.2010.492405","article-title":"Simultaneous scheduling of production and maintenance tasks in the job shop","volume":"49","author":"Sassi","year":"2011","journal-title":"Int. J. Prod. Res."},{"key":"ref_3","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1108\/JQME-04-2016-0014","article-title":"Preventive Maintenance (PM) planning: A review","volume":"23","author":"Basri","year":"2017","journal-title":"J. Qual. Maint. Eng."},{"key":"ref_4","doi-asserted-by":"publisher","first-page":"101339","DOI":"10.1016\/j.aei.2021.101339","article-title":"Integrated scheduling and flexible maintenance in deteriorating multi-state single machine system using a reinforcement learning approach","volume":"49","author":"Wang","year":"2021","journal-title":"Adv. Eng. Inform."},{"key":"ref_5","doi-asserted-by":"publisher","first-page":"105823","DOI":"10.1016\/j.cor.2022.105823","article-title":"Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer Q-learning algorithm","volume":"144","author":"Yan","year":"2022","journal-title":"Comput. Oper. Res."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Liu, L.X., and Shi, L.Y. (2022). Automatic Design of Efficient Heuristics for Two-Stage Hybrid Flow Shop Scheduling. Symmetry, 14.","DOI":"10.3390\/sym14040632"},{"key":"ref_7","doi-asserted-by":"publisher","first-page":"516","DOI":"10.1016\/j.compchemeng.2015.10.003","article-title":"Preprocessing and tightening methods for time-indexed chemical production scheduling models","volume":"84","author":"Merchan","year":"2016","journal-title":"Comput. Chem. Eng."},{"key":"ref_8","doi-asserted-by":"publisher","first-page":"960","DOI":"10.1016\/j.cor.2009.08.001","article-title":"A two-stage flow shop scheduling problem on a batching machine and a discrete machine with blocking and shared setup times","volume":"37","author":"Gong","year":"2010","journal-title":"Comput. Oper. Res."},{"key":"ref_9","doi-asserted-by":"publisher","first-page":"2543","DOI":"10.1016\/j.cor.2013.01.024","article-title":"A scheduling problem in blocking hybrid flow shop robotic cells with multiple robots","volume":"40","author":"Elmi","year":"2013","journal-title":"Comput. Oper. Res."},{"key":"ref_10","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.eswa.2019.06.069","article-title":"The blocking flow shop scheduling problem: A comprehensive and conceptual review","volume":"137","author":"Miyata","year":"2019","journal-title":"Expert. Syst. Appl."},{"key":"ref_11","first-page":"287","article-title":"Optimization and approximation in deterministic sequencing and scheduling: A survey","volume":"5","author":"Graham","year":"1979","journal-title":"ADM"},{"key":"ref_12","doi-asserted-by":"publisher","first-page":"510","DOI":"10.1287\/opre.44.3.510","article-title":"A survey of machine scheduling problems with blocking and no-wait in process","volume":"44","author":"Hall","year":"1996","journal-title":"Oper. Res."},{"key":"ref_13","doi-asserted-by":"publisher","first-page":"109143","DOI":"10.1016\/j.ress.2023.109143","article-title":"Integrated optimization of non-permutation flow shop scheduling and maintenance planning with variable processing speed","volume":"234","author":"Jiang","year":"2023","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_14","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1016\/S0925-5273(99)00104-8","article-title":"Minimizing makespan in a blocking flowshop using genetic algorithms","volume":"70","author":"Caraffa","year":"2001","journal-title":"Int. J. Prod. Econ."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Liang, J.J., Pan, Q.K., Chen, T.J., and Wang, L. (2010, January 23\u201326). Dynamic Multi-swarm Particle Swarm Optimizer for blocking flow shop scheduling. Proceedings of the IEEE International Conference on Fuzzy Systems, Changsha, China.","DOI":"10.1109\/BICTA.2010.5645309"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Li, M.B., Xiong, H., and Lei, D.M. (2022). An Artificial Bee Colony with Adaptive Competition for the Unrelated Parallel Machine Scheduling Problem with Additional Resources and Maintenance. Symmetry, 14.","DOI":"10.3390\/sym14071380"},{"key":"ref_17","doi-asserted-by":"publisher","first-page":"119898","DOI":"10.1016\/j.eswa.2023.119898","article-title":"A reinforcement learning-based hybrid Aquila Optimizer and improved Arithmetic Optimization Algorithm for global optimization","volume":"224","author":"Liu","year":"2023","journal-title":"Expert. Syst. Appl."},{"key":"ref_18","doi-asserted-by":"publisher","first-page":"108461","DOI":"10.1016\/j.compeleceng.2022.108461","article-title":"A novel hybrid Chaotic Aquila Optimization algorithm with Simulated Annealing for Unmanned Aerial Vehicles path planning","volume":"104","author":"Meraihi","year":"2022","journal-title":"Comput. Electr. Eng."},{"key":"ref_19","doi-asserted-by":"publisher","first-page":"105592","DOI":"10.1016\/j.engappai.2022.105592","article-title":"Binary Aquila Optimizer for 0\u20131 knapsack problems","volume":"118","author":"Bas","year":"2023","journal-title":"Eng. Appl. Artif. Intel."},{"key":"ref_20","doi-asserted-by":"publisher","first-page":"141","DOI":"10.32604\/cmc.2023.030074","article-title":"Intelligent aquila optimization algorithm-based node localization scheme for wireless sensor networks","volume":"74","author":"Agarwal","year":"2023","journal-title":"CMC-Comput. Mater. Con."},{"key":"ref_21","doi-asserted-by":"publisher","first-page":"106375","DOI":"10.1016\/j.ocecoaman.2022.106375","article-title":"An adaptive heuristic algorithm based on reinforcement learning for ship scheduling optimization problem","volume":"230","author":"Li","year":"2022","journal-title":"Ocean. Coast. Manage."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Mao, J., Hu, X.L., Pan, Q.K., Miao, Z., and Tasgetiren, M.F. (2020, January 27\u201329). An improved discrete artificial bee colony algorithm for the distributed permutation flowshop scheduling problem with preventive maintenance. Proceedings of the 39th Chinese Control Conference, Shenyang, China.","DOI":"10.23919\/CCC50068.2020.9188697"},{"key":"ref_23","doi-asserted-by":"publisher","first-page":"129248","DOI":"10.1016\/j.jclepro.2021.129248","article-title":"Mathematical model and enhanced cooperative co-evolutionary algorithm for scheduling energy-efficient manufacturing cell","volume":"326","author":"Cheng","year":"2021","journal-title":"J. Clean. Prod."},{"key":"ref_24","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1007\/s10696-021-09403-0","article-title":"Integrating preventive maintenance to two-stage assembly flow shop scheduling: MILP model, constructive heuristics and meta-heuristics","volume":"34","author":"Zhang","year":"2022","journal-title":"Flex. Serv. Manuf. J."},{"key":"ref_25","doi-asserted-by":"publisher","first-page":"1281","DOI":"10.1080\/0305215X.2018.1519558","article-title":"Permutation flowshop scheduling with simple linear deterioration","volume":"51","author":"Sun","year":"2019","journal-title":"Eng. Optim."},{"key":"ref_26","doi-asserted-by":"publisher","first-page":"672","DOI":"10.1080\/00207721.2014.900137","article-title":"Two-machine flow shop scheduling integrated with preventive maintenance planning","volume":"47","author":"Wang","year":"2016","journal-title":"Int. J. Syst. Sci."},{"key":"ref_27","doi-asserted-by":"publisher","first-page":"3314","DOI":"10.1016\/j.cor.2005.12.007","article-title":"Considering scheduling and preventive maintenance in the flowshop sequencing problem","volume":"34","author":"Ruiz","year":"2007","journal-title":"Comput. Oper. Res."},{"key":"ref_28","doi-asserted-by":"publisher","first-page":"302","DOI":"10.1016\/j.omega.2005.07.004","article-title":"The permutation flow shop problem with blocking. A tabu search approach","volume":"35","author":"Grabowski","year":"2007","journal-title":"Omega"},{"key":"ref_29","doi-asserted-by":"publisher","first-page":"107250","DOI":"10.1016\/j.cie.2021.107250","article-title":"Aquila optimizer: A novel meta-heuristic optimization algorithm","volume":"157","author":"Abualigah","year":"2021","journal-title":"Comput. Ind. Eng."},{"key":"ref_30","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1287\/ijoc.6.2.154","article-title":"Genetic algorithms and random keys for sequencing and optimization","volume":"6","author":"Bean","year":"1994","journal-title":"ORSA J. Comput."},{"key":"ref_31","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/s12065-013-0102-2","article-title":"Swarm intelligence based algorithms: A critical analysis","volume":"7","author":"Yang","year":"2014","journal-title":"Evol. Intell."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wineberg, M., and Oppacher, F. (2003, January 12\u201316). The underlying similarity of diversity measures used in evolutionary computation. Proceedings of the Genetic and Evolutionary Computation Conference, Chicago, IL, USA.","DOI":"10.1007\/3-540-45110-2_21"},{"key":"ref_33","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1016\/0377-2217(93)90182-M","article-title":"Benchmarks for basic scheduling problems","volume":"64","author":"Taillard","year":"1993","journal-title":"Eur. J. Oper. Res."},{"key":"ref_34","doi-asserted-by":"publisher","first-page":"666","DOI":"10.1016\/j.ejor.2014.07.033","article-title":"New hard benchmark for flowshop scheduling problems minimising makespan","volume":"240","author":"Vallada","year":"2017","journal-title":"Eur. J. Oper. Res."},{"key":"ref_35","doi-asserted-by":"publisher","first-page":"106778","DOI":"10.1016\/j.cie.2020.106778","article-title":"A Self-Learning Genetic Algorithm based on Reinforcement Learning for Flexible Job-shop Scheduling Problem","volume":"149","author":"Chen","year":"2020","journal-title":"Comput. Ind. Eng."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Long, X., Zhang, J., and Zhou, K. (2022). Dynamic Self-Learning Artificial Bee Colony Optimization Algorithm for Flexible Job-Shop Scheduling Problem with Job Insertion. Processes, 10.","DOI":"10.3390\/pr10030571"}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/15\/8\/1600\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:36:41Z","timestamp":1760128601000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/15\/8\/1600"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,18]]},"references-count":36,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["sym15081600"],"URL":"https:\/\/doi.org\/10.3390\/sym15081600","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,18]]}}}