{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T13:04:46Z","timestamp":1785416686806,"version":"3.56.0"},"reference-count":26,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T00:00:00Z","timestamp":1782518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China (NSFC), General Program","award":["52475559"],"award-info":[{"award-number":["52475559"]}]},{"award":["52475559"],"award-info":[{"award-number":["52475559"]}],"id":[{"id":"https:\/\/ror.org\/01h0zpd94","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>To address the challenge of maintenance decision-making for critical components in electro-hydraulic servo material fatigue testing machine, characterized by weak state observability and difficulty in degradation prediction, a multi-component joint maintenance decision-making method based on multi-head deep reinforcement learning is proposed. Considering the heterogeneity of the degradation mechanisms and observation methods for the four components\u2014bearing beam, fixture, main machine sensors, and hydraulic oil tank\u2014a continuous-discrete hybrid state Markov decision process (HS-MDP) is constructed. To account for differences in maintenance strategies across components, a differentiated discrete action space for each component is designed, and engineering feasibility constraints are explicitly integrated into the policy through action masking. A data-quality loss term, determined by the degradation level of the sensors, is introduced into the reward function to align the optimization objective with the metrological properties of the fatigue testing machine. Based on the Branching Dueling DQN framework, a Q-network structure is constructed, incorporating a shared encoder, an inter-component attention mechanism, and multi-head branched outputs. Taking a 100 kN electro-hydraulic servo fatigue testing machine as a case study, comparisons with baseline strategies such as periodic maintenance, threshold-based condition-based maintenance (CBM), independent DQN, and PPO indicate that the proposed method reduces the average annual total cost by 60.3% compared to periodic maintenance and by 42.6% compared to threshold-based CBM. The number of failures decreases from 9.8 times\/year to 1.4 times\/year, while data efficiency increases from 82.1% to 96.2%. Ablation experiments and robustness tests further verify the critical contributions of three key design elements: action masking, inter-component attention, and data-quality loss.<\/jats:p>","DOI":"10.3390\/s26134087","type":"journal-article","created":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T08:45:20Z","timestamp":1782722720000},"page":"4087","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Multi-Component Joint Maintenance Decision for Electro-Hydraulic Servo Fatigue Testing Machine Based on Multi-Head Deep Reinforcement Learning"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8774-7398","authenticated-orcid":false,"given":"Peng","family":"Liu","sequence":"first","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guotai","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jialu","family":"Xi","sequence":"additional","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaqi","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,6,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1016\/j.ejor.2019.09.047","article-title":"A review on maintenance optimization","volume":"285","author":"Scarf","year":"2020","journal-title":"Eur. J. Oper. Res."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Nicolai, R.P., and Dekker, R. (2008). Optimal Maintenance of Multi-component Systems: A Review. Complex System Maintenance Handbook, Springer.","DOI":"10.1007\/978-1-84800-011-7_11"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1016\/0143-8174(86)90099-5","article-title":"A survey of maintenance and replacement models for maintainability and reliability of multi-item systems","volume":"16","author":"Thomas","year":"1986","journal-title":"Reliab. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2815","DOI":"10.1016\/j.apm.2014.10.034","article-title":"Multi-item fuzzy-stochastic supply chain models for long-term contracts with a profit sharing scheme","volume":"39","author":"Saha","year":"2015","journal-title":"Appl. Math. Model."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"949","DOI":"10.1016\/j.ress.2011.03.014","article-title":"Reliability modeling of degradation of products with multiple performance characteristics based on gamma processes","volume":"96","author":"Pan","year":"2011","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/j.ejor.2016.12.015","article-title":"Decomposing the Luenberger\u2013Hicks\u2013Moorsteen Total Factor Productivity indicator: An application to U.S. agriculture","volume":"260","author":"Ang","year":"2017","journal-title":"Eur. J. Oper. Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ejor.2010.11.018","article-title":"Remaining useful life estimation\u2014A review on the statistical data driven approaches","volume":"213","author":"Si","year":"2011","journal-title":"Eur. J. Oper. Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/S0951-8320(01)00148-X","article-title":"A condition-based maintenance policy for stochastically deteriorating systems","volume":"76","author":"Grall","year":"2002","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.ress.2004.04.013","article-title":"A condition-based maintenance policy with non-periodic inspections for a two-unit series system","volume":"87","author":"Castanier","year":"2005","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"107094","DOI":"10.1016\/j.ress.2020.107094","article-title":"Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks","volume":"203","author":"Zhang","year":"2020","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"15549","DOI":"10.1007\/s00521-023-08542-9","article-title":"A deep reinforcement learning approach for maintenance planning of multi-component systems with complex structure","volume":"35","author":"Chen","year":"2023","journal-title":"Neural Comput. Applic."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"123144","DOI":"10.1016\/j.eswa.2024.123144","article-title":"Multi-agent deep reinforcement learning-based maintenance optimization for multi-dependent component systems","volume":"245","author":"Do","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_14","first-page":"1995","article-title":"Dueling network architectures for deep reinforcement learning","volume":"Volume 48","author":"Wang","year":"2016","journal-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning"},{"key":"ref_15","first-page":"4131","article-title":"Action Branching Architectures for Deep Reinforcement Learning","volume":"32","author":"Tavakoli","year":"2018","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_16","first-page":"21646","article-title":"Pure-Past Action Masking","volume":"38","author":"Varricchione","year":"2024","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, Z., Li, X., Sun, L., Zhang, H., Liu, H., and Wang, J. (2024). Learning State-Specific Action Masks for Reinforcement Learning. Algorithms, 17.","DOI":"10.3390\/a17020060"},{"key":"ref_18","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bai, Y., Du, J., Li, H., Bao, X., Li, L., Zhang, C., Yan, J., Wang, R., and Xu, Y. (2025). An Adaptive Framework for Remaining Useful Life Prediction Integrating Attention Mechanism and Deep Reinforcement Learning. Sensors, 25.","DOI":"10.3390\/s25206354"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"106854","DOI":"10.1016\/j.ress.2020.106854","article-title":"Dynamic opportunistic maintenance planning for multi-component redundant systems with various types of opportunities","volume":"198","author":"Vu","year":"2020","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1016\/j.ijpe.2017.10.030","article-title":"Condition-based maintenance for systems with economic dependence and load sharing","volume":"195","author":"Teunter","year":"2018","journal-title":"Int. J. Prod. Econ."},{"key":"ref_22","unstructured":"(2012). Information and Documentation: International Standard Identifier for Libraries and Related Organizations (ISIL) (Standard No. ISO 23788:2012)."},{"key":"ref_23","unstructured":"(2018). Metallic Materials\u2014Calibration and Verification of Static Uniaxial Testing Machines\u2014Part 1: Tension\/Compression Testing Machines\u2014Calibration and Verification of the Force-Measuring System (Standard No. ISO 7500-1:2018)."},{"key":"ref_24","unstructured":"(1964). Cleanliness Requirements of Parts Used in Hydraulic Systems (Standard No. NAS 1638)."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Phan, V.D., and Ahn, K.K. (2022). Optimized-Based Fault-Tolerant Control of an Electro-Hydraulic System with Disturbance Rejection. Appl. Sci., 12.","DOI":"10.3390\/app12189197"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5615","DOI":"10.1109\/TMECH.2022.3186390","article-title":"Extended-State-Observer-Based Adaptive Prescribed Performance Control for Hydraulic Systems With Full-State Constraints","volume":"27","author":"Xu","year":"2022","journal-title":"IEEE\/ASME Trans. Mechatron."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/13\/4087\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T04:22:43Z","timestamp":1783570963000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/13\/4087"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,27]]},"references-count":26,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["s26134087"],"URL":"https:\/\/doi.org\/10.3390\/s26134087","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,27]]}}}