{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,29]],"date-time":"2026-08-29T17:21:58Z","timestamp":1788024118717,"version":"build-2784847793"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"30","license":[{"start":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T00:00:00Z","timestamp":1736467200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T00:00:00Z","timestamp":1736467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["NWA.1431.20.004"],"award-info":[{"award-number":["NWA.1431.20.004"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Onderzoeksprogramma Artifici\u00eble Intelligentie (AI) Vlaanderen","award":["NWA.1431.20.002"],"award-info":[{"award-number":["NWA.1431.20.002"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2025,10]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In this paper, we introduce multi-objective deep centralized multi-agent actor-critic (MO-DCMAC), a multi-objective reinforcement learning method for infrastructural maintenance optimization, an area traditionally dominated by single-objective reinforcement learning (RL) approaches. Previous single-objective RL methods combine multiple objectives, such as probability of collapse and cost, into a singular reward signal through reward-shaping. In contrast, MO-DCMAC can optimize a policy for multiple objectives directly, even when the utility function is nonlinear. We evaluated MO-DCMAC using two utility functions, which use probability of collapse and cost as input. The first utility function is the threshold utility, in which MO-DCMAC should minimize cost so that the probability of collapse is never above the threshold. The second is based on the failure mode, effects, and criticality analysis methodology used by asset managers to assess maintenance plans. We evaluated MO-DCMAC, with both utility functions, in multiple maintenance environments, including ones based on a case study of the historical quay walls of Amsterdam. The performance of MO-DCMAC was compared against multiple rule-based policies based on heuristics currently used for constructing maintenance plans. Our results demonstrate that MO-DCMAC outperforms traditional rule-based policies across various environments and utility functions.<\/jats:p>","DOI":"10.1007\/s00521-024-10954-0","type":"journal-article","created":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T12:38:00Z","timestamp":1736512680000},"page":"24719-24742","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Deep multi-objective reinforcement learning for utility-based infrastructural maintenance optimization"],"prefix":"10.1007","volume":"37","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-1966-6907","authenticated-orcid":false,"given":"Jesse","family":"van Remmerden","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maurice","family":"Kenter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diederik M.","family":"Roijers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charalampos","family":"Andriotis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yingqian","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zaharah","family":"Bukhsh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,1,10]]},"reference":[{"key":"10954_CR1","doi-asserted-by":"publisher","unstructured":"Alegre LN, Bazzan AL, Roijers DM, et\u00a0al (2023) Sample-efficient multi-objective learning via generalized policy improvement prioritization. In: Proceedings of the international joint conference on autonomous agents and multiagent systems, AAMAS, vol 2023-May. International foundation for autonomous agents and multiagent systems (IFAAMAS), pp 2003\u20132012. https:\/\/doi.org\/10.5555\/3545946.3598872","DOI":"10.5555\/3545946.3598872"},{"key":"10954_CR2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ymssp.2023.110276","author":"A Aloisio","year":"2023","unstructured":"Aloisio A, Contento A, Alaggio R et al (2023) Physics-based models, surrogate models and experimental assessment of the vehicle-bridge interaction in braking conditions. Mech Syst Signal Process. https:\/\/doi.org\/10.1016\/j.ymssp.2023.110276","journal-title":"Mech Syst Signal Process"},{"key":"10954_CR3","unstructured":"Andriotis. CP, Metwally. Z (2023) Structural integrity management via hierarchical resource allocation and continuous-control reinforcement learning. In: 14th international conference on applications of statistics and probability in civil engineering, ICASP14, 2262\/103609"},{"key":"10954_CR4","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2019.04.036","author":"CP Andriotis","year":"2019","unstructured":"Andriotis CP, Papakonstantinou KG (2019) Managing engineering systems with large state and action spaces through deep reinforcement learning. Reliab Eng Syst Saf. https:\/\/doi.org\/10.1016\/j.ress.2019.04.036","journal-title":"Reliab Eng Syst Saf"},{"key":"10954_CR5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2021.107551","author":"CP Andriotis","year":"2021","unstructured":"Andriotis CP, Papakonstantinou KG (2021) Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints. Reliab Eng Syst Saf. https:\/\/doi.org\/10.1016\/j.ress.2021.107551","journal-title":"Reliab Eng Syst Saf"},{"key":"10954_CR6","unstructured":"Bellemare MG, Dabney W, Munos R (2017) A distributional perspective on reinforcement learning. In: 34th international conference on machine learning, ICML 2017"},{"key":"10954_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/0951-8320(95)00068-D","author":"JB Bowles","year":"1995","unstructured":"Bowles JB, Pel\u2019aez CE (1995) Fuzzy logic prioritization of failures in a system failure mode, effects and criticality analysis. Reliab Eng Syst Saf. https:\/\/doi.org\/10.1016\/0951-8320(95)00068-D","journal-title":"Reliab Eng Syst Saf"},{"key":"10954_CR8","doi-asserted-by":"publisher","DOI":"10.1186\/s12544-019-0388-y","author":"ZA Bukhsh","year":"2020","unstructured":"Bukhsh ZA, Stipanovic I, Doree AG (2020) Multi-year maintenance planning framework using multi-attribute utility theory and genetic algorithms. Eur Transp Res Rev. https:\/\/doi.org\/10.1186\/s12544-019-0388-y","journal-title":"Eur Transp Res Rev"},{"key":"10954_CR9","doi-asserted-by":"crossref","unstructured":"Chansombat S, Pongcharoen P (2018) Hicks C (2019) A mixed-integer linear programming model for integrated production and preventive maintenance scheduling in the capital goods industry. Int J Prod Res 10(1080\/00207543):1459923","DOI":"10.1080\/00207543.2018.1459923"},{"key":"10954_CR10","doi-asserted-by":"publisher","unstructured":"Correia MG, de Oliveira e. Bonates T, de Athayde Prata B, et al (2022) An integer linear programming approach for pavement maintenance and rehabilitation optimization. Int J Pavement Eng. https:\/\/doi.org\/10.1080\/10298436.2020.1869736","DOI":"10.1080\/10298436.2020.1869736"},{"key":"10954_CR11","doi-asserted-by":"publisher","first-page":"04014018","DOI":"10.1061\/(asce)be.1943-5592.0000586","volume":"19","author":"Y Dong","year":"2014","unstructured":"Dong Y, Frangopol DM, Saydam D (2014) Pre-earthquake multi-objective probabilistic retrofit optimization of bridge networks based on sustainability. J Bridge Eng 19:04014018. https:\/\/doi.org\/10.1061\/(asce)be.1943-5592.0000586","journal-title":"J Bridge Eng"},{"key":"10954_CR12","doi-asserted-by":"publisher","unstructured":"Dornheim J, Link N (2018) Multiobjective reinforcement learning for reconfigurable adaptive optimal control of manufacturing processes. In: 2018 13th international symposium on electronics and telecommunications, ISETC 2018\u2014conference proceedings. https:\/\/doi.org\/10.1109\/ISETC.2018.8583854","DOI":"10.1109\/ISETC.2018.8583854"},{"key":"10954_CR13","doi-asserted-by":"publisher","unstructured":"Godsill S (2019) Particle filtering: The first 25 years and beyond. In: ICASSP, IEEE international conference on acoustics, speech and signal processing\u2014proceedings. https:\/\/doi.org\/10.1109\/ICASSP.2019.8683411","DOI":"10.1109\/ICASSP.2019.8683411"},{"key":"10954_CR14","doi-asserted-by":"publisher","unstructured":"Gu S, Yang L, Du Y, et\u00a0al (2023) A review of safe reinforcement learning: methods, theory and applications. https:\/\/doi.org\/10.48550\/arXiv.2205.10330","DOI":"10.48550\/arXiv.2205.10330"},{"key":"10954_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclepro.2021.130103","volume":"334","author":"J Guan","year":"2022","unstructured":"Guan J, Yang X, You L et al (2022) Multi-objective optimization for sustainable road network maintenance under traffic equilibrium: incorporating costs and environmental impacts. J Clean Prod 334:130103","journal-title":"J Clean Prod"},{"key":"10954_CR16","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2023.109214","volume":"235","author":"Z Hamida","year":"2023","unstructured":"Hamida Z, Goulet JA (2023) Hierarchical reinforcement learning for transportation infrastructure maintenance planning. Reliab Eng Syst Saf 235:109214. https:\/\/doi.org\/10.1016\/j.ress.2023.109214","journal-title":"Reliab Eng Syst Saf"},{"key":"10954_CR17","unstructured":"Hausknecht M, Stone P (2015) Deep recurrent q-learning for partially observable mdps. In: AAAI fall symposium\u2014technical report"},{"key":"10954_CR18","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1007\/s10458-022-09552-y","volume":"36","author":"CF Hayes","year":"2022","unstructured":"Hayes CF, R\u0103dulescu R, Bargiacchi E et al (2022) A practical guide to multi-objective reinforcement learning and planning. Auton Agents Multi-Agent Syst 36:26. https:\/\/doi.org\/10.1007\/s10458-022-09552-y","journal-title":"Auton Agents Multi-Agent Syst"},{"key":"10954_CR19","doi-asserted-by":"publisher","unstructured":"Hayes CF, Verstraeten T, Roijers DM, et\u00a0al (2022) Multi-objective coordination graphs for the expected scalarised returns with generative flow models. https:\/\/doi.org\/10.48550\/arXiv.2207.00368","DOI":"10.48550\/arXiv.2207.00368"},{"key":"10954_CR20","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.ress.2016.10.002","volume":"158","author":"B de Jonge","year":"2017","unstructured":"de Jonge B, Teunter R, Tinga T (2017) The influence of practical factors on the benefits of condition-based maintenance over time-based maintenance. Reliab Eng Syst Saf 158:21\u201330. https:\/\/doi.org\/10.1016\/j.ress.2016.10.002","journal-title":"Reliab Eng Syst Saf"},{"key":"10954_CR21","doi-asserted-by":"publisher","unstructured":"Kerkkamp D, Bukhsh ZA, Zhang Y, et\u00a0al (2022) Grouping of maintenance actions with deep reinforcement learning and graph convolutional networks. In: International conference on agents and artificial intelligence. https:\/\/doi.org\/10.5220\/0010907500003116","DOI":"10.5220\/0010907500003116"},{"key":"10954_CR22","doi-asserted-by":"publisher","first-page":"727","DOI":"10.1007\/s11831-022-09815-7","volume":"30","author":"M Khodarahmi","year":"2023","unstructured":"Khodarahmi M, Maihami V (2023) A review on kalman filter models. Arch Comput Methods Eng 30:727. https:\/\/doi.org\/10.1007\/s11831-022-09815-7","journal-title":"Arch Comput Methods Eng"},{"key":"10954_CR23","unstructured":"Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th international conference on learning representations, ICLR 2017\u2014conference track proceedings"},{"key":"10954_CR24","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1680\/jfoen.21.00018","volume":"175","author":"M Korff","year":"2022","unstructured":"Korff M, Hemel MJ, Peters DJ (2022) Collapse of the grimburgwal, a historic quay in amsterdam, the netherlands. Proc Inst Civil Eng Forensic Eng 175:96. https:\/\/doi.org\/10.1680\/jfoen.21.00018","journal-title":"Proc Inst Civil Eng Forensic Eng"},{"key":"10954_CR25","unstructured":"Lathourakis C, Andriotis CP, Cicirello A (2023) Inference and maintenance planning of monitored structures through markov chain monte carlo and deep reinforcement learning. In: 14th international conference on applications of statistics and probability in civil engineering, ICASP14, 2262\/103339"},{"key":"10954_CR26","doi-asserted-by":"publisher","first-page":"769","DOI":"10.1080\/01441647.2011.596581","volume":"31","author":"D Lavee","year":"2011","unstructured":"Lavee D, Beniad G, Solomon C (2011) The effect of investment in transportation infrastructure on the debt-to-gdp ratio. Transp Rev 31:769. https:\/\/doi.org\/10.1080\/01441647.2011.596581","journal-title":"Transp Rev"},{"key":"10954_CR27","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1016\/j.infsof.2016.04.002","volume":"75","author":"LEG Martins","year":"2016","unstructured":"Martins LEG, Gorschek T (2016) Requirements engineering for safety-critical systems: a systematic literature review. Inf Softw Technol 75:71\u201389. https:\/\/doi.org\/10.1016\/j.infsof.2016.04.002","journal-title":"Inf Softw Technol"},{"key":"10954_CR28","doi-asserted-by":"publisher","unstructured":"M\u00e9ndez-Hern\u00e1ndez BM, Rodr\u00edguez-Bazan ED, Martinez-Jimenez Y, et\u00a0al (2019) A multi-objective reinforcement learning algorithm for jssp. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). https:\/\/doi.org\/10.1007\/978-3-030-30487-4_44","DOI":"10.1007\/978-3-030-30487-4_44"},{"key":"10954_CR29","doi-asserted-by":"publisher","first-page":"1403","DOI":"10.3390\/rs11121403","volume":"11","author":"P Milillo","year":"2019","unstructured":"Milillo P, Giardina G, Perissin D et al (2019) Pre-collapse space geodetic observations of critical infrastructure: the morandi bridge, genoa, italy. Remote Sens 11:1403. https:\/\/doi.org\/10.3390\/rs11121403","journal-title":"Remote Sens"},{"key":"10954_CR30","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih V, Kavukcuoglu K, Silver D et al (2015) Human-level control through deep reinforcement learning. Nature 518:529. https:\/\/doi.org\/10.1038\/nature14236","journal-title":"Nature"},{"key":"10954_CR31","unstructured":"Mnih V, Badia AP, Mirza L, et\u00a0al (2016) Asynchronous methods for deep reinforcement learning. In: 33rd international conference on machine learning, ICML"},{"key":"10954_CR32","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2023.109144","volume":"235","author":"PG Morato","year":"2023","unstructured":"Morato PG, Andriotis CP, Papakonstantinou KG et al (2023) Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through bayesian networks and deep reinforcement learning. Reliab Eng Syst Saf 235:109144. https:\/\/doi.org\/10.1016\/j.ress.2023.109144","journal-title":"Reliab Eng Syst Saf"},{"key":"10954_CR33","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1007\/s10458-023-09604-x","volume":"37","author":"M Reymond","year":"2023","unstructured":"Reymond M, Hayes CF, Steckelmacher D et al (2023) Actor-critic multi-objective reinforcement learning for non-linear utility functions. Auton Agents Multi-Agent Syst 37:23. https:\/\/doi.org\/10.1007\/s10458-023-09604-x","journal-title":"Auton Agents Multi-Agent Syst"},{"key":"10954_CR34","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1613\/jair.3987","volume":"48","author":"DM Roijers","year":"2013","unstructured":"Roijers DM, Vamplew P, Whiteson S et al (2013) A survey of multi-objective sequential decision-making. J Artif Intell Res 48:67. https:\/\/doi.org\/10.1613\/jair.3987","journal-title":"J Artif Intell Res"},{"key":"10954_CR35","unstructured":"Roijers DM, Whiteson S, Oliehoek FA (2015) Point-based planning for multi-objective pomdps. In: IJCAI international joint conference on artificial intelligence"},{"key":"10954_CR36","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-019-09433-x","author":"R R\u0103dulescu","year":"2020","unstructured":"R\u0103dulescu R, Mannion P, Roijers DM et al (2020) Multi-objective multi-agent decision making: a utility-based analysis and survey. Auton Agents Multi-Agent Syst. https:\/\/doi.org\/10.1007\/s10458-019-09433-x","journal-title":"Auton Agents Multi-Agent Syst"},{"key":"10954_CR37","unstructured":"Silver D, Veness J (2010) Monte-carlo planning in large pomdps. In: advances in neural information processing systems 23: 24th annual conference on neural information processing systems 2010, NIPS"},{"key":"10954_CR38","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2020.106600","volume":"147","author":"E Skordilis","year":"2020","unstructured":"Skordilis E, Moghaddass R (2020) A deep reinforcement learning approach for real-time sensor-driven decision making and predictive analytics. Comput Ind Eng 147:106600. https:\/\/doi.org\/10.1016\/j.cie.2020.106600","journal-title":"Comput Ind Eng"},{"key":"10954_CR39","doi-asserted-by":"publisher","unstructured":"Soh H, Demiris Y (2011) Evolving policies for multi-reward partially observable markov decision processes (mr-pomdps). In: Genetic and evolutionary computation conference, GECCO\u201911. https:\/\/doi.org\/10.1145\/2001576.2001674","DOI":"10.1145\/2001576.2001674"},{"key":"10954_CR40","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/S0925-5273(00)00067-0","volume":"70","author":"L Swanson","year":"2001","unstructured":"Swanson L (2001) Linking maintenance strategies to performance. Int J Prod Econ 70:237. https:\/\/doi.org\/10.1016\/S0925-5273(00)00067-0","journal-title":"Int J Prod Econ"},{"key":"10954_CR41","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1023\/A:1022676722315","volume":"8","author":"CJ Watkins","year":"1992","unstructured":"Watkins CJ, Dayan P (1992) Technical note: Q-learning. Mach Learn 8:279. https:\/\/doi.org\/10.1023\/A:1022676722315","journal-title":"Mach Learn"},{"key":"10954_CR42","doi-asserted-by":"publisher","first-page":"1484","DOI":"10.1016\/j.neunet.2009.05.011","volume":"22","author":"P Wawrzy\u0144ski","year":"2009","unstructured":"Wawrzy\u0144ski P (2009) Real-time reinforcement learning by sequential actor-critics and experience replay. Neural Netw 22:1484. https:\/\/doi.org\/10.1016\/j.neunet.2009.05.011","journal-title":"Neural Netw"},{"key":"10954_CR43","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1109\/TSTE.2020.2986586","volume":"12","author":"L Yang","year":"2021","unstructured":"Yang L, Li G, Zhang Z et al (2021) Operations & maintenance optimization of wind turbines integrating wind and aging information. IEEE Trans Sustain Energy 12:211. https:\/\/doi.org\/10.1109\/TSTE.2020.2986586","journal-title":"IEEE Trans Sustain Energy"},{"key":"10954_CR44","unstructured":"Yang R, Sun X, Narasimhan K (2019) A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In: Advances in neural information processing systems"},{"key":"10954_CR45","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2020.106889","volume":"150","author":"T Zonta","year":"2020","unstructured":"Zonta T, da Costa CA, da Rosa Righi R et al (2020) Predictive maintenance in the industry 4.0: s systematic literature review. Comput Ind Eng 150:106889. https:\/\/doi.org\/10.1016\/j.cie.2020.106889","journal-title":"Comput Ind Eng"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-024-10954-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-024-10954-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-024-10954-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T12:37:17Z","timestamp":1760013437000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-024-10954-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,10]]},"references-count":45,"journal-issue":{"issue":"30","published-print":{"date-parts":[[2025,10]]}},"alternative-id":["10954"],"URL":"https:\/\/doi.org\/10.1007\/s00521-024-10954-0","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,10]]},"assertion":[{"value":"1 April 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 December 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 January 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}