{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:06:16Z","timestamp":1787029576075,"version":"3.56.0"},"reference-count":58,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2023,9,12]],"date-time":"2023-09-12T00:00:00Z","timestamp":1694476800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The National Key Research and Development Program of China","award":["2022YFB3206602"],"award-info":[{"award-number":["2022YFB3206602"]}]},{"name":"The National Key Research and Development Program of China","award":["2022M720433"],"award-info":[{"award-number":["2022M720433"]}]},{"name":"China Postdoctoral Science Foundation Funded Project","award":["2022YFB3206602"],"award-info":[{"award-number":["2022YFB3206602"]}]},{"name":"China Postdoctoral Science Foundation Funded Project","award":["2022M720433"],"award-info":[{"award-number":["2022M720433"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The uncertain delay characteristic of actuators is a critical factor that affects the control effectiveness of the active suspension system. Therefore, it is crucial to develop a control algorithm that takes into account this uncertain delay in order to ensure stable control performance. This study presents a novel active suspension control algorithm based on deep reinforcement learning (DRL) that specifically addresses the issue of uncertain delay. In this approach, a twin-delayed deep deterministic policy gradient (TD3) algorithm with system delay is employed to obtain the optimal control policy by iteratively solving the dynamic model of the active suspension system, considering the delay. Furthermore, three different operating conditions were designed for simulation to evaluate the control performance: deterministic delay, semi-regular delay, and uncertain delay. The experimental results demonstrate that the proposed algorithm achieves excellent control performance under various operating conditions. Compared to passive suspension, the optimization of body vertical acceleration is improved by more than 30%, and the proposed algorithm effectively mitigates body vibration in the low frequency range. It consistently maintains a more than 30% improvement in ride comfort optimization even under the most severe operating conditions and at different speeds, demonstrating the algorithm\u2019s potential for practical application.<\/jats:p>","DOI":"10.3390\/s23187827","type":"journal-article","created":{"date-parts":[[2023,9,12]],"date-time":"2023-09-12T21:41:12Z","timestamp":1694554872000},"page":"7827","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Research on Deep Reinforcement Learning Control Algorithm for Active Suspension Considering Uncertain Time Delay"],"prefix":"10.3390","volume":"23","author":[{"given":"Yang","family":"Wang","sequence":"first","affiliation":[{"name":"School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100811, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cheng","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Automotive Simulation and Control, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shijie","family":"Zhao","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Automotive Simulation and Control, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Konghui","family":"Guo","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Automotive Simulation and Control, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,9,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"901","DOI":"10.4028\/www.scientific.net\/AMR.706-708.901","article-title":"PID Control Strategy of Vehicle Active Suspension Based on Considering Time-Delay and Stability","volume":"706\u2013708","author":"Yan","year":"2013","journal-title":"Adv. Mater. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/S0167-2789(03)00049-6","article-title":"Effects of Time Delayed Position Feedback on a van Der Pol\u2013Duffing Oscillator","volume":"180","author":"Xu","year":"2003","journal-title":"Phys. D Nonlinear Phenom."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"885","DOI":"10.1109\/TSMC.2016.2523935","article-title":"Topology Identification and Module\u2013Phase Synchronization of Neural Network with Time Delay","volume":"47","author":"Zhang","year":"2017","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1578","DOI":"10.1049\/iet-cta.2016.1139","article-title":"Neural Network-Based Output-Feedback Control for Stochastic High-Order Non-Linear Time-Delay Systems with Application to Robot System","volume":"11","author":"Min","year":"2017","journal-title":"IET Control. Theory Appl."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"6874","DOI":"10.1109\/TITS.2023.3263239","article-title":"The Upper Bounds of Cellular Vehicle-to-Vehicle Communication Latency for Platoon-Based Autonomous Driving","volume":"24","author":"Chen","year":"2023","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"04023066","DOI":"10.1061\/JTEPBS.TEENG-7478","article-title":"A Planning Control Strategy Based on Dynamic Safer Buffer to Avoid Traffic Collisions in an Emergency for CAVs at Nonsignalized Intersections","volume":"149","author":"Li","year":"2023","journal-title":"J. Transp. Eng. Part A Syst."},{"key":"ref_7","first-page":"1","article-title":"Car Following Models for Alleviating the Degeneration of CACC Function of CAVs in Weak Platoon Intensity","volume":"15","author":"Xu","year":"2023","journal-title":"Transp. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1959","DOI":"10.1007\/s12083-023-01487-9","article-title":"An Optimal Model for Enhancing Network Lifetime and Cluster Head Selection Using Hybrid Snake Whale Optimization","volume":"16","author":"Samiayya","year":"2023","journal-title":"Peer-to-Peer Netw. Appl."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2188406","DOI":"10.1080\/21642583.2023.2188406","article-title":"Hybrid AC\/DC Control Techniques with Improved Harmonic Conditions Using DBN Based Fuzzy Controller and Compensator Modules","volume":"11","author":"Reddy","year":"2023","journal-title":"Syst. Sci. Control Eng."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"11401","DOI":"10.1109\/TIE.2022.3231281","article-title":"Voltage Balancing of Series-Connected SiC Mosfets with Adaptive-Impedance Self-Powered Gate Drivers","volume":"70","author":"Wang","year":"2023","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_11","first-page":"1","article-title":"Comparison of Vehicle Suspension Dynamic Responses for Simplified and Advanced Adjustable Damper Models with Friction, Hysteresis and Actuation Delay for Different Comfort-Oriented Control Strategies","volume":"17","author":"Klockiewicz","year":"2023","journal-title":"Acta Mech. Autom."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ji, G., Li, S., Feng, G., and Wang, H. (2023). Enhanced Variable Universe Fuzzy Control of Vehicle Active Suspension Based on Adaptive Contracting-Expanding Factors. Int. J. Fuzzy Syst., 1\u201315.","DOI":"10.1007\/s40815-023-01549-3"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"983","DOI":"10.1002\/asjc.1419","article-title":"Approximation Optimal Vibration for Networked Nonlinear Vehicle Active Suspension with Actuator Time Delay","volume":"19","author":"Han","year":"2017","journal-title":"Asian J. Control."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1080\/10584587.2016.1165574","article-title":"Optimal Vibration Control of Nonlinear Systems with Multiple Time-Delays: An Application to Vehicle Suspension","volume":"170","author":"Lei","year":"2016","journal-title":"Integr. Ferroelectr."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/s12555-015-2009-4","article-title":"Constrained Model Predictive Control for Time-Varying Delay Systems: Application to an Active Car Suspension","volume":"14","author":"Bououden","year":"2016","journal-title":"Int. J. Control Autom. Syst."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1002\/stc.128","article-title":"Active Control of Structures Using Time Delayed Positive Feedback Proportional Control Designs","volume":"13","author":"Udwadia","year":"2006","journal-title":"Struct. Control. Health Monit."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2663","DOI":"10.1109\/TITS.2015.2414657","article-title":"Finite-Time Stabilization for Vehicle Active Suspension Systems with Hard Constraints","volume":"16","author":"Pan","year":"2015","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1423","DOI":"10.1061\/(ASCE)0733-9399(1992)118:7(1423)","article-title":"Aseismic Hybrid Control of Nonlinear and Hysteretic Structures I","volume":"118","author":"Yang","year":"1992","journal-title":"J. Eng. Mech."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1109\/TAC.1980.1102288","article-title":"Feedback Stabilization of Linear Systems with Delayed Control","volume":"25","author":"Kwon","year":"1980","journal-title":"IEEE Trans. Autom. Control."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1016\/j.jsv.2006.09.022","article-title":"H\u221e Control of Active Vehicle Suspensions with Actuator Time Delay","volume":"301","author":"Du","year":"2007","journal-title":"J. Sound Vib."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1109\/TIE.2013.2242418","article-title":"Output-Feedback-Based H\u0131nfty Control for Vehicle Suspension Systems with Control Delay","volume":"61","author":"Li","year":"2014","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"105584","DOI":"10.1016\/j.conengprac.2023.105584","article-title":"Model Predictive Control of a Semi-Active Suspension with a Shift Delay Compensation Using Preview Road Information","volume":"137","author":"Kim","year":"2023","journal-title":"Control Eng. Pract."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00207179.2023.2284254","article-title":"Experimental Research on Vehicle Active Suspension Based on Time-Delay Control","volume":"96","author":"Wu","year":"2023","journal-title":"Int. J. Control"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, G., Huang, Q., Hu, G., Ding, R., Zhu, W., and Zeng, L. (2023). Semi-Active Fuzzy Cooperative Control of Vehicle Suspension with a Magnetorheological Damper. J. Intell. Mater. Syst. Struct., 1045389X231157353.","DOI":"10.1177\/1045389X231157353"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1080\/21642583.2021.1949403","article-title":"Adaptive Control for the Nonlinear Suspension Systems with Stochastic Disturbances and Unknown Time Delay","volume":"10","author":"Wang","year":"2022","journal-title":"Syst. Sci. Control Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3255","DOI":"10.1109\/TSMC.2022.3224739","article-title":"A New Optimization Control Policy for Fuzzy Vehicle Suspension Systems Under Membership Functions Online Learning","volume":"53","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1608","DOI":"10.1002\/acs.3590","article-title":"Robust Fuzzy Fault Tolerant Control for Nonlinear Active Suspension Systems via Adaptive Hybrid Triggered Scheme","volume":"37","author":"Xie","year":"2023","journal-title":"Int. J. Adapt. Control Signal Process."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"6052","DOI":"10.1002\/rnc.6681","article-title":"State Observer-Based Predictive Proportional-Integral Tracking Control for Fuzzy Input Time-Delay Systems","volume":"33","author":"Sakthivel","year":"2023","journal-title":"Int. J. Robust Nonlinear Control"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1080\/00051144.2022.2059205","article-title":"A Novel Robust Finite Time Control Approach for a Nonlinear Disturbed Quarter-Vehicle Suspension System with Time Delay Actuation","volume":"63","author":"Gu","year":"2022","journal-title":"Automatika"},{"key":"ref_30","first-page":"335","article-title":"Multi-Objective H-2\/H-8 Control of Uncertain Active Suspension Systems with Interval Time-Varying Delay","volume":"237","author":"Ma","year":"2023","journal-title":"Proc. Inst. Mech. Eng. Part I J. Syst. Control Eng."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"21068","DOI":"10.1109\/ACCESS.2023.3250643","article-title":"Sampled-Data L-2 - L-8 Filter-Based Fuzzy Control for Active Suspensions","volume":"11","author":"Lee","year":"2023","journal-title":"IEEE Access"},{"key":"ref_32","unstructured":"Ma, G., Wang, Z., Yuan, Z., Wang, X., Yuan, B., and Tao, D. (2022). A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"3060","DOI":"10.1177\/09544070211063081","article-title":"Driver-like Decision-Making Method for Vehicle Longitudinal Autonomous Driving Based on Deep Reinforcement Learning","volume":"236","author":"Gao","year":"2022","journal-title":"Proc. Inst. Mech. Eng. Part D J. Automob. Eng."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Fares, A., and Bani Younes, A. (2020). Online Reinforcement Learning-Based Control of an Active Suspension System Using the Actor Critic Approach. Appl. Sci., 10.","DOI":"10.3390\/app10228060"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"9978","DOI":"10.1109\/ACCESS.2020.2964116","article-title":"Semi-Active Suspension Control Based on Deep Reinforcement Learning","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Pang, H., Luo, J., Wang, M., and Wang, L. (2023). A Stability Guaranteed Nonfragile Fault-Tolerant Control Approach for Markov-Type Vehicle Active Suspension System Subject to Faults and Disturbances. J. Vib. Control, 10775463231160807.","DOI":"10.1177\/10775463231160807"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Kozek, M., Smoter, A., and Lalik, K. (2023). Neural-Assisted Synthesis of a Linear Quadratic Controller for Applications in Active Suspension Systems of Wheeled Vehicles. Energies, 16.","DOI":"10.3390\/en16041677"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"4021","DOI":"10.1109\/TSMC.2021.3089768","article-title":"Neural Network Adaptive Output-Feedback Optimal Control for Active Suspension Systems","volume":"52","author":"Li","year":"2022","journal-title":"IEEE Trans. Syst. Man Cybern Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1417","DOI":"10.1049\/cth2.12317","article-title":"A Reinforcement Learning Backstepping-Based Control Design for a Full Vehicle Active Macpherson Suspension System","volume":"16","author":"Lin","year":"2022","journal-title":"IET Control Theory Appl."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1109\/TIE.2022.3153805","article-title":"Suspension Control Strategies Using Switched Soft Actor-Critic Models for Real Roads","volume":"70","author":"Yong","year":"2023","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/TVT.2022.3207510","article-title":"Deep Reinforcement Learning of Semi-Active Suspension Controller for Vehicle Ride Comfort","volume":"72","author":"Lee","year":"2023","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1111\/mice.12934","article-title":"A Hierarchical Framework for Improving Ride Comfort of Autonomous Vehicles via Deep Reinforcement Learning with External Knowledge","volume":"38","author":"Du","year":"2022","journal-title":"Comput.-Aided Civ. Infrastruct. Eng."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Han, S.-Y., and Liang, T. (2022). Reinforcement-Learning-Based Vibration Control for a Vehicle Semi-Active Suspension System via the PPO Approach. Appl. Sci., 12.","DOI":"10.3390\/app12063078"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"16878132231180480","DOI":"10.1177\/16878132231180480","article-title":"A New Approach to Controlling an Active Suspension System Based on Reinforcement Learning","volume":"15","author":"Dridi","year":"2023","journal-title":"Adv. Mech. Eng."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1016\/j.sna.2006.03.015","article-title":"A Novel Hysteretic Model for Magnetorheological Fluid Dampers and Parameter Identification Using Particle Swarm Optimization","volume":"132","author":"Kwok","year":"2006","journal-title":"Sens. Actuators A Phys."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Krauze, P., and Kasprzyk, J. (2020). Driving Safety Improved with Control of Magnetorheological Dampers in Vehicle Suspension. Appl. Sci., 10.","DOI":"10.3390\/app10248892"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1115\/1.2745846","article-title":"Mixed Sky-Hook and ADD: Approaching the Filtering Limits of a Semi-Active Suspension","volume":"129","author":"Savaresi","year":"2006","journal-title":"J. Dyn. Syst. Meas. Control"},{"key":"ref_48","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2019). Continuous Control with Deep Reinforcement Learning. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-Level Control through Deep Reinforcement Learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_51","unstructured":"Fujimoto, S., Hoof, H., and Meger, D. (2018, January 3). Addressing Function Approximation Error in Actor-Critic Methods. Proceedings of the Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"4877","DOI":"10.1109\/TIE.2019.2926056","article-title":"Regionless Explicit Model Predictive Control of Active Suspension Systems with Preview","volume":"67","author":"Theunissen","year":"2020","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Liang, G., Zhao, T., and Wei, Y. (2021, January 29\u201331). DDPG Based Self-Learning Active and Model-Constrained Semi-Active Suspension Control. Proceedings of the 2021 5th CAA International Conference on Vehicular Control and Intelligence (CVCI), Tianjin, China.","DOI":"10.1109\/CVCI54083.2021.9661158"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Baek, S., Baek, J., Choi, J., and Han, S. (2022, January 8\u201310). A Reinforcement Learning-Based Adaptive Time-Delay Control and Its Application to Robot Manipulators. Proceedings of the 2022 American Control Conference (ACC), Atlanta, GA, USA.","DOI":"10.23919\/ACC53348.2022.9867835"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"4171","DOI":"10.1109\/TSMC.2018.2870724","article-title":"Reinforcement Learning Neural Network-Based Adaptive Control for State and Input Time-Delayed Wheeled Mobile Robots","volume":"50","author":"Li","year":"2020","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_56","first-page":"7844719","article-title":"Application Research of Time Delay System Control in Mobile Sensor Networks Based on Deep Reinforcement Learning","volume":"2022","author":"Zhu","year":"2022","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.neucom.2021.04.015","article-title":"Delay-Aware Model-Based Reinforcement Learning for Continuous Control","volume":"450","author":"Chen","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_58","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7827\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:49:28Z","timestamp":1760129368000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7827"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,12]]},"references-count":58,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["s23187827"],"URL":"https:\/\/doi.org\/10.3390\/s23187827","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,12]]}}}