{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T17:55:23Z","timestamp":1791309323815,"version":"4.3.1"},"reference-count":24,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T00:00:00Z","timestamp":1791244800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>Building on recent insights that augmenting reinforcement\u2010learning policies with disturbance estimates improves robustness and sim-to-real transfer, this paper proposes a disturbance-aware actor\u2013critic RL framework for high\u2010precision robotic manipulators. We derive the dynamics of manipulators ranging from two to six degrees of freedom and design a nonlinear disturbance observer that provides real-time estimates of lumped uncertainties. Unlike conventional feedforward-only designs, the proposed DACRL framework deeply integrates DOB information into the actor-critic\u2019s learning state, cost function, and update laws to explicitly compensate for unknown dynamics and disturbances. A Lyapunov-based analysis proves that tracking errors, observer errors and neural-network weight errors remain uniformly ultimately bounded. Extensive simulations show that the proposed controller achieves sub-degree tracking errors across multiple DOF cases, with the 2-DOF example achieving steady-state errors below \u00b10.02 rad (Joint 1) and \u00b10.05 rad (Joint 2) and delivering faster convergence and smoother torques than conventional RL and DOB baselines. The disturbance-aware controller generalizes across trajectories and payloads, offering improved robustness while retaining learning flexibility. While the current study focuses on performance and robustness, future work could explore the integration of disturbance-observer-based control barrier functions to formally address safety constraints during the learning process. Simulation results suggest that the disturbance-aware controller can improve robustness while retaining learning flexibility. Real-platform validation remains an important direction for future work.<\/jats:p>","DOI":"10.1371\/journal.pone.0354740","type":"journal-article","created":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T17:38:08Z","timestamp":1791308288000},"page":"e0354740","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Robust disturbance-aware actor\u2013critic reinforcement learning for multi-DOF robotic manipulators"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-6012-4846","authenticated-orcid":true,"given":"Viet Ngu","family":"Nguyen","sequence":"first","affiliation":[{"name":"Hung Yen University of Technology and Education","place":["Vietnam"],"department":["Faculty of Electrical and Electronic Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5147-5370","authenticated-orcid":true,"given":"Thi Minh Tam","family":"Le","sequence":"additional","affiliation":[{"name":"Hung Yen University of Technology and Education","place":["Vietnam"],"department":["Faculty of Electrical and Electronic Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3344-1593","authenticated-orcid":true,"given":"Duc Hung","family":"Pham","sequence":"additional","affiliation":[{"name":"Hung Yen University of Technology and Education","place":["Vietnam"],"department":["Faculty of Electrical and Electronic Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huu Cuong","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Hung Yen University of Technology and Education","place":["Vietnam"],"department":["Faculty of Electrical and Electronic Engineering"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,10,6]]},"reference":[{"key":"pone.0354740.ref001","author":"C Tang","year":"2024"},{"key":"pone.0354740.ref002","doi-asserted-by":"crossref","first-page":"120495","DOI":"10.1016\/j.eswa.2023.120495","article-title":"Reinforcement learning algorithms: A brief survey","volume":"231","author":"AK Shakya","year":"2023","journal-title":"Expert Systems with Applications"},{"key":"pone.0354740.ref003","author":"SE Li","year":"2023","journal-title":"Reinforcement Learning for Sequential Decision and Optimal Control. Singapore: Springer"},{"issue":"2","key":"pone.0354740.ref004","article-title":"A survey on model-based reinforcement learning","volume":"67","author":"F-M Luo","year":"2024","journal-title":"Sci China Inf Sci"},{"issue":"18","key":"pone.0354740.ref005","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1016\/j.ifacol.2024.09.056","article-title":"Model predictive control and reinforcement learning: A unified framework based on dynamic programming","volume":"58","author":"DP Bertsekas","year":"2024","journal-title":"IFAC-PapersOnLine"},{"issue":"2","key":"pone.0354740.ref006","doi-asserted-by":"crossref","DOI":"10.1007\/s10462-025-11439-9","article-title":"Reinforcement learning for single-agent to multi-agent systems: from basic theory to industrial application progress, a survey","volume":"59","author":"D Zhang","year":"2025","journal-title":"Artif Intell Rev"},{"issue":"8093","key":"pone.0354740.ref007","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1038\/s41586-025-09761-x","article-title":"Discovering state-of-the-art reinforcement learning algorithms","volume":"648","author":"J Oh","year":"2025","journal-title":"Nature"},{"key":"pone.0354740.ref008","author":"M Guerrier","year":"2024"},{"key":"pone.0354740.ref009","unstructured":"Cheng Y, Zhao P, Hovakimyan N. Safe and efficient reinforcement learning using disturbance-observer-based control barrier functions. In: Proceedings of the Learning for Dynamics and Control (L4DC). 2023;1\u201312."},{"key":"pone.0354740.ref010","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1007\/s11431-022-2126-7","article-title":"A disturbance observer based intelligent control for nonstrict-feedback nonlinear systems","volume":"66","author":"H Ren","year":"2023","journal-title":"Science China Technological Sciences"},{"issue":"2","key":"pone.0354740.ref011","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1007\/s12555-024-0624-7","article-title":"An internal model disturbance observer based robust trajectory tracking control for articulated manipulators","volume":"23","author":"W Ha","year":"2025","journal-title":"Int J Control Autom Syst"},{"issue":"3","key":"pone.0354740.ref012","doi-asserted-by":"crossref","first-page":"1743","DOI":"10.31763\/ijrcs.v5i3.1905","article-title":"Disturbance Observer-Based Intelligent Control for Trajectory Tracking in Redundant Robotic Manipulators","volume":"5","author":"MHA Al-Mola","year":"2025","journal-title":"Int J Robotics Control Syst"},{"issue":"11","key":"pone.0354740.ref013","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1016\/j.cja.2023.06.028","volume":"36","author":"X Liang","year":"2023","journal-title":"Yournal of Aeronautics"},{"issue":"11","key":"pone.0354740.ref014","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1016\/j.cja.2023.06.028","article-title":"Disturbance observer based actor-critic learning control for uncertain nonlinear systems","volume":"36","author":"X Liang","year":"2023","journal-title":"Chinese J Aeronautics"},{"key":"pone.0354740.ref015","article-title":"Observer-based adaptive prescribed time optimal control for constrained nonlinear systems via actor-critic neural network","author":"J Geng","year":"2026","journal-title":"Neurocomputing"},{"issue":"7","key":"pone.0354740.ref016","doi-asserted-by":"crossref","first-page":"1364","DOI":"10.1002\/acs.3793","article-title":"Data\u2010driven disturbance compensation control for discrete\u2010time systems based on reinforcement learning","volume":"39","author":"L Li","year":"2024","journal-title":"Adaptive Control & Signal"},{"issue":"10","key":"pone.0354740.ref017","doi-asserted-by":"crossref","first-page":"1028","DOI":"10.3390\/e27101028","article-title":"ADP-based fault-tolerant control with stability guarantee for nonlinear systems","volume":"27","author":"L Liu","year":"2025","journal-title":"Entropy (Basel)"},{"issue":"3","key":"pone.0354740.ref018","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.11591\/ijpeds.v11.i3.pp1123-1131","article-title":"Adaptive dynamic programing based optimal control for a robot manipulator","volume":"11","author":"DP Nam","year":"2020","journal-title":"IJPEDS"},{"issue":"4","key":"pone.0354740.ref019","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1007\/s12555-021-0972-5","article-title":"Disturbance observer-based neural network integral sliding mode control for a constrained flexible joint robotic manipulator","volume":"21","author":"Q Wen","year":"2023","journal-title":"Int J Control Autom Syst"},{"key":"pone.0354740.ref020","doi-asserted-by":"crossref","first-page":"127179","DOI":"10.1016\/j.eswa.2025.127179","article-title":"Disturbance observer-based adaptive fuzzy finite-time cooperative control for high-order multi-agent systems with input saturation","volume":"276","author":"F Hu","year":"2025","journal-title":"Expert Systems with Applications"},{"issue":"6","key":"pone.0354740.ref021","doi-asserted-by":"crossref","first-page":"2621","DOI":"10.1109\/TCYB.2025.3557681","article-title":"Reinforcement-learning-based finite time fault tolerant control for a manipulator with actuator faults","volume":"55","author":"P Yang","year":"2025","journal-title":"IEEE Trans Cybern"},{"issue":"2","key":"pone.0354740.ref022","doi-asserted-by":"crossref","first-page":"4057","DOI":"10.48084\/etasr.2598","article-title":"Modern control laws for an articulated robotic arm: Modeling and simulation","volume":"9","author":"J Iqbal","year":"2019","journal-title":"Eng Technol Appl Sci Res"},{"issue":"4","key":"pone.0354740.ref023","doi-asserted-by":"crossref","first-page":"6131","DOI":"10.48084\/etasr.3648","article-title":"Robust adaptive tracking control of manipulator arms with Fuzzy Neural Networks","volume":"10","author":"M Fouzia","year":"2020","journal-title":"Eng Technol Appl Sci Res"},{"issue":"3","key":"pone.0354740.ref024","doi-asserted-by":"crossref","first-page":"22459","DOI":"10.48084\/etasr.10145","article-title":"Modeling and nonlinear backstepping control of a 3-DoF robot manipulator","volume":"15","author":"A Akil","year":"2025","journal-title":"Eng Technol Appl Sci Res"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0354740","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T17:38:16Z","timestamp":1791308296000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0354740"}},"subtitle":[],"editor":[{"given":"Longhui","family":"Qin","sequence":"first","affiliation":[{"name":"Southeast University","place":["CHINA"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,10,6]]},"references-count":24,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,10,6]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0354740","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,6]]}}}