{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,19]],"date-time":"2025-12-19T15:32:09Z","timestamp":1766158329676,"version":"build-2065373602"},"reference-count":57,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2019,6,12]],"date-time":"2019-06-12T00:00:00Z","timestamp":1560297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>This work suggests a solution for the output reference model (ORM) tracking control problem, based on approximate dynamic programming. General nonlinear systems are included in a control system (CS) and subjected to state feedback. By linear ORM selection, indirect CS feedback linearization is obtained, leading to favorable linear behavior of the CS. The Value Iteration (VI) algorithm ensures model-free nonlinear state feedback controller learning, without relying on the process dynamics. From linear to nonlinear parameterizations, a reliable approximate VI implementation in continuous state-action spaces depends on several key parameters such as problem dimension, exploration of the state-action space, the state-transitions dataset size, and a suitable selection of the function approximators. Herein, we find that, given a transition sample dataset and a general linear parameterization of the Q-function, the ORM tracking performance obtained with an approximate VI scheme can reach the performance level of a more general implementation using neural networks (NNs). Although the NN-based implementation takes more time to learn due to its higher complexity (more parameters), it is less sensitive to exploration settings, number of transition samples, and to the selected hyper-parameters, hence it is recommending as the de facto practical implementation. Contributions of this work include the following: VI convergence is guaranteed under general function approximators; a case study for a low-order linear system in order to generalize the more complex ORM tracking validation on a real-world nonlinear multivariable aerodynamic process; comparisons with an offline deep deterministic policy gradient solution; implementation details and further discussions on the obtained results.<\/jats:p>","DOI":"10.3390\/a12060121","type":"journal-article","created":{"date-parts":[[2019,6,12]],"date-time":"2019-06-12T10:55:19Z","timestamp":1560336919000},"page":"121","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8410-6547","authenticated-orcid":false,"given":"Mircea-Bogdan","family":"Radac","sequence":"first","affiliation":[{"name":"Department of Automation and Applied Informatics, Politehnica University of Timisoara, 2 Bd. V. Parvan, 300223 Timisoara, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timotei","family":"Lala","sequence":"additional","affiliation":[{"name":"Department of Automation and Applied Informatics, Politehnica University of Timisoara, 2 Bd. V. Parvan, 300223 Timisoara, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,6,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2925","DOI":"10.1109\/TNNLS.2015.2460258","article-title":"Model-free primitive-based iterative learning control approach to trajectory tracking of MIMO systems with experimental validation","volume":"26","author":"Radac","year":"2015","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_3","unstructured":"Bertsekas, D.P., and Tsitsiklis, J.N. (1996). Neuro-Dynamic Programming, Athena Scientific."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/MCI.2009.932261","article-title":"Adaptive dynamic programming: an introduction","volume":"4","author":"Wang","year":"2009","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/MCS.2012.2214134","article-title":"Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers","volume":"32","author":"Lewis","year":"2012","journal-title":"IEEE Control Syst. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/MCAS.2009.933854","article-title":"Reinforcement learning and adaptive dynamic programming for feedback control","volume":"9","author":"Lewis","year":"2009","journal-title":"IEEE Circ. Syst. Mag."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1109\/TSMCC.2002.801727","article-title":"Adaptive dynamic programming","volume":"32","author":"Murray","year":"2002","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2270","DOI":"10.1109\/TCYB.2014.2384016","article-title":"Optimal tracking control of unknown discrete-time linear systems using input\u2013output measured data","volume":"45","author":"Kiumarsi","year":"2015","journal-title":"IEEE Trans. Cybern."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1016\/j.automatica.2014.02.015","article-title":"Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics","volume":"50","author":"Kiumarsi","year":"2014","journal-title":"Automatica"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1071","DOI":"10.1080\/00207721.2016.1236423","article-title":"Model-free control performance improvement using virtual reference feedback tuning and reinforcement Q-learning","volume":"48","author":"Radac","year":"2017","journal-title":"Int. J. Syst. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Lala, T., and Radac, M.B. (2019, January 1\u20134). Parameterized value iteration for output reference model tracking of a high order nonlinear aerodynamic system. Proceedings of the 2019 27th Mediterranean Conference on Control and Automation (MED), Akko, Israel.","DOI":"10.1109\/MED.2019.8798580"},{"key":"ref_13","first-page":"2089","article-title":"Tree-based batch mode reinforcement learning","volume":"6","author":"Ernst","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1007\/s10994-011-5235-x","article-title":"Reinforcement learning in feedback control. Challenges and benchmarks from technical process control","volume":"84","author":"Hafner","year":"2011","journal-title":"Mach. Learn."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2089","DOI":"10.1007\/s00500-013-1110-y","article-title":"A supervised actor critic approach for adaptive cruise control","volume":"17","author":"Zhao","year":"2013","journal-title":"Soft Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1019","DOI":"10.1109\/TSMC.2016.2645699","article-title":"Adaptive neural network control of AUVs with Control input nonlinearities using reinforcement learning","volume":"47","author":"Cui","year":"2017","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"762","DOI":"10.1109\/TNNLS.2012.2236354","article-title":"Online learning control using adaptive critic designs with sparse kernel machines","volume":"24","author":"Xu","year":"2013","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.neucom.2011.05.031","article-title":"A three-network architecture for on-line learning and optimization based on adaptive dynamic programming","volume":"78","author":"He","year":"2012","journal-title":"Neurocomputing"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2550","DOI":"10.1109\/TNNLS.2015.2441749","article-title":"H\u221e Tracking control of completely unknown continuous-time systems via off-policy reinforcement learning","volume":"26","author":"Modares","year":"2015","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2434","DOI":"10.1109\/TNNLS.2016.2609500","article-title":"Off-policy reinforcement learning for synchronization in multiagent graphical games","volume":"28","author":"Li","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1109\/TNNLS.2015.2503980","article-title":"Value and policy iterations in optimal control and adaptive dynamic programming","volume":"28","author":"Bertsekas","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1929","DOI":"10.1109\/TNNLS.2017.2654324","article-title":"Hamiltonian-driven adaptive dynamic programming for continuous nonlinear dynamical systems","volume":"28","author":"Yang","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1109\/TNNLS.2015.2511658","article-title":"Model-based reinforcement learning for infinite horizon approximate optimal tracking","volume":"28","author":"Kamalapurkar","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"992","DOI":"10.1016\/j.asoc.2018.09.035","article-title":"Data-driven MIMO Model-free reference tracking control with nonlinear state-feedback and fractional order controllers","volume":"73","author":"Radac","year":"2018","journal-title":"Appl. Soft Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1337","DOI":"10.1016\/S0005-1098(02)00032-8","article-title":"Virtual Reference Feedback Tuning: A direct method for the design of feedback controllers","volume":"38","author":"Campi","year":"2002","journal-title":"Automatica"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1002\/acs.714","article-title":"Iterative Feedback Tuning\u2014An overview","volume":"16","author":"Hjalmarsson","year":"2002","journal-title":"Int. J. Adapt. Control Signal Process."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Janssens, P., Pipeleers, G., and Swevers, J.L. (July, January 29). Model-free iterative learning control for LTI systems and experimental validation on a linear motor test setup. Proceedings of the 2011 American Control Conference (ACC), San Francisco, CA, USA.","DOI":"10.1109\/ACC.2011.5990798"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.compind.2015.03.004","article-title":"Optimal behavior prediction using a primitive-based data-driven model-free iterative learning control approach","volume":"74","author":"Radac","year":"2015","journal-title":"Comput. Ind."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1109\/TSMC.2017.2693397","article-title":"An improved data-driven point-to-point ILC using additional on-line control inputs with experimental verification","volume":"49","author":"Chi","year":"2017","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1018","DOI":"10.1080\/00207179.2016.1193223","article-title":"On ramp metering: towards a better understanding of ALINEA via model-free control","volume":"90","author":"Abouaissa","year":"2017","journal-title":"Int. J. Control"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1914","DOI":"10.1109\/TNNLS.2016.2561702","article-title":"Lazy-learning-based data-driven model-free adaptive predictive control for a class of discrete-time nonlinear systems","volume":"28","author":"Hou","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1016\/j.isatra.2018.01.014","article-title":"Data-driven model reference control of MIMO vertical tank systems with model-free VRFT and Q-learning","volume":"73","author":"Radac","year":"2018","journal-title":"ISA Trans."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"3728","DOI":"10.1002\/rnc.3611","article-title":"Data-driven multivariable ILC: enhanced performance by eliminating L and Q filters","volume":"28","author":"Bolder","year":"2018","journal-title":"Int. J. Robot. Nonlinear Control"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"4121","DOI":"10.1109\/TIE.2016.2617830","article-title":"An indirect data-driven method for trajectory tracking control of a class of nonlinear discrete-time systems","volume":"64","author":"Wang","year":"2017","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.jprocont.2018.07.013","article-title":"Control of a bioreactor using a new partially supervised reinforcement learning algorithm","volume":"69","author":"Pandian","year":"2018","journal-title":"J. Proc. Control"},{"key":"ref_36","unstructured":"Diaz, H., Armesto, L., and Sala, A. (2018). Fitted q-function control methodology based on takagi-sugeno systems. IEEE Trans. Control Syst. Tech."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Wang, W., Chen, X., Fu, H., and Wu, M. (2019). Data-driven adaptive dynamic programming for partially observable nonzero-sum games via Q-learning method. Int. J. Syst. Sci.","DOI":"10.1080\/00207721.2019.1599463"},{"key":"ref_38","unstructured":"Mu, C., and Zhang, Y. (2019). Learning-based robust tracking control of quadrotor with time-varying and coupling uncertainties. IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2298","DOI":"10.1080\/00207721.2018.1498557","article-title":"Model-free adaptive control design for nonlinear discrete-time processes with reinforcement learning techniques","volume":"49","author":"Liu","year":"2018","journal-title":"Int. J. Syst. Sci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"3106","DOI":"10.1109\/TIE.2018.2842756","article-title":"Data-driven iterative feedforward tuning for a wafer stage: A high-order approach based on instrumental variables","volume":"66","author":"Song","year":"2019","journal-title":"IEEE Trans. Ind. Electr."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kofinas, P., and Dounis, A. (2018). Fuzzy Q-learning agent for online tuning of PID controller for DC motor speed control. Algorithms, 11.","DOI":"10.3390\/a11100148"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.engappai.2016.06.009","article-title":"Three-level hierarchical model-free learning approach to trajectory tracking control","volume":"55","author":"Radac","year":"2016","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1000","DOI":"10.1049\/iet-cta.2014.0187","article-title":"Data-based two-degree-of-freedom iterative control approach to constrained non-linear systems","volume":"9","author":"Radac","year":"2015","journal-title":"IET Control Theory Appl."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1016\/j.eswa.2017.12.036","article-title":"Measuring the emotional state among interacting agents: A game theory approach using reinforcement learning","volume":"97","author":"Salgado","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.eswa.2019.04.056","article-title":"A reinforcement learning-based multi-agent framework applied for solving routing and scheduling problems","volume":"131","author":"Silva","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Radac, M.-B., Precup, R.-E., Hedrea, E.-L., and Mituletu, I.-C. (2018, January 19\u201322). Data-driven model-free model-reference nonlinear virtual state feedback control from input-output data. Proceedings of the 2018 26th Mediterranean Conference on Control and Automation (MED), Zadar, Croatia.","DOI":"10.1109\/MED.2018.8442894"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1778","DOI":"10.1016\/j.automatica.2011.04.002","article-title":"Virtual reference tuning for non-minimum phase plants","volume":"47","author":"Campestrini","year":"2011","journal-title":"Automatica"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1109\/TSMCB.2008.926614","article-title":"Discrete-time nonlinear HJB Solution using approximate dynamic programming: Convergence proof","volume":"38","author":"Lewis","year":"2008","journal-title":"IEEE Trans. Syst. Man Cybern. Cybern."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1049\/ip-cta:20050094","article-title":"Relaxed dynamic programming in switching systems","volume":"153","author":"Rantzer","year":"2006","journal-title":"IEEE Proc. Control Theory Appl."},{"key":"ref_50","unstructured":"Inteco, LTD (2007). Two Rotor Aerodynamical System, Inteco, LTD. Available online: http:\/\/ee.sharif.edu\/~lcsl\/lab\/Tras_um_PCI.pdf."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1016\/j.neucom.2017.08.036","article-title":"Data-driven model-free slip control of anti-lock braking systems using reinforcement Q-learning","volume":"275","author":"Radac","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_52","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016). Continuous control with deep reinforcement learning. Mach. Learn."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/s00521-003-0369-9","article-title":"Neural Q-learning","volume":"12","author":"Krose","year":"2003","journal-title":"Neural Comput. Appl."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Radac, M.B., and Precup, R.E. (2019). Data-Driven model-free tracking reinforcement learning control with VRFT-based adaptive actor-critic. Appl. Sci., 9.","DOI":"10.3390\/app9091807"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1118","DOI":"10.1109\/TNNLS.2012.2196708","article-title":"Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update","volume":"23","author":"Dierks","year":"2012","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"301","DOI":"10.2514\/1.G001154","article-title":"Theoretical and numerical analysis of approximate dynamic programming with approximation errors","volume":"39","author":"Heydari","year":"2016","journal-title":"J. Gui Control Dyn."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"2733","DOI":"10.1109\/TCYB.2014.2314612","article-title":"Revisiting approximate dynamic programming and its convergence","volume":"44","author":"Heydari","year":"2014","journal-title":"IEEE Trans. Cybern."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/12\/6\/121\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:57:51Z","timestamp":1760187471000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/12\/6\/121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,12]]},"references-count":57,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2019,6]]}},"alternative-id":["a12060121"],"URL":"https:\/\/doi.org\/10.3390\/a12060121","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2019,6,12]]}}}