{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:29:13Z","timestamp":1750220953863,"version":"3.41.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2019,7,24]],"date-time":"2019-07-24T00:00:00Z","timestamp":1563926400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["S8AS 1723997 and CCF:1421353"],"award-info":[{"award-number":["S8AS 1723997 and CCF:1421353"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2019,7,31]]},"abstract":"<jats:p>The successful deployment of autonomous real-time systems is contingent on their ability to recover from performance degradation of sensors, actuators, and other electro-mechanical subsystems with low latency. In this article, we introduce ALERA, a novel framework for real-time control law adaptation in nonlinear control systems assisted by system state encodings that generate an error signal when the code properties are violated in the presence of failures. The fundamental contributions of this methodology are twofold\u2014first, we show that the time-domain error signal contains perturbed system parameters\u2019 diagnostic information that can be used for quick control law adaptation to failure conditions and second, this quick adaptation is performed via reinforcement learning algorithms that relearn the control law of the perturbed system from a starting condition dictated by the diagnostic information, thus achieving significantly faster recovery. The fast (up to 80X faster than traditional reinforcement learning paradigms) performance recovery enabled by ALERA is demonstrated on an inverted pendulum balancing problem, a brake-by-wire system, and a self-balancing robot.<\/jats:p>","DOI":"10.1145\/3338123","type":"journal-article","created":{"date-parts":[[2019,7,25]],"date-time":"2019-07-25T12:34:33Z","timestamp":1564058073000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["ALERA"],"prefix":"10.1145","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5188-1651","authenticated-orcid":false,"given":"Suvadeep","family":"Banerjee","sequence":"first","affiliation":[{"name":"Intel Labs, Santa Clara, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abhijit","family":"Chatterjee","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, Atlanta, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,7,24]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVT.2007.895604"},{"volume-title":"IEEE 19th International On-Line Testing Symposium (IOLTS\u201913)","author":"Banerjee S.","key":"e_1_2_1_2_1","unstructured":"S. Banerjee , A. Banerjee , A. Chatterjee , and J. A. Abraham . 2013. Real-time checking of linear control systems using analog checksums . In IEEE 19th International On-Line Testing Symposium (IOLTS\u201913) . 122--127. S. Banerjee, A. Banerjee, A. Chatterjee, and J. A. Abraham. 2013. Real-time checking of linear control systems using analog checksums. In IEEE 19th International On-Line Testing Symposium (IOLTS\u201913). 122--127."},{"volume-title":"2017 22nd IEEE European Test Symposium (ETS). 1--6.","author":"Banerjee S.","key":"e_1_2_1_3_1","unstructured":"S. Banerjee and A. Chatterjee . 2017. Real-time self-learning for control law adaptation in nonlinear systems using encoded check states . In 2017 22nd IEEE European Test Symposium (ETS). 1--6. S. Banerjee and A. Chatterjee. 2017. Real-time self-learning for control law adaptation in nonlinear systems using encoded check states. In 2017 22nd IEEE European Test Symposium (ETS). 1--6."},{"volume-title":"2016 IEEE International Test Conference (ITC). 1--10","author":"Banerjee S.","key":"e_1_2_1_4_1","unstructured":"S. Banerjee , A. Chatterjee , and J. A. Abraham . 2016. Efficient cross-layer concurrent error detection in nonlinear control systems using mapped predictive check states . In 2016 IEEE International Test Conference (ITC). 1--10 . S. Banerjee, A. Chatterjee, and J. A. Abraham. 2016. Efficient cross-layer concurrent error detection in nonlinear control systems using mapped predictive check states. In 2016 IEEE International Test Conference (ITC). 1--10."},{"key":"e_1_2_1_5_1","unstructured":"S. Banerjee B. Samynathan J. Abraham and A. Chatterjee. 2019. Real-time error detection in nonlinear control systems using machine learning assisted state-space encoding. IEEE Transactions on Dependable and Secure Computing (2019) 1--1.  S. Banerjee B. Samynathan J. Abraham and A. Chatterjee. 2019. Real-time error detection in nonlinear control systems using machine learning assisted state-space encoding. IEEE Transactions on Dependable and Secure Computing (2019) 1--1."},{"key":"e_1_2_1_6_1","first-page":"5","article-title":"Neuronlike adaptive elements that can solve difficult learning control problems","volume":"13","author":"Barto A. G.","year":"1983","unstructured":"A. G. Barto , R. S. Sutton , and C. W. Anderson . 1983 . Neuronlike adaptive elements that can solve difficult learning control problems . IEEE Transactions on Systems, Man, and Cybernetics 13 , 5 (Sept 1983), 834--846. A. G. Barto, R. S. Sutton, and C. W. Anderson. 1983. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics 13, 5 (Sept 1983), 834--846.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.automatica.2009.07.008"},{"volume-title":"2007 European Control Conference (ECC). 1676--1681","author":"Buehner M. R.","key":"e_1_2_1_8_1","unstructured":"M. R. Buehner , C. W. Anderson , P. M. Young , K. A. Bush , and D. C. Hittle . 2007. Improving performance using robust recurrent reinforcement learning control . In 2007 European Control Conference (ECC). 1676--1681 . M. R. Buehner, C. W. Anderson, P. M. Young, K. A. Bush, and D. C. Hittle. 2007. Improving performance using robust recurrent reinforcement learning control. In 2007 European Control Conference (ECC). 1676--1681."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1049\/iet-cps.2017.0048"},{"volume-title":"2014 IEEE Global Communications Conference. 4387--4393","author":"Com\u015fa I. S.","key":"e_1_2_1_10_1","unstructured":"I. S. Com\u015fa , S. Zhang , M. Aydin , J. Chen , P. Kuonen , and J. F. Wagen . 2014. Adaptive proportional fair parameterization based LTE scheduling using continuous actor-critic reinforcement learning . In 2014 IEEE Global Communications Conference. 4387--4393 . I. S. Com\u015fa, S. Zhang, M. Aydin, J. Chen, P. Kuonen, and J. F. Wagen. 2014. Adaptive proportional fair parameterization based LTE scheduling using continuous actor-critic reinforcement learning. In 2014 IEEE Global Communications Conference. 4387--4393."},{"volume-title":"2015 IEEE International Conference on Robotics and Automation (ICRA). 2605--2612","author":"Cutler M.","key":"e_1_2_1_11_1","unstructured":"M. Cutler and J. P. How . 2015. Efficient reinforcement learning for robots using informative simulated priors . In 2015 IEEE International Conference on Robotics and Automation (ICRA). 2605--2612 . M. Cutler and J. P. How. 2015. Efficient reinforcement learning for robots using informative simulated priors. In 2015 IEEE International Conference on Robotics and Automation (ICRA). 2605--2612."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2017.1"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976600300015961"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2012.2218595"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2011.2170565"},{"key":"e_1_2_1_17_1","volume-title":"American Control Conference","volume":"6","author":"Gupta V.","year":"2004","unstructured":"V. Gupta , B. Hassibi , and R. M. Murray . 2004. On the synthesis of control laws for a network of autonomous agents . In American Control Conference , 2004 ., Vol. 6 . IEEE, 4927--4932. V. Gupta, B. Hassibi, and R. M. Murray. 2004. On the synthesis of control laws for a network of autonomous agents. In American Control Conference, 2004., Vol. 6. IEEE, 4927--4932."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/00207170500324175"},{"key":"e_1_2_1_19_1","first-page":"5","article-title":"Policy improvement by a model-free dyna architecture","volume":"24","author":"Hwang K. S.","year":"2013","unstructured":"K. S. Hwang and C. Y. Lo . 2013 . Policy improvement by a model-free dyna architecture . IEEE Transactions on Neural Networks and Learning Systems 24 , 5 (May 2013), 776--788. K. S. Hwang and C. Y. Lo. 2013. Policy improvement by a model-free dyna architecture. IEEE Transactions on Neural Networks and Learning Systems 24, 5 (May 2013), 776--788.","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.1996.477855"},{"volume-title":"Automotive Control Systems","author":"Kiencke Uwe","key":"e_1_2_1_21_1","unstructured":"Uwe Kiencke and Lars Nielsen . 2005. Automotive Control Systems . Springer-Verlag , Berlin . Uwe Kiencke and Lars Nielsen. 2005. Automotive Control Systems. Springer-Verlag, Berlin."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCST.2013.2293958"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0363012901385691"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2004.834123"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCAS.2009.933854"},{"key":"e_1_2_1_27_1","first-page":"6","article-title":"Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers","volume":"32","author":"Lewis F. L.","year":"2012","unstructured":"F. L. Lewis , D. Vrabie , and K. G. Vamvoudakis . 2012 . Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers . IEEE Control Systems 32 , 6 (Dec 2012), 76--105. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis. 2012. Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers. IEEE Control Systems 32, 6 (Dec 2012), 76--105.","journal-title":"IEEE Control Systems"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2017.2761841"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TWC.2014.022014.130840"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2015.2427198"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2015.2477810"},{"volume-title":"2017 IEEE 35th VLSI Test Symposium (VTS). 1--6.","author":"Momtaz M. I.","key":"e_1_2_1_32_1","unstructured":"M. I. Momtaz , S. Banerjee , and A. Chatterjee . 2017. On-line diagnosis and compensation for parametric failures in linear state variable circuits and systems using time-domain checksum observers . In 2017 IEEE 35th VLSI Test Symposium (VTS). 1--6. M. I. Momtaz, S. Banerjee, and A. Chatterjee. 2017. On-line diagnosis and compensation for parametric failures in linear state variable circuits and systems using time-domain checksum observers. In 2017 IEEE 35th VLSI Test Symposium (VTS). 1--6."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSMC.2016.2623867"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2007.11.026"},{"key":"e_1_2_1_35_1","unstructured":"Pololu Robotics and Electronics. {n.d.}. Balboa 32U4 balancing robot kit. ({n.d.}). https:\/\/goo.gl\/FUJeEW.  Pololu Robotics and Electronics. {n.d.}. Balboa 32U4 balancing robot kit. ({n.d.}). https:\/\/goo.gl\/FUJeEW."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICTAI.2012.100"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2015.2421338"},{"key":"e_1_2_1_38_1","volume-title":"Barto","author":"Sutton Richard S.","year":"1998","unstructured":"Richard S. Sutton and Andrew G . Barto . 1998 . Generalization and Function Approximation. MIT Press , 193--226. http:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=6282963. Richard S. Sutton and Andrew G. Barto. 1998. Generalization and Function Approximation. MIT Press, 193--226. http:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=6282963."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1177\/0142331216645176"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1177\/0142331215581638"},{"volume-title":"2015 5th International Conference on Information Science and Technology (ICIST). 243--248","author":"Wang B.","key":"e_1_2_1_41_1","unstructured":"B. Wang , D. Zhao , C. Li , and Y. Dai . 2015. Design and implementation of an adaptive cruise control system based on supervised actor-critic learning . In 2015 5th International Conference on Information Science and Technology (ICIST). 243--248 . B. Wang, D. Zhao, C. Li, and Y. Dai. 2015. Design and implementation of an adaptive cruise control system based on supervised actor-critic learning. In 2015 5th International Conference on Information Science and Technology (ICIST). 243--248."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.2015.2492941"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.2015.2478885"},{"key":"e_1_2_1_44_1","volume-title":"1999 American Control Conference.","volume":"6","author":"Yoshida K.","year":"1999","unstructured":"K. Yoshida . 1999 . Swing-up control of an inverted pendulum by energy-based methods . In 1999 American Control Conference. , Vol. 6 . 4045--4047 vol. 6. K. Yoshida. 1999. Swing-up control of an inverted pendulum by energy-based methods. In 1999 American Control Conference., Vol. 6. 4045--4047 vol. 6."},{"key":"e_1_2_1_45_1","first-page":"5","article-title":"Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method","volume":"64","author":"Zhang H.","year":"2017","unstructured":"H. Zhang , H. Jiang , Y. Luo , and G. Xiao . 2017 . Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method . IEEE Transactions on Industrial Electronics 64 , 5 (May 2017), 4091--4100. H. Zhang, H. Jiang, Y. Luo, and G. Xiao. 2017. Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method. IEEE Transactions on Industrial Electronics 64, 5 (May 2017), 4091--4100.","journal-title":"IEEE Transactions on Industrial Electronics"},{"volume-title":"2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). 1--6.","author":"Zhu Y.","key":"e_1_2_1_46_1","unstructured":"Y. Zhu and D. Zhao . 2014. A data-based online reinforcement learning algorithm with high-efficient exploration . In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). 1--6. Y. Zhu and D. Zhao. 2014. A data-based online reinforcement learning algorithm with high-efficient exploration. In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). 1--6."},{"key":"e_1_2_1_47_1","volume-title":"11th World Congress on Intelligent Control and Automation. 581--586","author":"Zhu Yuanheng","year":"2014","unstructured":"Yuanheng Zhu , Dongbin Zhao , and Haibo He . 2014 . An high-efficient online reinforcement learning algorithm for continuous-state systems . In 11th World Congress on Intelligent Control and Automation. 581--586 . Yuanheng Zhu, Dongbin Zhao, and Haibo He. 2014. An high-efficient online reinforcement learning algorithm for continuous-state systems. In 11th World Congress on Intelligent Control and Automation. 581--586."}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3338123","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3338123","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3338123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:54:02Z","timestamp":1750204442000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3338123"}},"subtitle":["Accelerated Reinforcement Learning Driven Adaptation to Electro-Mechanical Degradation in Nonlinear Control Systems Using Encoded State Space Error Signatures"],"short-title":[],"issued":{"date-parts":[[2019,7,24]]},"references-count":45,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,7,31]]}},"alternative-id":["10.1145\/3338123"],"URL":"https:\/\/doi.org\/10.1145\/3338123","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2019,7,24]]},"assertion":[{"value":"2019-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}