{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T04:08:01Z","timestamp":1768536481196,"version":"3.49.0"},"reference-count":46,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,3,9]],"date-time":"2021-03-09T00:00:00Z","timestamp":1615248000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Advances in machine learning technologies in recent years have facilitated developments in autonomous robotic systems. Designing these autonomous systems typically requires manually specified models of the robotic system and world when using classical control-based strategies, or time consuming and computationally expensive data-driven training when using learning-based strategies. Combination of classical control and learning-based strategies may mitigate both requirements. However, the performance of the combined control system is not obvious given that there are two separate controllers. This paper focuses on one such combination, which uses gravity-compensation together with reinforcement learning (RL). We present a study of the effects of gravity compensation on the performance of two reinforcement learning algorithms when solving reaching tasks using a simulated seven-degree-of-freedom robotic arm. The results of our study demonstrate that gravity compensation coupled with RL can reduce the training required in reaching tasks involving elevated target locations, but not all target locations.<\/jats:p>","DOI":"10.3390\/robotics10010046","type":"journal-article","created":{"date-parts":[[2021,3,9]],"date-time":"2021-03-09T06:22:32Z","timestamp":1615270952000},"page":"46","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["On the Impact of Gravity Compensation on Reinforcement Learning in Goal-Reaching Tasks for Robotic Manipulators"],"prefix":"10.3390","volume":"10","author":[{"given":"Jonathan","family":"Fugal","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY 40506, USA"}]},{"given":"Jihye","family":"Bae","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY 40506, USA"}]},{"given":"Hasan A.","family":"Poonawala","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, University of Kentucky, Lexington, KY 40506, USA"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,9]]},"reference":[{"key":"ref_1","unstructured":"Rottmann, A., Mozos, O.M., Stachniss, C., and Burgard, W. (2005, January 9\u201313). Semantic Place Classification of Indoor Environments with Mobile Robots Using Boosting. Proceedings of the 20th National Conference on Artificial Intelligence, AAAI\u201905, Pittsburgh, PA, USA."},{"key":"ref_2","unstructured":"Spong, M.W., Hutchinson, S., and Vidyasagar, M. (2006). Robot Modeling and Control, Wiley and Sons."},{"key":"ref_3","unstructured":"Safiotti, A. (1997, January 5). Fuzzy logic in autonomous robotics: Behavior coordination. Proceedings of the 6th International Fuzzy Systems Conference, Barcelona, Spain."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Chen, C.H., Wang, C.C., Wang, Y.T., and Wang, P.T. (2017). Fuzzy Logic Controller Design for Intelligent Robots. Math. Probl. Eng.","DOI":"10.1155\/2017\/8984713"},{"key":"ref_5","unstructured":"Antonelli, G., Chiaverini, S., Sarkar, N., and West, M. (1999, January 8\u20139). Adaptive control of an autonomous underwater vehicle: Experimental results on ODIN. Proceedings of the 1999 IEEE International Symposium on Computational Intelligence in Robotics and Automation, CIRA\u201999 (Cat. No.99EX375), Monterey, CA, USA."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wei, B. (2019). A Tutorial on Robust Control, Adaptive Control and Robust Adaptive Control\u2014Application to Robotic Manipulators. Inventions, 4.","DOI":"10.3390\/inventions4030049"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/S0921-8890(97)00004-3","article-title":"The dynamic approach to autonomous robotics demonstrated on a low-level vehicle platform","volume":"21","author":"Bicho","year":"1997","journal-title":"Robot. Auton. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1016\/0921-8890(95)00049-6","article-title":"Dynamics of behavior: Theory and applications for autonomous robot architectures","volume":"16","author":"Dose","year":"1995","journal-title":"Robot. Auton. Syst."},{"key":"ref_9","unstructured":"Fu, J., Luo, K., and Levine, S. (2018). Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.artint.2018.07.002","article-title":"Multi-robot inverse reinforcement learning under occlusion with estimation of state transitions","volume":"263","author":"Bogert","year":"2018","journal-title":"Artif. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1177\/0278364913495721","article-title":"Reinforcement learning in robotics: A survey","volume":"32","author":"Kober","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_12","unstructured":"Gu, S., Holly, E., Lillicrap, T., and Levine, S. (June, January 29). Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Singapore."},{"key":"ref_13","unstructured":"Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2015). End-to-End Training of Deep Visuomotor Policies. arXiv."},{"key":"ref_14","unstructured":"Li, W., and Todorov, E. (2004, January 25\u201328). Iterative Linear Quadratic Regulator Design For Nonlinear Biological Movement Systems. Proceedings of the First International Conference on Informatics in Control, Automation and Robotics, Set\u00fabal, Portugal."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1109\/TBME.2008.926699","article-title":"Coadaptive Brain\u2013Machine Interface via Reinforcement Learning","volume":"56","author":"DiGiovanna","year":"2009","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Bae, J., Sanchez Giraldo, L.G., Pohlmeyer, E.A., Francis, J.T., Sanchez, J.C., and Pr\u00edncipe, J.C. (2015). Kernel Temporal Differences for Neural Decoding. Intell. Neurosci., 2015.","DOI":"10.1155\/2015\/481375"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Abbeel, P., Coates, A., Quigley, M., and Ng, A.Y. (2007, January 4\u20137). An application of reinforcement learning to aerobatic helicopter flight. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.","DOI":"10.7551\/mitpress\/7503.003.0006"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Mart\u00edn-Mart\u00edn, R., Lee, M.A., Gardner, R., Savarese, S., Bohg, J., and Garg, A. (2019). Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks. arXiv.","DOI":"10.1109\/IROS40897.2019.8968201"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J.A., Solowjow, E., and Levine, S. (2019, January 20\u201324). Residual Reinforcement Learning for Robot Control. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794127"},{"key":"ref_20","first-page":"1","article-title":"Gravity compensation in robotics","volume":"30","author":"Arakelian","year":"2015","journal-title":"Adv. Robot."},{"key":"ref_21","unstructured":"Ernesto, H., and Pedro, J.O. (2015). Reinforcement learning in robotics: A survey. Math. Probl. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yu, C., Li, Z., and Liu, H. (2019, January 8\u201312). Research on Gravity Compensation of Robot Arm Based on Model Learning. Proceedings of the 2019 IEEE\/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Hong Kong, China.","DOI":"10.1109\/AIM.2019.8868673"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Montalvo, W., Escobar-Naranjo, J., Garcia, C.A., and Garcia, M.V. (2020). Low-Cost Automation for Gravity Compensation of Robotic Arm. Appl. Sci., 10.","DOI":"10.3390\/app10113823"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Alothman, Y., and Gu, D. (2016, January 28\u201330). Quadrotor transporting cable-suspended load using iterative linear quadratic regulator (ilqr) optimal control. Proceedings of the 2016 8th Computer Science and Electronic Engineering (CEEC), Colchester, UK.","DOI":"10.1109\/CEEC.2016.7835908"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Puterman, M.L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc.. [1st ed.].","DOI":"10.1002\/9780470316887"},{"key":"ref_26","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.]."},{"key":"ref_27","unstructured":"Wu, Y., Mansimov, E., Liao, S., Grosse, R.B., and Ba, J. (2017). Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation. arXiv."},{"key":"ref_28","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2017). Deep Reinforcement Learning that Matters. arXiv.","DOI":"10.1609\/aaai.v32i1.11694"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Aumjaud, P., McAuliffe, D., Rodriguez Lera, F.J., and Cardiff, P. (2020, January 19\u201320). Reinforcement Learning Experiments and Benchmark for Solving Robotic Reaching Tasks. Proceedings of the 21st International Workshop of Physical Agents (WAF 2020), Madrid, Spain.","DOI":"10.1007\/978-3-030-62579-5_22"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1162\/089976698300017746","article-title":"Natural Gradient Works Efficiently in Learning","volume":"10","author":"Amari","year":"1998","journal-title":"Neural Comput."},{"key":"ref_32","unstructured":"Kakade, S.M. (2001). A Natural Policy Gradient. Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, MIT Press."},{"key":"ref_33","unstructured":"Schulman, J., Levine, S., Moritz, P., Jordan, M.I., and Abbeel, P. (2017). Trust Region Policy Optimization. arXiv."},{"key":"ref_34","unstructured":"Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O., and Nichol, A. (2019, June 01). Stable Baselines. Available online: https:\/\/github.com\/hill-a\/stable-baselines."},{"key":"ref_35","unstructured":"Grosse, R., and Martens, J. (2016). A Kronecker-factored approximate Fisher matrix for convolution layers. arXiv."},{"key":"ref_36","unstructured":"Fan, L., Zhu, Y., Zhu, J., Liu, Z., Zeng, O., Gupta, A., Creus-Costa, J., Savarese, S., and Fei-Fei, L. (2018, January 29\u201331). SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark. Proceedings of the Conference on Robot Learning, Z\u00fcrich, Switzerland."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Todorov, E., Erez, T., and Tassa, Y. (2012, January 7\u201312). MuJoCo: A physics engine for model-based control. Proceedings of the 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, Portugal.","DOI":"10.1109\/IROS.2012.6386109"},{"key":"ref_38","unstructured":"Babaeizadeh, M., Frosio, I., Tyree, S., Clemons, J., and Kautz, J. (2016). GA3C: GPU-based A3C for Deep Reinforcement Learning. arXiv."},{"key":"ref_39","unstructured":"Falkner, S., Klein, A., and Hutter, F. (2018). BOHB: Robust and Efficient Hyperparameter Optimization at Scale. arXiv."},{"key":"ref_40","first-page":"6765","article-title":"Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization","volume":"18","author":"Li","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"ref_41","unstructured":"Paul, S., Kurin, V., and Whiteson, S. (2019). Fast Efficient Hyperparameter Tuning for Policy Gradients. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1109\/TSMC.2014.2358639","article-title":"Multiobjective Reinforcement Learning: A Comprehensive Overview","volume":"45","author":"Liu","year":"2015","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Pohlmeyer, E.A., Mahmoudi, B., Geng, S., Prins, N.W., and Sanchez, J.C. (2014). Using Reinforcement Learning to Provide Stable Brain-Machine Interface Control Despite Neural Input Reorganization. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0087253"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"An, J., Yadav, T., Ahmadi, M.B., Tarigoppula, V.S.A., and Francis, J.T. (2018, January 17\u201321). Near Perfect Neural Critic from Motor Cortical Activity Toward an Autonomously Updating Brain Machine Interface. Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.","DOI":"10.1109\/EMBC.2018.8512274"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Shaikh, S., So, R., Sibindi, T., Libedinsky, C., and Basu, A. (2020, January 10\u201321). Towards Autonomous Intra-Cortical Brain Machine Interfaces: Applying Bandit Algorithms for Online Reinforcement Learning. Proceedings of the 2020 IEEE International Symposium on Circuits and Systems (ISCAS), Seville, Spain.","DOI":"10.1109\/ISCAS45731.2020.9180906"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"3089","DOI":"10.1109\/TNSRE.2020.3039970","article-title":"Task Learning Over Multi-Day Recording via Internally Rewarded Reinforcement Learning Based Brain Machine Interfaces","volume":"28","author":"Shen","year":"2020","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/10\/1\/46\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:35:23Z","timestamp":1760160923000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/10\/1\/46"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,9]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["robotics10010046"],"URL":"https:\/\/doi.org\/10.3390\/robotics10010046","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,9]]}}}