{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:04:40Z","timestamp":1787029480130,"version":"build-2736575974"},"reference-count":40,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T00:00:00Z","timestamp":1750723200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Deep Reinforcement Learning (DRL) is a powerful framework for solving complex sequential decision-making problems, particularly in robotic control. However, its practical deployment is often hindered by the substantial amount of experience required for learning, which results in high computational and time costs. In this work, we propose a novel integration of DRL with semantic knowledge in the form of Knowledge Graph Embeddings (KGEs), aiming to enhance learning efficiency by providing contextual information to the agent. Our architecture combines KGEs with visual observations, enabling the agent to exploit environmental knowledge during training. Experimental validation with robotic manipulators in environments featuring both fixed and randomized target attributes demonstrates that our method achieves up to 60% reduction in learning time and improves task accuracy by approximately 15 percentage points, without increasing training time or computational complexity. These results highlight the potential of semantic knowledge to reduce sample complexity and improve the effectiveness of DRL in robotic applications.<\/jats:p>","DOI":"10.3390\/robotics14070086","type":"journal-article","created":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T10:44:41Z","timestamp":1750761881000},"page":"86","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Boosting Deep Reinforcement Learning with Semantic Knowledge for Robotic Manipulators"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2426-3748","authenticated-orcid":false,"given":"Luc\u00eda","family":"G\u00fcitta-L\u00f3pez","sequence":"first","affiliation":[{"name":"Institute for Research in Technology (IIT), ICAI School of Engineering, Comillas Pontifical University, Rey Francisco, 4, 28008 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1199-8358","authenticated-orcid":false,"given":"Vincenzo","family":"Suriani","sequence":"additional","affiliation":[{"name":"Department of Engineering, University of Basilicata, Via dell\u2019Ateneo Lucano, 10, 85100 Potenza, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7547-0942","authenticated-orcid":false,"given":"Jaime","family":"Boal","sequence":"additional","affiliation":[{"name":"Institute for Research in Technology (IIT), ICAI School of Engineering, Comillas Pontifical University, Rey Francisco, 4, 28008 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9879-5603","authenticated-orcid":false,"given":"\u00c1lvaro J.","family":"L\u00f3pez-L\u00f3pez","sequence":"additional","affiliation":[{"name":"Institute for Research in Technology (IIT), ICAI School of Engineering, Comillas Pontifical University, Rey Francisco, 4, 28008 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6606-200X","authenticated-orcid":false,"given":"Daniele","family":"Nardi","sequence":"additional","affiliation":[{"name":"Department of Computer, Control and Management Engineering, Sapienza University of Rome, Via Ariosto, 25, 00185 Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1421","DOI":"10.1613\/jair.1.12412","article-title":"Deep Reinforcement Learning: A State-of-the-Art Walkthrough","volume":"69","author":"Lazaridis","year":"2020","journal-title":"J. Artif. Intell. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1561\/2200000071","article-title":"An introduction to deep reinforcement learning","volume":"11","author":"Henderson","year":"2018","journal-title":"Found. Trends Mach. Learn."},{"key":"ref_3","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_4","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"Lecun","year":"2015","journal-title":"Nature"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"4316","DOI":"10.1080\/00207543.2021.1973138","article-title":"Deep reinforcement learning in production systems: A systematic literature review","volume":"60","author":"Panzer","year":"2022","journal-title":"Int. J. Prod. Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.neunet.2022.05.013","article-title":"A survey for deep reinforcement learning in markovian cyber\u2013physical systems: Common problems and solutions","volume":"153","author":"Rupprecht","year":"2022","journal-title":"Neural Netw."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"945","DOI":"10.1007\/s10462-021-09997-9","article-title":"Reinforcement learning in robotic applications: A comprehensive survey","volume":"55","author":"Singh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.neunet.2018.03.014","article-title":"Intrinsically motivated reinforcement learning for human\u2013robot interaction in the real-world","volume":"107","author":"Qureshi","year":"2018","journal-title":"Neural Netw."},{"key":"ref_10","unstructured":"Davidson, G., and Lake, B.M. (2020). Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning. Comput. Res. Repos."},{"key":"ref_11","first-page":"1","article-title":"A review of robot learning for manipulation: Challenges, representations, and algorithms","volume":"22","author":"Kroemer","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cabi, S., Colmenarejo, S.G., Novikov, A., Konyushkova, K., Reed, S.E., Jeong, R., Zolna, K., Aytar, Y., Budden, D., and Vecer\u00edk, M. (2019). Scaling data-driven robotics with reward sketching and batch reinforcement learning. Robot. Sci. Syst. XVI.","DOI":"10.15607\/RSS.2020.XVI.076"},{"key":"ref_13","unstructured":"Wang, D., Walters, R., Zhu, X., and Platt, R. (2022, January 14\u201318). Equivariant q learning in spatial action spaces. Proceedings of the 5th Conference on Robot Learning (CoRL). PMLR, Auckland, New Zealand."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1177\/0278364919868017","article-title":"Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching","volume":"41","author":"Zeng","year":"2022","journal-title":"Int. J. Robot. Res."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Lee, Y., Hu, E.S., and Lim, J.J. (June, January 30). IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9560986"},{"key":"ref_16","unstructured":"Zhu, Y., Wong, J., Mandlekar, A., Mart\u00edn-Mart\u00edn, R., Joshi, A., Nasiriany, S., and Zhu, Y. (2020). robosuite: A modular simulation framework and benchmark for robot learning. arXiv."},{"key":"ref_17","unstructured":"Delhaisse, B., Rozo, L., and Caldwell, D.G. (November, January 30). PyRoboLearn: A Python Framework for Robot Learning Practitioners. Proceedings of the Conference on Robot Learning, PMLR, Virtual."},{"key":"ref_18","unstructured":"Ammanabrolu, P., and Hausknecht, M. (2020). Graph constrained reinforcement learning for natural language action spaces. arXiv."},{"key":"ref_19","unstructured":"Dambekodi, S., Frazier, S., Ammanabrolu, P., and Riedl, M.O. (2020). Playing text-based games with common sense. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Pternea, M., Singh, P., Chakraborty, A., Oruganti, Y., Milletari, M., Bapat, S., and Jiang, K. (2024). The RL\/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models. J. Artif. Int. Res., 80.","DOI":"10.1613\/jair.1.15960"},{"key":"ref_21","unstructured":"Wang, S., Zhang, S., Zhang, J., Hu, R., Li, X., Zhang, T., Li, J., Wu, F., Wang, G., and Hovy, E. (2025). Reinforcement Learning Enhanced LLMs: A Survey. arXiv."},{"key":"ref_22","unstructured":"Xie, T., Zhao, S., Wu, C.H., Liu, Y., Luo, Q., Zhong, V., Yang, Y., and Yu, T. (2023). Text2reward: Automated dense reward function generation for reinforcement learning. arXiv."},{"key":"ref_23","unstructured":"Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.Y. (2023, January 23\u201329). Grounding large language models in interactive environments with online reinforcement learning. Proceedings of the 40th International Conference on Machine Learning. PMLR, Honolulu, HI, USA."},{"key":"ref_24","unstructured":"Ma, Y.J., Liang, W., Wang, G., Huang, D.A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. (2024, January 7\u201311). Eureka: Human-Level Reward Design via Coding Large Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria."},{"key":"ref_25","unstructured":"Quartey, B., Shah, A., and Konidaris, G. (2023). Exploiting Contextual Structure to Generate Useful Auxiliary Tasks. arXiv."},{"key":"ref_26","unstructured":"Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. (2023, January 23\u201329). Guiding Pretraining in Reinforcement Learning with Large Language Models. Proceedings of the 40th International Conference on Machine Learning, PMLR, Honolulu, HI, USA."},{"key":"ref_27","unstructured":"Reid, M., Yamada, Y., and Gu, S.S. (2022). Can wikipedia help offline reinforcement learning?. arXiv."},{"key":"ref_28","unstructured":"Dasgupta, I., Kaeser-Chen, C., Marino, K., Ahuja, A., Babayan, S., Hill, F., and Fergus, R. (2023). Collaborating with language models for embodied reasoning. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Li, K., Wang, J., Yang, L., Lu, C., and Dai, B. (2025). SemGrasp: Semantic grasp generation via language aligned discretization. Computer Vision\u2014ECCV 2024, Springer.","DOI":"10.1007\/978-3-031-72627-9_7"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1109\/TETCI.2022.3222545","article-title":"Reinforcement learning on graphs: A survey","volume":"7","author":"Nie","year":"2023","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Miao, R., Jia, Q., Sun, F., Chen, G., Huang, H., and Miao, S. (2023). Semantic Representation of Robot Manipulation with Knowledge Graph. Entropy, 25.","DOI":"10.3390\/e25040657"},{"key":"ref_32","unstructured":"Pag\u00e8s, J., Marchionni, L., and Ferro, F. (2025, June 22). TIAGo: The Modular Robot That Adapts to Different Research Needs. Available online: https:\/\/www.clawar.org\/wp-content\/uploads\/2016\/10\/P2.pdf."},{"key":"ref_33","unstructured":"ABB (2023). IRB 120: Product Specification, ABB."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Todorov, E., Erez, T., and Tassa, Y. (2012, January 7\u201312). MuJoCo: A physics engine for model-based control. Proceedings of the IEEE International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, Portugal.","DOI":"10.1109\/IROS.2012.6386109"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_36","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Harley, T., Lillicrap, T.P., Silver, D., and Kavukcuoglu, K. (2016, January 20\u201322). Asynchronous methods for deep reinforcement learning. Proceedings of the 33rd International Conference on International Conference on Machine Learning\u2014Volume 48, New York, NY, USA. ICML\u201916."},{"key":"ref_37","unstructured":"Gu, Z., Jia, Z., and Choset, H. (30\u20133, January 30). Adversary A3C for Robust Reinforcement Learning. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1291","DOI":"10.1109\/TSMCC.2012.2218595","article-title":"A survey of actor-critic reinforcement learning: Standard and natural policy gradients","volume":"42","author":"Grondman","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_39","unstructured":"Babaeizadeh, M., Frosio, I., Tyree, S., Clemons, J., and Kautz, J. (2017, January 24\u201326). Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_40","first-page":"26","article-title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","volume":"4","author":"Tieleman","year":"2012","journal-title":"COURSERA Neural Netw. Mach. Learn."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/14\/7\/86\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:57:32Z","timestamp":1760032652000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/14\/7\/86"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,24]]},"references-count":40,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["robotics14070086"],"URL":"https:\/\/doi.org\/10.3390\/robotics14070086","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,24]]}}}