{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T18:04:44Z","timestamp":1772215484942,"version":"3.50.1"},"reference-count":63,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T00:00:00Z","timestamp":1772150400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>This paper presents the design and implementation of an integrated robotic system capable of detecting objects through computer vision and making decisions based on logic strategies to perform physical tasks. For that, the system uses a robotic arm to play the Tic-Tac-Toe game utilizing a Q-learning algorithm to determine optimal moves. The system can be controlled using a graphical interface that enables real-time monitoring, facilitating seamless interaction between the user and the robotic arm. Three algorithms with different decision strategies were developed: a random decision algorithm, the MiniMax algorithm, and Q-learning, a reinforcement-learning algorithm. The results obtained highlight the control of the robotic arm using kinematic equations, the training of a robust YOLOv5 model, and the effective learning capability of a Q-learning algorithm. The proposed system presents practical implementation of the robotic system which can be used as a basis for further projects and for teaching robotics.<\/jats:p>","DOI":"10.3390\/robotics15030050","type":"journal-article","created":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T17:05:16Z","timestamp":1772211916000},"page":"50","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Robotic Arm Control Using a Q-Learning Reinforcement Algorithm"],"prefix":"10.3390","volume":"15","author":[{"given":"Afonso M.","family":"Tim\u00f3teo","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, Institute of Engineering\u2014Polytechnic of Porto (ISEP\/IPP), Rua Dr. Ant\u00f3nio Bernardino de Almeida, 431, 4249-015 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7410-8872","authenticated-orcid":false,"given":"Ramiro S.","family":"Barbosa","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Institute of Engineering\u2014Polytechnic of Porto (ISEP\/IPP), Rua Dr. Ant\u00f3nio Bernardino de Almeida, 431, 4249-015 Porto, Portugal"},{"name":"GECAD\u2014Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development, Institute of Engineering\u2014Polytechnic of Porto (ISEP\/IPP), 4249-015 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7545-5822","authenticated-orcid":false,"given":"Isabel S.","family":"Jesus","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Institute of Engineering\u2014Polytechnic of Porto (ISEP\/IPP), Rua Dr. Ant\u00f3nio Bernardino de Almeida, 431, 4249-015 Porto, Portugal"},{"name":"GECAD\u2014Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development, Institute of Engineering\u2014Polytechnic of Porto (ISEP\/IPP), 4249-015 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2026,2,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/j.dsm.2021.12.002","article-title":"Machine learning-based approach: Global trends, research directions, and regulatory standpoints","volume":"4","author":"Pugliese","year":"2021","journal-title":"Data Sci. Manag."},{"key":"ref_2","unstructured":"Agarwal, N., and Yadav, D. (2024). A Comprehensive Analysis of Classical Machine Learning and Modern Deep Learning Methodologies. Int. J. Eng. Res. Technol., 13."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1007\/s42979-021-00815-1","article-title":"Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions","volume":"2","author":"Sarker","year":"2021","journal-title":"SN Comput. Sci."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1007\/s42979-021-00592-x","article-title":"Machine Learning: Algorithms, Real-World Applications and Research Directions","volume":"2","author":"Sarker","year":"2021","journal-title":"SN Comput. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1007\/s11701-006-0002-x","article-title":"Evolution of robotic arms","volume":"1","author":"Moran","year":"2007","journal-title":"J. Robot. Surg."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.artint.2014.11.003","article-title":"Deliberation for autonomous robots: A survey","volume":"247","author":"Ingrand","year":"2017","journal-title":"Artif. Intell."},{"key":"ref_7","unstructured":"Matari\u0107, J.M. (2007). The Robotics Primer, Massachusetts Institute of Technology."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1109\/TITS.2019.2962338","article-title":"A Survey of Deep Learning Applications to Autonomous Vehicle Control","volume":"22","author":"Kuutti","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Parekh, D., Poddar, N., Rajpurkar, A., Chahal, M., Kumar, N., Joshi, G.P., and Cho, W. (2022). A Review on Autonomous Vehicles: Progress, Methods and Challenges. Electronics, 11.","DOI":"10.3390\/electronics11142162"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Leal, H.M., Barbosa, R.S., and Jesus, I.S. (2025). Control of a Mobile Line-Following Robot Using Neural Networks. Algorithms, 18.","DOI":"10.3390\/a18010051"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/s11701-007-0021-2","article-title":"A history of robots: From science fiction to surgical robotics","volume":"1","author":"Hockstein","year":"2007","journal-title":"J. Robot. Surg."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Li, M., Milojevi\u0107, A., and Handroos, H. (2020). Robotics in Manufacturing-The Past and the Present. Technical, Economic and Societal Effects of Manufacturing 4.0: Automation, Adaption and Manufacturing in Finland and Beyond, Palgrave Macmillan.","DOI":"10.1007\/978-3-030-46103-4_4"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1007\/s11701-010-0202-2","article-title":"History of robotic surgery","volume":"4","author":"Kalan","year":"2010","journal-title":"J. Robot. Surg."},{"key":"ref_14","unstructured":"Zamalloa, I., Kojcev, R., Hern\u00e1ndez, A., Muguruza, I., Usategui, L., Bilbao, A., and Mayoral, V. (2017). Dissecting Robotics\u2014Historical overview and future perspectives. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chen, M., Wang, X., Law, R., and Zhang, M. (2023). Research on the Frontier and Prospect of Service Robots in the Tourism and Hospitality Industry Based on International Core Journals: A Review. Behav. Sci., 13.","DOI":"10.3390\/bs13070560"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, J., and Herath, D. (2022). What Makes Robots? Sensors, Actuators, and Algorithms. Foundations of Robotics, Springer.","DOI":"10.1007\/978-981-19-1983-1_7"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"i","DOI":"10.1016\/S1052-5149(20)30067-8","article-title":"Machine Learning and Other Artificial Intelligence Applications","volume":"30","author":"Forghani","year":"2020","journal-title":"Neuroimaging Clin."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1140","DOI":"10.1126\/science.aar6404","article-title":"A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sivamayil, K., Rajasekar, E., Aljafari, B., Nikolovski, S., Vairavasundaram, S., and Vairavasundaram, I. (2023). A Systematic Study on Reinforcement Learning Based Applications. Energies, 16.","DOI":"10.3390\/en16031512"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"100471","DOI":"10.1016\/j.rico.2024.100471","article-title":"Multi-agent Dual Level Reinforcement Learning of Strategy and Tactics in Competitive Games","volume":"16","author":"Yuan","year":"2024","journal-title":"Results Control Optim."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1007\/s00354-019-00054-2","article-title":"Machine Discovery of Comprehensible Strategies for Simple Games Using Meta-interpretive Learning","volume":"37","author":"Muggleton","year":"2019","journal-title":"New Gener. Comput."},{"key":"ref_22","first-page":"251","article-title":"Deep Reinforcement Learning Applied to a Robotic Pick-and-Place Application","volume":"Volume 1488","author":"Gomes","year":"2021","journal-title":"Communications in Computer and Information Science (CCIS), Proceedings of the Optimization, Learning Algorithms and Applications\u2014First International Conference, OL2A 2021, Revised Selected Papers"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lobbezoo, A., Qian, Y., and Kwon, H.J. (2021). Reinforcement Learning for Pick and Place Operations in Robotics: A Survey. Robotics, 10.","DOI":"10.3390\/robotics10030105"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wan, F., Wang, H., Liu, X., Yang, L., and Song, C. (2020). DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation. arXiv.","DOI":"10.1109\/AIM43001.2020.9159011"},{"key":"ref_25","unstructured":"Lebling, R.W. (2025, September 01). Robots of Ages Past. Available online: https:\/\/www.aramcoworld.com\/Articles\/November-2019\/Robots-of-Ages-Past."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Patidar, V., and Tiwari, R. (2016, January 7\u20139). Survey of robotic arm and parameters. Proceedings of the 2016 International Conference on Computer Communication and Informatics, ICCCI 2016, Coimbatore, India.","DOI":"10.1109\/ICCCI.2016.7479938"},{"key":"ref_27","unstructured":"Baichun, M.Z., and Ceccarelli, M. (2019). From the Unimate to the Delta Robot: The Early Decades of Industrial Robotics. Proceedings of the Explorations in the History and Heritage of Machines and Mechanisms, Springer International Publishing."},{"key":"ref_28","unstructured":"Muhamedyev, R.I. (2015). Machine learning methods: An overview. Computer Modelling and New Technologies, Springer."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"10651","DOI":"10.1007\/s10462-023-10438-y","article-title":"An efficient lightweight convolutional neural network for industrial surface defect detection","volume":"56","author":"Zhang","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_30","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_31","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.]. A Bradford Book."},{"key":"ref_32","first-page":"90","article-title":"Design and Implementation of Chess-Playing Robotic System","volume":"5","author":"Mohammed","year":"2015","journal-title":"Int. J. Sci. Eng. Comput. Technol."},{"key":"ref_33","unstructured":"Banerjee, N., Saha, D., Singh, A., and Sanyal, G. (2012, January 6). A Simple Autonomous Robotic Manipulator for playing Chess against any opponent in Real Time. Proceedings of the International Conference on Computational Vision and Robotics, Bhubaneshwar, India."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/0954-1810(95)00016-X","article-title":"A high school project on artificial intelligence in robotics","volume":"10","author":"Fok","year":"1996","journal-title":"J. Artif. Intell. Eng."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kalra, B. (2022, January 25\u201327). Generalised agent for solving higher board states of tic tac toe using Reinforcement Learning. Proceedings of the 2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC), Solan, India.","DOI":"10.1109\/PDGC56933.2022.10053317"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Spulber, I.A., Doloiu, M.D., Indreica, I., M\u0103ce\u015fanu, G., Sibi\u015fan, B., and Cocia\u015f, T.T. (2025). Real-Time Robotic System for Interactive Tic-Tac-Toe Using Computer Vision. Eng. Proc., 113.","DOI":"10.3390\/engproc2025113052"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Karmanova, E., Serpiva, V., Perminov, S., Ibrahimov, R., Fedoseev, A., and Tsetserukou, D. (2021). SwarmPlay: A Swarm of Nano-Quadcopters Playing Tic-Tac-Toe Board Game against a Human. Proceedings of the ACM SIGGRAPH \u201921 Emerging Technologies, ACM.","DOI":"10.1145\/3450550.3465346"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1109\/LRA.2019.2891311","article-title":"Robot-Assisted Training in Laparoscopy Using Deep Reinforcement Learning","volume":"4","author":"Tan","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Wang, H., Tan, X., Qiu, X., and Qu, C. (2024, January 13\u201317). Subequivariant Reinforcement Learning Framework for Coordinated Motion Control. Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan.","DOI":"10.1109\/ICRA57147.2024.10610563"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.ins.2021.01.077","article-title":"Deep reinforcement learning based moving object grasping","volume":"565","author":"Chen","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_41","unstructured":"Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018, January 29\u201331). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Proceedings of the 2nd Conference on Robot Learning (CoRL), Zurich, Switzerland."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"162202","DOI":"10.1007\/s11432-023-4282-8","article-title":"One model, two skills: Active vision and action learning model for robotic manipulation","volume":"68","author":"Wang","year":"2025","journal-title":"Sci. China Inf. Sci."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"8286","DOI":"10.1038\/s41598-025-93175-2","article-title":"Deep Reinforcement Learning Trajectory Planning for Robotic Manipulator Based on Simulation-Efficient Training","volume":"15","author":"Zhao","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Abdi, A., Ranjbar, M.H., and Park, J.H. (2022). Computer Vision-Based Path Planning for Robot Arms in Three-Dimensional Workspaces Using Q-Learning and Neural Networks. Sensors, 22.","DOI":"10.3390\/s22051697"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Cordova-Cardenas, R., Amor, D., and Guti\u00e9rrez, \u00c1. (2025). Edge AI in Practice: A Survey and Deployment Framework for Neural Networks on Embedded Systems. Electronics, 14.","DOI":"10.3390\/electronics14244877"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Feng, H., Mu, G., Zhong, S., Zhang, P., and Yuan, T. (2022). Benchmark Analysis of YOLO Performance on Edge Intelligence Devices. Cryptography, 6.","DOI":"10.3390\/cryptography6020016"},{"key":"ref_47","unstructured":"Mahmood, A.R., Korenkevych, D., Vasan, G., Ma, W., and Bergstra, J. (2018, January 29\u201331). Benchmarking Reinforcement Learning Algorithms on Real-World Robots. Proceedings of the 2nd Conference on Robot Learning (CoRL), Zurich, Switzerland."},{"key":"ref_48","unstructured":"Cutler, E., Xing, Y., Cui, T., Zhou, B., van Rijnsoever, K., Hart, B., Valencia, D., Ong, L.V.C., Gee, T., and Liarokapis, M. (2024). Benchmarking Reinforcement Learning Methods for Dexterous Robotic Manipulation with a Three-Fingered Gripper. arXiv."},{"key":"ref_49","unstructured":"Adeept (2025, March 23). Adeept 5-DOF Robotic Arm Kit for Raspberry Pi 4 B 3 B+ B A+. Available online: https:\/\/www.adeept.com\/robotic-arm-kit-rpi-black_p0368.html."},{"key":"ref_50","unstructured":"Adafruit (2025, March 18). PCA9685. Available online: https:\/\/cdn-shop.adafruit.com\/datasheets\/PCA9685.pdf."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Lynch, K.M., and Park, F.C. (2017). Modern Robotics: Mechanics, Planning, and Control, Cambridge University Press.","DOI":"10.1017\/9781316661239"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"9243","DOI":"10.1007\/s11042-022-13644-y","article-title":"Object detection using YOLO: Challenges, architectural successors, datasets and applications","volume":"82","author":"Diwan","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Hussain, M. (2023). YOLO-v1 to YOLO-v8, the Rise of YOLO and Its Complementary Nature toward Digital Manufacturing and Industrial Defect Detection. Machines, 11.","DOI":"10.3390\/machines11070677"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Xu, R., Lin, H., Lu, K., Cao, L., and Liu, Y. (2021). A Forest Fire Detection System Based on Ensemble Learning. Forests, 12.","DOI":"10.3390\/f12020217"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Liu, H., Sun, F., Gu, J., and Deng, L. (2022). SF-YOLOv5: A Lightweight Small Object Detection Algorithm Based on Improved Feature Fusion Mode. Sensors, 22.","DOI":"10.3390\/s22155817"},{"key":"ref_57","first-page":"279","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"133653","DOI":"10.1109\/ACCESS.2019.2941229","article-title":"Q-Learning Algorithms: A Comprehensive Classification and Applications","volume":"7","author":"Jang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_59","unstructured":"Lamba, A. (2025, March 15). An Introduction to Q-Learning: Reinforcement Learning. Available online: https:\/\/medium.com\/free-code-camp\/an-introduction-to-q-learning-reinforcement-learning-14ac0b4493cc."},{"key":"ref_60","unstructured":"Nelson, J. (2025, March 20). Roboflow: Computer Vision Tools for Developers and Enterprises. Available online: https:\/\/roboflow.com\/."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"123515","DOI":"10.1109\/ACCESS.2025.3586673","article-title":"A Benchmark Review of YOLO Algorithm Developments for Object Detection","volume":"13","author":"Hua","year":"2025","journal-title":"IEEE Access"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Khanam, R., Asghar, T., and Hussain, M. (2025). Comparative Performance Evaluation of YOLOv5, YOLOv8, and YOLOv11 for Solar Panel Defect Detection. Solar, 5.","DOI":"10.20944\/preprints202501.0788.v1"},{"key":"ref_63","unstructured":"Jorgensen, B. (2025, March 25). Minimax. Available online: https:\/\/beej.us\/blog\/data\/minimax\/."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/15\/3\/50\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T17:22:06Z","timestamp":1772212926000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/15\/3\/50"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,27]]},"references-count":63,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["robotics15030050"],"URL":"https:\/\/doi.org\/10.3390\/robotics15030050","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,27]]}}}