{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T05:13:59Z","timestamp":1772169239121,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,6,8]],"date-time":"2025-06-08T00:00:00Z","timestamp":1749340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Electronics"],"abstract":"<jats:p>Communication barriers between deaf and hearing individuals present significant challenges to social inclusion, highlighting the need for effective technological aids. This study aimed to bridge this gap by developing a mobile system for the real-time translation of Portuguese Sign Language (LGP) alphabet gestures into text, addressing a specific technological void for LGP. The core of the solution is a mobile application integrating two distinct machine learning approaches trained on a custom LGP dataset: firstly, a Convolutional Neural Network (CNN) optimized with TensorFlow Lite for efficient, on-device image classification, enabling offline use; secondly, a method utilizing MediaPipe for hand landmark extraction from the camera feed, with classification performed by a server-side Multilayer Perceptron (MLP). Evaluation tests confirmed that both approaches could recognize LGP alphabet gestures with good accuracy (F1-scores of approximately 76% for the CNN and 77% for the MediaPipe+MLP) and processing speed (1 to 2 s per gesture on high-end devices using the CNN and 3 to 5 s under typical network conditions using MediaPipe+MLP), facilitating efficient real-time translation, though performance trade-offs regarding speed versus accuracy under different conditions were observed. The implementation of this dual-method system provides crucial flexibility, adapting to varying network conditions and device capabilities, and offers a scalable foundation for future expansion to include more complex gestures. This work delivers a practical tool that may contribute to improve communication accessibility and the societal integration of the deaf community in Portugal.<\/jats:p>","DOI":"10.3390\/electronics14122351","type":"journal-article","created":{"date-parts":[[2025,6,9]],"date-time":"2025-06-09T05:54:05Z","timestamp":1749448445000},"page":"2351","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Real-Time Mobile Application for Translating Portuguese Sign Language to Text Using Machine Learning"],"prefix":"10.3390","volume":"14","author":[{"given":"Gon\u00e7alo","family":"Fonseca","sequence":"first","affiliation":[{"name":"Lisbon School of Engineering (ISEL), Polytechnic University of Lisbon (IPL), 1959-007 Lisboa, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3674-0145","authenticated-orcid":false,"given":"Gon\u00e7alo","family":"Marques","sequence":"additional","affiliation":[{"name":"Lisbon School of Engineering (ISEL), Polytechnic University of Lisbon (IPL), 1959-007 Lisboa, Portugal"},{"name":"NOVA LINCS, ISEL, 1959-007 Lisboa, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8159-1364","authenticated-orcid":false,"given":"Pedro","family":"Albuquerque Santos","sequence":"additional","affiliation":[{"name":"NOVA LINCS, ISEL, 1959-007 Lisboa, Portugal"},{"name":"School of Business Administration (ESCE), Polytechnic University of Set\u00fabal (IPS), 2914-503 Set\u00fabal, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1869-6491","authenticated-orcid":false,"given":"Rui","family":"Jesus","sequence":"additional","affiliation":[{"name":"Lisbon School of Engineering (ISEL), Polytechnic University of Lisbon (IPL), 1959-007 Lisboa, Portugal"},{"name":"NOVA LINCS, ISEL, 1959-007 Lisboa, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,8]]},"reference":[{"key":"ref_1","unstructured":"Lucas, C. (2001). Sociolinguistic Variation in American Sign Language, Cambridge University Press."},{"key":"ref_2","unstructured":"Klima, E.S., and Bellugi, U. (1979). Signs of Language, Harvard University Press."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1093\/deafed\/eni001","article-title":"Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf","volume":"10","author":"Stokoe","year":"2005","journal-title":"J. Deaf. Stud. Deaf. Educ."},{"key":"ref_4","unstructured":"Assembleia da Rep\u00fablica (2024, July 22). Lei Constitucional n.\u00ba 1\/97, de 20 de Setembro. Point (h) of Paragraph 2 of Article 74. Available online: https:\/\/diariodarepublica.pt\/dr\/detalhe\/lei-constitucional\/1-653562."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Pfau, R., Steinbach, M., and Woll, B. (2012). Sign Language: An International Handbook, De Gruyter.","DOI":"10.1515\/9783110261325"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1016\/0031-3203(88)90048-9","article-title":"Recognition of sign language motion images","volume":"21","author":"Tamura","year":"1988","journal-title":"Pattern Recognit."},{"key":"ref_7","unstructured":"Tanibata, N., Shimada, N., and Shirai, Y. (2002, January 8\u201310). Extraction of Hand Features for Recognition of Sign Language. Proceedings of the International Conference on Vision Interface, New York, NY, USA."},{"key":"ref_8","unstructured":"Guerin, G. (2024, July 22). Sign Language Recognition\u2014Using MediaPipe & DTW. Available online: https:\/\/data-ai.theodo.com\/en\/technical-blog\/sign-language-recognition-using-mediapipe."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2719","DOI":"10.1007\/s10586-017-1435-x","article-title":"Hand gesture recognition based on convolution neural network","volume":"22","author":"Li","year":"2019","journal-title":"Clust. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"91170","DOI":"10.1109\/ACCESS.2020.2993650","article-title":"Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-Latent Space","volume":"8","author":"Papastratis","year":"2020","journal-title":"IEEE Access"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Cihan Camgoz, N., Koller, O., Hadfield, S., and Bowden, R. (2020, January 14\u201319). Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.","DOI":"10.1109\/CVPR42600.2020.01004"},{"key":"ref_12","first-page":"9","article-title":"Real-time Vernacular Sign Language Recognition using MediaPipe and Machine Learning","volume":"2","author":"Tayade","year":"2021","journal-title":"Int. J. Res. Publ. Rev."},{"key":"ref_13","unstructured":"Harris, M., and Agoes, A.S. (2021, January 12). Applying Hand Gesture Recognition for User Guide Application Using MediaPipe. Proceedings of the 2nd International Seminar of Science and Applied Technology (ISSAT 2021), Online."},{"key":"ref_14","unstructured":"Feng, Q., Yang, C., Wu, X., and Li, Z. (2013, January 20\u201322). A smart TV interaction system based on hand gesture recognition by using RGB-D Sensor. Proceedings of the 2013 International Conference on Mechatronic Sciences, Electric Engineering and Computer (MEC), Shengyang, China."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Mora-Zarate, J.E., Garz\u00f3n-Castro, C.L., and Castellanos Rivillas, J.A. (2024, January 13\u201315). Construction and Evaluation of a Dynamic Sign Dataset for the Colombian Sign Language. Proceedings of the 2024 IEEE Latin American Conference on Computational Intelligence (LA-CCI), Bogota, Colombia.","DOI":"10.1109\/LA-CCI62337.2024.10814896"},{"key":"ref_16","unstructured":"Neiva, I.G.S. (2014). Desenvolvimento de um Tradutor de L\u00edngua Gestual Portuguesa. [Master\u2019s Thesis, NOVA School of Science and Technology, NOVA University Lisbon]. Available online: http:\/\/hdl.handle.net\/10362\/14753."},{"key":"ref_17","unstructured":"Ribeiro, P.R. (2019). Sistema de Reconhecimento de L\u00edngua Gestual Portuguesa Recorrendo \u00e0 Kinect. [Master\u2019s Thesis, School of Engineering, University of Minho]. Available online: https:\/\/hdl.handle.net\/1822\/72165."},{"key":"ref_18","unstructured":"Oliveira, O.R.S.A. (2013). Tradutor da L\u00edngua Gestual Portuguesa Modelo de Tradu\u00e7\u00e3o Bidireccional. [Master\u2019s Thesis, ISEP\u2014Porto School of Engineering, Polytechnic University of Porto]. Available online: http:\/\/hdl.handle.net\/10400.22\/6246."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"6999","DOI":"10.1109\/TNNLS.2021.3084827","article-title":"A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects","volume":"33","author":"Li","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kumar, A., and Mozar, S. (2021). CNN Model for American Sign Language Recognition. Proceedings of the ICCCE 2020, Springer.","DOI":"10.1007\/978-981-15-7961-5"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/JPROC.2020.3004555","article-title":"A Comprehensive Survey on Transfer Learning","volume":"109","author":"Zhuang","year":"2021","journal-title":"Proc. IEEE"},{"key":"ref_23","unstructured":"TecPerson (2024, July 22). Sign Language MNIST: Drop-In Replacement for MNIST for Hand Gesture Recognition Tasks. Available online: https:\/\/www.kaggle.com\/datasets\/datamunge\/sign-language-mnist."},{"key":"ref_24","unstructured":"Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., and Lee, J. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv."},{"key":"ref_25","unstructured":"Google AI (2024, July 22). MediaPipe Gesture Recognizer. Available online: https:\/\/ai.google.dev\/edge\/mediapipe\/solutions\/vision\/gesture_recognizer."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kruse, R., Mostaghim, S., Borgelt, C., Braune, C., and Steinbrecher, M. (2022). Multi-layer Perceptrons. Computational Intelligence: A Methodological Introduction, Springer International Publishing.","DOI":"10.1007\/978-3-030-42227-1"},{"key":"ref_27","unstructured":"Takahashi, S. (2024, July 22). Hand Gesture Recognition Using MediaPipe. Available online: https:\/\/github.com\/Kazuhito00\/hand-gesture-recognition-using-mediapipe."},{"key":"ref_28","unstructured":"Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.L., and Grundmann, M. (2020). MediaPipe Hands: On-device Real-time Hand Tracking. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1109\/TIT.1967.1053964","article-title":"Nearest neighbor pattern classification","volume":"13","author":"Cover","year":"1967","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1109\/TKDE.2008.239","article-title":"Learning from Imbalanced Data","volume":"21","author":"He","year":"2009","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.neunet.2018.07.011","article-title":"A systematic study of the class imbalance problem in convolutional neural networks","volume":"106","author":"Buda","year":"2018","journal-title":"Neural Netw."},{"key":"ref_33","first-page":"37","article-title":"Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation","volume":"2","author":"Powers","year":"2011","journal-title":"J. Mach. Learn. Technol."},{"key":"ref_34","unstructured":"Fahey, O., and Lexset (2024, July 22). Synthetic ASL Alphabet. Available online: https:\/\/www.kaggle.com\/datasets\/lexset\/synthetic-asl-alphabet\/."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201323). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_36","unstructured":"Google (2024, July 22). Flutter MediaPipe. Available online: https:\/\/github.com\/google\/flutter-mediapipe."}],"container-title":["Electronics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-9292\/14\/12\/2351\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:48:32Z","timestamp":1760032112000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-9292\/14\/12\/2351"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,8]]},"references-count":36,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["electronics14122351"],"URL":"https:\/\/doi.org\/10.3390\/electronics14122351","relation":{},"ISSN":["2079-9292"],"issn-type":[{"value":"2079-9292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,8]]}}}