{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T15:37:14Z","timestamp":1778081834525,"version":"3.51.4"},"reference-count":29,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2023,10,25]],"date-time":"2023-10-25T00:00:00Z","timestamp":1698192000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Portugal 2020, under the Competitiveness and Internationalization Operational Program (POCI)","award":["POCI-01-0247-FEDER-068605"],"award-info":[{"award-number":["POCI-01-0247-FEDER-068605"]}]},{"name":"Portugal 2020, under the Competitiveness and Internationalization Operational Program (POCI)","award":["C644866286-00000011"],"award-info":[{"award-number":["C644866286-00000011"]}]},{"name":"Portugal 2020, under the Competitiveness and Internationalization Operational Program (POCI)","award":["UIDB\/04033\/2020"],"award-info":[{"award-number":["UIDB\/04033\/2020"]}]},{"name":"Portugal 2020, under the Competitiveness and Internationalization Operational Program (POCI)","award":["LA\/P\/0126\/2020"],"award-info":[{"award-number":["LA\/P\/0126\/2020"]}]},{"name":"RRP\u2014Recovery and Resilience Plan and the European NextGeneration EU Funds","award":["POCI-01-0247-FEDER-068605"],"award-info":[{"award-number":["POCI-01-0247-FEDER-068605"]}]},{"name":"RRP\u2014Recovery and Resilience Plan and the European NextGeneration EU Funds","award":["C644866286-00000011"],"award-info":[{"award-number":["C644866286-00000011"]}]},{"name":"RRP\u2014Recovery and Resilience Plan and the European NextGeneration EU Funds","award":["UIDB\/04033\/2020"],"award-info":[{"award-number":["UIDB\/04033\/2020"]}]},{"name":"RRP\u2014Recovery and Resilience Plan and the European NextGeneration EU Funds","award":["LA\/P\/0126\/2020"],"award-info":[{"award-number":["LA\/P\/0126\/2020"]}]},{"name":"National Funds from the FCT-Portuguese Foundation for Science and Technology","award":["POCI-01-0247-FEDER-068605"],"award-info":[{"award-number":["POCI-01-0247-FEDER-068605"]}]},{"name":"National Funds from the FCT-Portuguese Foundation for Science and Technology","award":["C644866286-00000011"],"award-info":[{"award-number":["C644866286-00000011"]}]},{"name":"National Funds from the FCT-Portuguese Foundation for Science and Technology","award":["UIDB\/04033\/2020"],"award-info":[{"award-number":["UIDB\/04033\/2020"]}]},{"name":"National Funds from the FCT-Portuguese Foundation for Science and Technology","award":["LA\/P\/0126\/2020"],"award-info":[{"award-number":["LA\/P\/0126\/2020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Communication between Deaf and hearing individuals remains a persistent challenge requiring attention to foster inclusivity. Despite notable efforts in the development of digital solutions for sign language recognition (SLR), several issues persist, such as cross-platform interoperability and strategies for tokenizing signs to enable continuous conversations and coherent sentence construction. To address such issues, this paper proposes a non-invasive Portuguese Sign Language (L\u00edngua Gestual Portuguesa or LGP) interpretation system-as-a-service, leveraging skeletal posture sequence inference powered by long-short term memory (LSTM) architectures. To address the scarcity of examples during machine learning (ML) model training, dataset augmentation strategies are explored. Additionally, a buffer-based interaction technique is introduced to facilitate LGP terms tokenization. This technique provides real-time feedback to users, allowing them to gauge the time remaining to complete a sign, which aids in the construction of grammatically coherent sentences based on inferred terms\/words. To support human-like conditioning rules for interpretation, a large language model (LLM) service is integrated. Experiments reveal that LSTM-based neural networks, trained with 50 LGP terms and subjected to data augmentation, achieved accuracy levels ranging from 80% to 95.6%. Users unanimously reported a high level of intuition when using the buffer-based interaction strategy for terms\/words tokenization. Furthermore, tests with an LLM\u2014specifically ChatGPT\u2014demonstrated promising semantic correlation rates in generated sentences, comparable to expected sentences.<\/jats:p>","DOI":"10.3390\/jimaging9110235","type":"journal-article","created":{"date-parts":[[2023,10,26]],"date-time":"2023-10-26T02:40:07Z","timestamp":1698288007000},"page":"235","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Empowering Deaf-Hearing Communication: Exploring Synergies between Predictive and Generative AI-Based Strategies towards (Portuguese) Sign Language Interpretation"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2727-0014","authenticated-orcid":false,"given":"Telmo","family":"Ad\u00e3o","sequence":"first","affiliation":[{"name":"Department of Engineering, School of Sciences and Technology, University of Tr\u00e1s-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal"},{"name":"ALGORITMI Research Centre\/LASI, University of Minho, 4800-058 Guimar\u00e3es, Portugal"}]},{"given":"Jo\u00e3o","family":"Oliveira","sequence":"additional","affiliation":[{"name":"Centro de Computa\u00e7\u00e3o Gr\u00e1fica-CCG\/zgdv, University of Minho, Campus de Azur\u00e9m, Edif\u00edcio 14, 4800-058 Guimar\u00e3es, Portugal"}]},{"given":"Somayeh","family":"Shahrabadi","sequence":"additional","affiliation":[{"name":"Centro de Computa\u00e7\u00e3o Gr\u00e1fica-CCG\/zgdv, University of Minho, Campus de Azur\u00e9m, Edif\u00edcio 14, 4800-058 Guimar\u00e3es, Portugal"}]},{"given":"Hugo","family":"Jesus","sequence":"additional","affiliation":[{"name":"Centro de Computa\u00e7\u00e3o Gr\u00e1fica-CCG\/zgdv, University of Minho, Campus de Azur\u00e9m, Edif\u00edcio 14, 4800-058 Guimar\u00e3es, Portugal"}]},{"given":"Marco","family":"Fernandes","sequence":"additional","affiliation":[{"name":"Polytechnic Institute of Bragan\u00e7a, School of Communication, Administration and Tourism, Campus do Cruzeiro, 5370-202 Mirandela, Portugal"}]},{"given":"\u00c2ngelo","family":"Costa","sequence":"additional","affiliation":[{"name":"Associa\u00e7\u00e3o Portuguesa de Surdos (APS), 1600-796 Lisboa, Portugal"}]},{"given":"V\u00e2nia","family":"Ferreira","sequence":"additional","affiliation":[{"name":"Associa\u00e7\u00e3o Portuguesa de Surdos (APS), 1600-796 Lisboa, Portugal"}]},{"given":"Martinho","family":"Gon\u00e7alves","sequence":"additional","affiliation":[{"name":"Polytechnic Institute of Bragan\u00e7a, School of Communication, Administration and Tourism, Campus do Cruzeiro, 5370-202 Mirandela, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7814-1653","authenticated-orcid":false,"given":"Miguel","family":"Lop\u00e9z","sequence":"additional","affiliation":[{"name":"ALGORITMI Research Centre\/LASI, University of Minho, 4800-058 Guimar\u00e3es, Portugal"},{"name":"Instituto Polit\u00e9cnico de Set\u00fabal, Escola Superior de Tecnologia de Set\u00fabal, 2914-508 Set\u00fabal, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5669-7976","authenticated-orcid":false,"given":"Emanuel","family":"Peres","sequence":"additional","affiliation":[{"name":"Department of Engineering, School of Sciences and Technology, University of Tr\u00e1s-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal"},{"name":"Centre for the Research and Technology of Agro-Environmental and Biological Sciences, University of Tr\u00e1s-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal"},{"name":"Institute for Innovation, Capacity Building and Sustainability of Agri-Food Production, University of Tr\u00e1s-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal"}]},{"given":"Lu\u00eds","family":"Magalh\u00e3es","sequence":"additional","affiliation":[{"name":"ALGORITMI Research Centre\/LASI, University of Minho, 4800-058 Guimar\u00e3es, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1016\/j.procs.2015.09.269","article-title":"Virtual Sign\u2014A Real Time Bidirectional Translator of Portuguese Sign Language","volume":"67","author":"Escudeiro","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"ref_2","unstructured":"Mayea, C., Garcia, D., Guevara Lopez, M.A., Peres, E., Magalh\u00e3es, L., and Ad\u00e3o, T. (2022, January 3\u20134). Building Portuguese Sign Language Datasets for Computational Learning Purposes. Proceedings of the 2022 International Conference on Graphics and Interaction (ICGI), Aveiro, Portugal."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Podder, K.K., Chowdhury, M.E.H., Tahir, A.M., Mahbub, Z.B., Khandakar, A., Hossain, M.S., and Kadir, M.A. (2022). Bangla Sign Language (BdSL) Alphabets and Numerals Classification Using a Deep Learning Model. Sensors, 22.","DOI":"10.3390\/s22020574"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Abraham, E., Nayak, A., and Iqbal, A. (2019, January 18\u201320). Real-Time Translation of Indian Sign Language Using LSTM. Proceedings of the 2019 Global Conference for Advancement in Technology (GCAT), Bangaluru, India.","DOI":"10.1109\/GCAT47503.2019.8978343"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"723","DOI":"10.1080\/03772063.2020.1838342","article-title":"Vision-Based Hand Gesture Recognition for Indian Sign Language Using Convolution Neural Network","volume":"69","author":"Gangrade","year":"2023","journal-title":"IETE J. Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"4517","DOI":"10.1109\/ACCESS.2017.2684186","article-title":"Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM","volume":"5","author":"Zhu","year":"2017","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bohacek, M., and Hruz, M. (2022, January 4\u20138). Sign Pose-Based Transformer for Word-Level Sign Language Recognition. Proceedings of the 2022 IEEE\/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), Waikoloa, HI, USA.","DOI":"10.1109\/WACVW54805.2022.00024"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1007\/s12652-021-02920-8","article-title":"Real-Time Isolated Hand Sign Language Recognition Using Deep Networks and SVD","volume":"13","author":"Rastgoo","year":"2022","journal-title":"J. Ambient. Intell. Hum. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"216718","DOI":"10.1109\/ACCESS.2020.3039539","article-title":"Multi-Information Spatial\u2013Temporal LSTM Fusion Continuous Sign Language Neural Machine Translation","volume":"8","author":"Xiao","year":"2020","journal-title":"IEEE Access"},{"key":"ref_10","unstructured":"(2023, August 03). SLAIT\u2014Real-Time Sign Language Translator with AI. Available online: https:\/\/slait.ai."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Li, D., Opazo, C.R., Yu, X., and Li, H. (2020, January 1\u20135). Word-Level Deep Sign Language Recognition from Video: A New Large-Scale Dataset and Methods Comparison. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA.","DOI":"10.1109\/WACV45572.2020.9093512"},{"key":"ref_12","unstructured":"Ronchetti, F., Quiroga, F., Estrebou, C., Lanzarini, L., and Rosete, A. (2016, January 3\u20137). LSA64: An Argentinian Sign Language Dataset. Proceedings of the XXII Congreso Argentino de Ciencias de la Computaci\u00f3n (CACIC), San Luis, Argentina."},{"key":"ref_13","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_14","unstructured":"OpenAI (2023, August 11). GPT-4 Technical Report. Available online: https:\/\/cdn.openai.com\/papers\/gpt-4.pdf."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Singh, A., Wadhawan, A., Rakhra, M., Mittal, U., Ahdal, A.A., and Jha, S.K. (2022, January 13\u201314). Indian Sign Language Recognition System for Dynamic Signs. Proceedings of the 2022 10th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India.","DOI":"10.1109\/ICRITO56286.2022.9964940"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sridhar, A., Ganesan, R.G., Kumar, P., and Khapra, M. (2020, January 12). INCLUDE: A Large Scale Dataset for Indian Sign Language Recognition. Proceedings of the 28th ACM International Conference on Multimedia, Association for Computing Machinery, New York, NY, USA.","DOI":"10.1145\/3394171.3413528"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1016\/j.cviu.2015.09.013","article-title":"Continuous Sign Language Recognition: Towards Large Vocabulary Statistical Recognition Systems Handling Multiple Signers","volume":"141","author":"Koller","year":"2015","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Duarte, A., Palaskar, S., Ventura, L., Ghadiyaram, D., DeHaan, K., Metze, F., Torres, J., and Giro-i-Nieto, X. (2021, January 20\u201325). How2Sign: A Large-Scale Multimodal Dataset for Continuous American Sign Language. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00276"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"114179","DOI":"10.1016\/j.eswa.2020.114179","article-title":"A Multimodal LIBRAS-UFOP Brazilian Sign Language Dataset of Minimal Pairs Using a Microsoft Kinect Sensor","volume":"167","author":"Cerna","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Brashear, H., Henderson, V., Park, K.-H., Hamilton, H., Lee, S., and Starner, T. (2006, January 23). American Sign Language Recognition in Game Development for Deaf Children. Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Association for Computing Machinery, New York, NY, USA.","DOI":"10.1145\/1168987.1169002"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Blevins, T., Gonen, H., and Zettlemoyer, L. (2023, January 9\u201314). Prompting Language Models for Linguistic Structure. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, ON, Canada.","DOI":"10.18653\/v1\/2023.acl-long.367"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Chang, E.Y. (2023, January 8\u201311). Prompting Large Language Models with the Socratic Method. Proceedings of the 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.","DOI":"10.1109\/CCWC57344.2023.10099179"},{"key":"ref_23","unstructured":"Zhang, B., Haddow, B., and Birch, A. (2023). Prompting Large Language Model for Machine Translation: A Case Study. arXiv."},{"key":"ref_24","unstructured":"Wei, J., Kim, S., Jung, H., and Kim, Y.-H. (2023). Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data. arXiv."},{"key":"ref_25","unstructured":"(2023, August 07). Infop\u00e9dia Infopedia.pt\u2014Dicion\u00e1rios Porto Editora. Available online: https:\/\/www.infopedia.pt\/dicionarios\/lingua-gestual."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Chowdhary, K.R. (2020). Fundamentals of Artificial Intelligence, Springer.","DOI":"10.1007\/978-81-322-3972-7"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Alves, V., Ribeiro, J., Romero, L., Faria, P.M., Costa, \u00c2., and Ferreira, V. (2023, January 27\u201329). A Gloss Based Translation From European Portuguese to Portuguese Sign Language. Proceedings of the 2023 30th International Conference on Systems, Signals and Image Processing (IWSSIP), Ohrid, North Macedonia.","DOI":"10.1109\/IWSSIP58668.2023.10180304"},{"key":"ref_28","unstructured":"(2023, August 11). Introducing ChatGPT. Available online: https:\/\/openai.com\/blog\/chatgpt."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cer, D., Yang, Y., Kong, S., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-Cespedes, M., Yuan, S., and Tar, C. (2018). Universal Sentence Encoder. arXiv.","DOI":"10.18653\/v1\/D18-2029"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/11\/235\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:11:50Z","timestamp":1760130710000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/11\/235"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,25]]},"references-count":29,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,11]]}},"alternative-id":["jimaging9110235"],"URL":"https:\/\/doi.org\/10.3390\/jimaging9110235","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,25]]}}}