{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T13:59:25Z","timestamp":1778162365366,"version":"3.51.4"},"reference-count":58,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2024,12,1]],"date-time":"2024-12-01T00:00:00Z","timestamp":1733011200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"FSE REACT-EU\u2014PON Ricerca e Innovazione","award":["2014\u20132020"],"award-info":[{"award-number":["2014\u20132020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Italian Sign Language (LIS) is the primary form of communication for many members of the Italian deaf community. Despite being recognized as a fully fledged language with its own grammar and syntax, LIS still faces challenges in gaining widespread recognition and integration into public services, education, and media. In recent years, advancements in technology, including artificial intelligence and machine learning, have opened up new opportunities to bridge communication gaps between the deaf and hearing communities. This paper presents a novel educational tool designed to teach LIS through SIGNIFY, a Machine Learning-based interactive serious game. The game incorporates a tutorial section, guiding users to learn the sign alphabet, and a classic hangman game that reinforces learning through practice. The developed system employs advanced hand gesture recognition techniques for learning and perfecting sign language gestures. The proposed solution detects and overlays 21 hand landmarks and a bounding box on live camera feeds, making use of an open-source framework to provide real-time visual feedback. Moreover, the study compares the effectiveness of two camera systems: the Azure Kinect, which provides RGB-D information, and a standard RGB laptop camera. Results highlight both systems\u2019 feasibility and educational potential, showcasing their respective advantages and limitations. Evaluations with primary school children demonstrate the tool\u2019s ability to make sign language education more accessible and engaging. This article emphasizes the work\u2019s contribution to inclusive education, highlighting the integration of technology to enhance learning experiences for deaf and hard-of-hearing individuals.<\/jats:p>","DOI":"10.3390\/fi16120447","type":"journal-article","created":{"date-parts":[[2024,12,3]],"date-time":"2024-12-03T04:04:04Z","timestamp":1733198644000},"page":"447","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["SIGNIFY: Leveraging Machine Learning and Gesture Recognition for Sign Language Teaching Through a Serious Game"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8407-0660","authenticated-orcid":false,"given":"Luca","family":"Ulrich","sequence":"first","affiliation":[{"name":"Management and Production Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]},{"given":"Giulio","family":"Carmassi","sequence":"additional","affiliation":[{"name":"Biomedical Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]},{"given":"Paolo","family":"Garelli","sequence":"additional","affiliation":[{"name":"Biomedical Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]},{"given":"Gianluca","family":"Lo Presti","sequence":"additional","affiliation":[{"name":"Biomedical Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5940-3806","authenticated-orcid":false,"given":"Gioele","family":"Ramondetti","sequence":"additional","affiliation":[{"name":"Computer Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2359-4409","authenticated-orcid":false,"given":"Giorgia","family":"Marullo","sequence":"additional","affiliation":[{"name":"Management and Production Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7839-6218","authenticated-orcid":false,"given":"Chiara","family":"Innocente","sequence":"additional","affiliation":[{"name":"Management and Production Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]},{"given":"Enrico","family":"Vezzetti","sequence":"additional","affiliation":[{"name":"Management and Production Engineering, Politecnico di Torino, C.so Duca degli Abruzzi, 24, 10129 Torino, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,1]]},"reference":[{"key":"ref_1","unstructured":"Mindess, A. (2004). Reading Between the Signs Workbook: A Cultural Guide for Sign Language Students and Interpreters, Hachette."},{"key":"ref_2","unstructured":"World Health Organization (2015). Deafness and Hearing Loss. Fact Sheet N 300, World Health Organization."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1080\/02667363.2018.1557113","article-title":"Exploring the social inclusion of deaf young people in mainstream schools, using their lived experience","volume":"35","author":"Edmondson","year":"2019","journal-title":"Educ. Psychol. Pract."},{"key":"ref_4","unstructured":"Spence, C. (2024, June 25). How Learning a New Language Changes Your Brain|Cambridge English. Available online: https:\/\/www.cambridge.org\/elt\/blog\/2022\/04\/29\/learning-language-changes-your-brain\/."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1080\/14643154.2024.2341573","article-title":"Inclusive education for deaf students: Pass or fail","volume":"26","author":"Levesque","year":"2024","journal-title":"Deaf. Educ. Int."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Li, Y., Chen, D., and Deng, X. (2024). The impact of digital educational games on student\u2019s motivation for learning: The mediating effect of learning engagement and the moderating effect of the digital environment. PLoS ONE, 19.","DOI":"10.1371\/journal.pone.0294350"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"105825","DOI":"10.1016\/j.chb.2018.12.003","article-title":"An educational game to teach numbers in Brazilian Sign Language while having fun","volume":"107","author":"Pontes","year":"2020","journal-title":"Comput. Hum. Behav."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Chouhan, T., Panse, A., Voona, A.K., and Sameer, S. (2014, January 26\u201327). Smart glove with gesture recognition ability for the hearing and speech impaired. Proceedings of the 2014 IEEE Global Humanitarian Technology Conference-South Asia Satellite (GHTC-SAS), Trivandrum, India.","DOI":"10.1109\/GHTC-SAS.2014.6967567"},{"key":"ref_9","unstructured":"Assaleh, K., Shanableh, T., and Zourob, M. (2012, January 12\u201315). Low complexity classification system for glove-based arabic sign language recognition. Proceedings of the Neural Information Processing: 19th International Conference, ICONIP 2012, Doha, Qatar. Proceedings, Part III 19."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.procs.2015.12.276","article-title":"A New Data Glove Approach for Malaysian Sign Language Detection","volume":"76","author":"Shukor","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"ref_11","unstructured":"Mohandes, M., A-Buraiky, S., Halawani, T., and Al-Baiyat, S. (2004, January 19\u201323). Automation of the Arabic sign language recognition. Proceedings of the 2004 IEEE International Conference on Information and Communication Technologies: From Theory to Applications, Damascus, Syria."},{"key":"ref_12","unstructured":"Hongo, H., Ohya, M., Yasumoto, M., Niwa, Y., and Yamamoto, K. (2000, January 26\u201330). Focus of attention for face and hand gesture recognition using multiple cameras. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), Grenoble, France."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Zhang, H., Wang, Y., and Deng, C. (2011, January 12\u201314). Application of gesture recognition based on simulated annealing BP neural network. Proceedings of the IEEE International Conference on Electronic and Mechanical Engineering and Information Technology, EMEIT 2011, Harbin, China.","DOI":"10.1109\/EMEIT.2011.6022891"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1064","DOI":"10.1109\/TSMCA.2011.2116004","article-title":"A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors","volume":"41","author":"Zhang","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern.-Part A Syst. Hum."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chuan, C.H., Regina, E., and Guardino, C. (2014, January 3\u20136). American sign language recognition using leap motion sensor. Proceedings of the 2014 13th IEEE International Conference on Machine Learning and Applications, Detroit, MI, USA.","DOI":"10.1109\/ICMLA.2014.110"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1581","DOI":"10.1007\/s40747-023-01173-6","article-title":"Computer vision-based hand gesture recognition for human-robot interaction: A review","volume":"10","author":"Qi","year":"2024","journal-title":"Complex Intell. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"e48258","DOI":"10.2196\/48258","article-title":"The role of AI in serious games and gamification for health: Scoping review","volume":"12","author":"Tolks","year":"2024","journal-title":"JMIR Serious Games"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"012078","DOI":"10.1088\/1757-899X\/226\/1\/012078","article-title":"Gesture Recognition for Educational Games: Magic Touch Math","volume":"226","author":"Kye","year":"2017","journal-title":"IOP Conf. Ser. Mater. Sci. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1080\/10494820.2022.2115077","article-title":"A systematic literature review of game-based learning in Artificial Intelligence education","volume":"32","author":"Zhan","year":"2024","journal-title":"Interact. Learn. Environ."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1007\/978-3-642-29347-4_46","article-title":"Sign Language Recognition Using Kinect","volume":"Volume 7267","author":"Rutkowski","year":"2012","journal-title":"Artificial Intelligence and Soft Computing"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Soares, F., Esteves, J.S., Carvalho, V., Lopes, G., Barbosa, F., and Ribeiro, P. (November, January 30). Development of a serious game for Portuguese Sign Language. Proceedings of the 2015 7th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), Ghent, Belgium.","DOI":"10.1109\/ICUMT.2015.7382432"},{"key":"ref_22","unstructured":"Wang, J., Ivrissimtzis, I., Li, Z., and Shi, L. (2024). The Impact of 2D and 3D Gamified VR on Learning American Sign Language. arXiv."},{"key":"ref_23","first-page":"101934","article-title":"Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language","volume":"36","author":"Arooj","year":"2024","journal-title":"J. King Saud Univ.-Comput. Inf. Sci."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"29375","DOI":"10.1007\/s11042-020-09479-0","article-title":"Analysis of RGB-D camera technologies for supporting different facial usage scenarios","volume":"79","author":"Ulrich","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1384","DOI":"10.1016\/j.procs.2023.01.117","article-title":"Real-time Assamese Sign Language Recognition using MediaPipe and Deep Learning","volume":"218","author":"Bora","year":"2023","journal-title":"Procedia Comput. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1177\/00238309231169502","article-title":"Kinect-ing the dots: Using motion-capture technology to distinguish sign language linguistic from gestural expressions","volume":"67","author":"Stamp","year":"2024","journal-title":"Lang. Speech"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Nimisha, K.P., and Jacob, A. (2020, January 28\u201330). A Brief Review of the Recent Trends in Sign Language Recognition. Proceedings of the International Conference on Communication and Signal Processing, Melmaruvathur, India.","DOI":"10.1109\/ICCSP48568.2020.9182351"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Uboweja, E., Tian, D., Wang, Q., Kuo, Y.C., Zou, J., Wang, L., Sung, G., and Grundmann, M. (2023, January 4\u20136). On-device Real-time Custom Hand Gesture Recognition. Proceedings of the 2023 IEEE\/CVF International Conference on Computer Vision Workshops (ICCVW), Paris, France.","DOI":"10.1109\/ICCVW60793.2023.00461"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3275","DOI":"10.11591\/eei.v13i5.7803","article-title":"Application of neural networks ensemble method for the Kazakh sign language recognition","volume":"13","author":"Amirgaliyev","year":"2024","journal-title":"Bull. Electr. Eng. Inform."},{"key":"ref_30","first-page":"4149","article-title":"Recognization of Hand Gestures Using MediaPipe Hands","volume":"4","author":"Kavana","year":"2022","journal-title":"Int. Res. J. Mod. Eng. Technol. Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/978-981-16-2594-7_13","article-title":"American sign language identification using hand trackpoint analysis","volume":"Volume 1","author":"Bajaj","year":"2022","journal-title":"Proceedings of the International Conference on Innovative Computing and Communications: Proceedings of ICICC 2021"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ren Ewe, E.L., Lee, C.P., Lim, K.M., Kwek, L.C., and Alqahtani, A. (2024). LAVRF: Sign language recognition via Lightweight Attentive VGG16 with Random Forest. PLoS ONE, 19.","DOI":"10.1371\/journal.pone.0298699"},{"key":"ref_33","unstructured":"M. Donnici, G.M. (2024, May 18). Italian sign language fingerspelling recognition. Available online: https:\/\/github.com\/maghid\/italian_fingerspelling_recognition."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Fagiani, M., Principi, E., Squartini, S., and Piazza, F. (2012, January 11\u201314). A new Italian sign language database. Proceedings of the Advances in Brain Inspired Cognitive Systems: 5th International Conference, BICS 2012, Shenyang, China. Proceedings 5.","DOI":"10.1007\/978-3-642-31561-9_18"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Escalera, S., Bar\u00f3, X., Gonzalez, J., Bautista, M.A., Madadi, M., Reyes, M., Ponce-L\u00f3pez, V., Escalante, H.J., Shotton, J., and Guyon, I. (September, January 6\u20137,). Chalearn looking at people challenge 2014: Dataset and results. Proceedings of the Computer Vision-ECCV 2014 Workshops, Zurich, Switzerland. Proceedings, Part I 13.","DOI":"10.1007\/978-3-319-16178-5_32"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Romeo, L., Marani, R., Malosio, M., Perri, A.G., and D\u2019Orazio, T. (2021, January 22\u201325). Performance analysis of body tracking with the microsoft azure kinect. Proceedings of the 2021 29th IEEE Mediterranean Conference on Control and Automation (MED), Puglia, Italy.","DOI":"10.1109\/MED51440.2021.9480177"},{"key":"ref_37","unstructured":"Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.L., and Grundmann, M. (2020). Mediapipe hands: On-device real-time hand tracking. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"493","DOI":"10.32628\/CSEIT183821","article-title":"Multiclass Classification Using Random Forest Classifier","volume":"5","author":"Talla","year":"2019","journal-title":"Int. J. Sci. Res. Comput. Sci. Eng. Inf. Technol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Su, R., Chen, X., Cao, S., and Zhang, X. (2016). Random forest-based recognition of isolated sign language subwords using data from accelerometers and surface electromyographic sensors. Sensors, 16.","DOI":"10.3390\/s16010100"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Tang, D., Taylor, J., Kohli, P., Keskin, C., Kim, T.K., and Shotton, J. (2015, January 7\u201313). Opening the black box: Hierarchical sampling optimization for estimating human hand pose. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.380"},{"key":"ref_42","first-page":"237","article-title":"The Comparison of K-Nearest Neighbors and Random Forest Algorithm to Recognize Indonesian Sign Language in a Real-Time","volume":"11","author":"Goenawan","year":"2024","journal-title":"Sci. J. Inform."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016, January 13\u201317). Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_44","first-page":"200056","article-title":"Machine learning methods for sign language recognition: A critical review and analysis","volume":"12","author":"Adeyanju","year":"2021","journal-title":"Intell. Syst. Appl."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Joshi, H., Golhar, V., Gundawar, J., Gangurde, A., Yenkikar, A., and Sable, N.P. (2024, October 13). Real-Time Sign Language Recognition and Sentence Generation. Available at SSRN: Joshi, Harita and Golhar, Vaibhav and Gundawar, Janhavi and Gangurde, Akash and Yenkikar, Anuradha and Sable, Nilesh P, Real-Time Sign Language Recognition and Sentence Generation. Available online: http:\/\/dx.doi.org\/10.2139\/ssrn.4992818.","DOI":"10.2139\/ssrn.4992818"},{"key":"ref_46","unstructured":"Logothetis, I., Papadourakis, G., Katsaris, I., Katsios, K., and Vidakis, N. (September, January 30). Transforming classic learning games with the use of AR: The case of the word hangman game. Proceedings of the International Conference on Human-Computer Interaction, Bari, Italy."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"e19037","DOI":"10.2196\/19037","article-title":"Quality Criteria for Serious Games: Serious Part, Game Part, and Balance","volume":"8","author":"Caserman","year":"2020","journal-title":"JMIR Serious Games"},{"key":"ref_48","unstructured":"Grandini, M., Bagli, E., and Visani, G. (2020). Metrics for multi-class classification: An overview. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1038\/381520a0","article-title":"Speed of processing in the human visual system","volume":"381","author":"Thorpe","year":"1996","journal-title":"Nature"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Violante, M.G., Marcolin, F., Vezzetti, E., Ulrich, L., Billia, G., and Di Grazia, L. (2019). 3D facial expression recognition for defining users\u2019 inner requirements\u2014An emotional design case study. Appl. Sci., 9.","DOI":"10.3390\/app9112218"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Ulrich, L., Dugelay, J.L., Vezzetti, E., Moos, S., and Marcolin, F. (2019). Perspective morphometric criteria for facial beauty and proportion assessment. Appl. Sci., 10.","DOI":"10.3390\/app10010008"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Malvasi, V., Gil-Quintana, J., and Bocciolesi, E. (2022). The Projection of Gamification and Serious Games in the Learning of Mathematics Multi-Case Study of Secondary Schools in Italy. Mathematics, 10.","DOI":"10.3390\/math10030336"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1080\/15391523.2020.1867938","article-title":"Perceived usability evaluation of educational technology using the System Usability Scale (SUS): A systematic review","volume":"54","author":"Vlachogianni","year":"2021","journal-title":"J. Res. Technol. Educ."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1016\/j.ipm.2015.03.003","article-title":"An empirical evaluation of the User Engagement Scale (UES) in online news environments","volume":"51","author":"Cairns","year":"2015","journal-title":"Inf. Process. Manag."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Dama\u0161evi\u010dius, R., Maskeli\u016bnas, R., and Bla\u017eauskas, T. (2023). Serious games and gamification in healthcare: A meta-review. Information, 14.","DOI":"10.3390\/info14020105"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Freire, M., Serrano-Laguna, \u00c1., Manero Iglesias, B., Mart\u00ednez-Ortiz, I., Moreno-Ger, P., and Fern\u00e1ndez-Manj\u00f3n, B. (2023). Game learning analytics: Learning analytics for serious games. Learning, Design, and Technology: An International Compendium of Theory, Research, Practice, and Policy, Springer.","DOI":"10.1007\/978-3-319-17461-7_21"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"785","DOI":"10.1007\/s11831-019-09384-2","article-title":"Sign Language Recognition Systems: A Decade Systematic Literature Review","volume":"28","author":"Wadhawan","year":"2021","journal-title":"Arch. Comput. Methods Eng."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Bustamante, A., Belmonte, L.M., Morales, R., Pereira, A., and Fern\u00e1ndez-Caballero, A. (2022). Video Processing from a Virtual Unmanned Aerial Vehicle: Comparing Two Approaches to Using OpenCV in Unity. Appl. Sci., 12.","DOI":"10.3390\/app12125958"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/12\/447\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:44:05Z","timestamp":1760114645000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/12\/447"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,1]]},"references-count":58,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["fi16120447"],"URL":"https:\/\/doi.org\/10.3390\/fi16120447","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,1]]}}}