{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T00:47:45Z","timestamp":1779929265874,"version":"3.53.1"},"reference-count":32,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2023,5,31]],"date-time":"2023-05-31T00:00:00Z","timestamp":1685491200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Universiti Tun Hussein Onn Malaysia","award":["vot H917"],"award-info":[{"award-number":["vot H917"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>This research proposes a study on two-way communication between deaf\/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf\/mute and normal people, especially by using Bahasa Isyarat Malaysia (BIM). This project consists of three parts: First, we use BIM letters, which enables the recognition of BIM letters and BIM combined letters to form a word. In this part, a MobileNet pre-trained model is implemented to train the model with a total of 87,000 images for 29 classes, with a 10% test size and a 90% training size. The second part is BIM word hand gestures, which consists of five classes that are trained with the SSD-MobileNet-V2 FPNLite 320 \u00d7 320 pre-trained model with a speed of 22 s\/frame rate and COCO mAP of 22.2, with a total of 500 images for all five classes and first-time training set to 2000 steps, while the second- and third-time training are set to 2500 steps. The third part is Android application development using Android Studio, which contains the features of the BIM letters and BIM word hand gestures, with the trained models converted into TensorFlow Lite. This feature also includes the conversion of speech to text, whereby this feature allows converting speech to text through the Android application. Thus, BIM letters obtain 99.75% accuracy after training the models, while BIM word hand gestures obtain 61.60% accuracy. The suggested system is validated as a result of these simulations and tests.<\/jats:p>","DOI":"10.3390\/info14060319","type":"journal-article","created":{"date-parts":[[2023,5,31]],"date-time":"2023-05-31T07:21:51Z","timestamp":1685517711000},"page":"319","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP"],"prefix":"10.3390","volume":"14","author":[{"given":"Iffah Zulaikha","family":"Saiful Bahri","sequence":"first","affiliation":[{"name":"Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2706-3258","authenticated-orcid":false,"given":"Sharifah","family":"Saon","sequence":"additional","affiliation":[{"name":"Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3985-6549","authenticated-orcid":false,"given":"Abd Kadir","family":"Mahamad","sequence":"additional","affiliation":[{"name":"Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Khalid","family":"Isa","sequence":"additional","affiliation":[{"name":"Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Umi","family":"Fadlilah","sequence":"additional","affiliation":[{"name":"Teknik Elektro, Fakultas Teknik, Kampus 2, Universitas Muhammadiyah Surakarta (UMS), Jl. Ahmad Yani, Tromol Pos 1, Surakarta 57169, Jawa Tengah, Indonesia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0952-7151","authenticated-orcid":false,"given":"Mohd Anuaruddin Bin","family":"Ahmadon","sequence":"additional","affiliation":[{"name":"Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube 755-8611, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0579-8501","authenticated-orcid":false,"given":"Shingo","family":"Yamaguchi","sequence":"additional","affiliation":[{"name":"Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube 755-8611, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ahire, P.G., Tilekar, K.B., Jawake, T.A., and Warale, P.B. (2015, January 26\u201327). Two Way Communicator between Deaf and Dumb People and Normal People. Proceedings of the 2015 International Conference on Computing Communication Control and Automation, Pune, India.","DOI":"10.1109\/ICCUBEA.2015.131"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kanvinde, A., Revadekar, A., Tamse, M., Kalbande, D.R., and Bakereywala, N. (2021, January 25\u201327). Bidirectional Sign Language Translation. Proceedings of the 2021 International Conference on Communication information and Computing Technology (ICCICT), Mumbai, India.","DOI":"10.1109\/ICCICT50803.2021.9510146"},{"key":"ref_3","first-page":"5226","article-title":"Application for Iraqi sign language translation","volume":"10","author":"Alobaidy","year":"2020","journal-title":"Int. J. Electr. Comput. Eng. (IJECE)"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Dewasurendra, D., Kumar, A., Perera, I., Jayasena, D., and Thelijjagoda, S. (2020, January 15). Emergency Communication Application for Speech and Hearing-Impaired Citizens. Proceedings of the 2020 From Innovation to Impact (FITI), Colombo, Sri Lanka.","DOI":"10.1109\/FITI52050.2020.9424899"},{"key":"ref_5","first-page":"928","article-title":"A Real Time Visual-Audio Translator for Disabled People to Communicate Using Human-Computer Interface System","volume":"8","author":"Patil","year":"2021","journal-title":"Int. Res. J. Eng. Technol. (IRJET)"},{"key":"ref_6","unstructured":"Mazlina, A.M., Masrulehsan, M., and Ruzaini, A.A. (2014, January 28\u201329). MCMSL Translator: Malaysian Text Translator for Manually Coded Malay Sign Language. Proceedings of the IEEE Symposium on Computers & Informatics (ISCI 2014), Kota Kinabalu, Sabah."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mahyuddin, N.M., Mat Noor, N.R., and Mat Sakim, H.A. (2022). Proceedings of the 11th International Conference on Robotics Vision, Signal Processing and Power Applications, Penang, Malaysia 5\u20136 April 2021, Springer. Lecture Notes in Electrical Engineering.","DOI":"10.1007\/978-981-16-8129-5"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"651","DOI":"10.35940\/ijrte.B1102.0982S1119","article-title":"A Real Time Malaysian Sign Language Detection Algorithm Based on YOLOv3","volume":"8","author":"Asri","year":"2019","journal-title":"Int. J. Recent Technol. Eng. (IJRTE)"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"459","DOI":"10.4314\/jfas.v9i4S.26","article-title":"Malaysian sign language dataset for automatic sign language recognition system","volume":"9","author":"Karbasi","year":"2017","journal-title":"J. Fundam. Appl. Sci."},{"key":"ref_10","first-page":"90","article-title":"Communication between Deaf-Dumb People and Normal People: Chat Assist","volume":"7","author":"Jayatilake","year":"2017","journal-title":"Int. J. Sci. Res. Publ."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yugopuspito, P., Murwantara, I.M., and Sean, J. (2018, January 19\u201321). Mobile Sign Language Recognition for Bahasa Indonesia using Convolutional Neural Network. Proceedings of the 16th International Conference on Advances in Mobile Computing and Multimedia, (MoMM2018), Yogyakarta, Indonesia.","DOI":"10.1145\/3282353.3282356"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"181340","DOI":"10.1109\/ACCESS.2020.3028072","article-title":"AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods","volume":"8","author":"Sincan","year":"2020","journal-title":"IEEE Access"},{"key":"ref_13","first-page":"15","article-title":"Development of Malaysian Sign Language in Malaysia","volume":"8","author":"Yee","year":"2018","journal-title":"J. Spec. Needs Educ."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"012004","DOI":"10.1088\/1742-6596\/1860\/1\/012004","article-title":"A mobile learning application for Malaysian sign language education","volume":"1860","author":"Siong","year":"2021","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Sahid, A.F., Ismail, W.S., and Ghani, D.A. (2016, January 16\u201317). Malay Sign Language (MSL) for Beginner using android application. Proceedings of the 2016 International Conference on Information and Communication Technology (ICICTM), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ICICTM.2016.7890798"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"5512","DOI":"10.11591\/ijece.v9i6.pp5512-5518","article-title":"Malaysian sign language mobile learning application: A recommendation app to communicate with hearing-impaired communities","volume":"9","author":"Hafit","year":"2019","journal-title":"Int. J. Electr. Comput. Eng. (IJECE)"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2271","DOI":"10.22214\/ijraset.2021.36841","article-title":"Conversation Engine for Deaf and Dumb","volume":"9","author":"Monika","year":"2021","journal-title":"Int. J. Res. Appl. Sci. Eng. Technol. (IJRASET)"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"29","DOI":"10.21037\/mhealth.2020.01.04","article-title":"Design suggestions for an mHealth app to facilitate communication between pharmacists and the Deaf: Perspective of the Deaf community (HEARD Project)","volume":"7","author":"Jacob","year":"2021","journal-title":"mHealth"},{"key":"ref_19","first-page":"246","article-title":"Sign Language Translator","volume":"29","author":"Mishra","year":"2020","journal-title":"Int. J. Adv. Sci. Technol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"573555","DOI":"10.3389\/fpsyg.2020.573555","article-title":"Gesture Use and Processing: A Review on Individual Differences in Cognitive Resources","volume":"11","year":"2020","journal-title":"Front. Psychol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ferr\u00e9, G. (2018, January 13\u201316). Gesture\/speech integration in the perception of prosodic emphasis. Proceedings of the Speech Prosody, Pozna\u0144, Poland.","DOI":"10.21437\/SpeechProsody.2018-7"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"552533","DOI":"10.3389\/fpsyg.2020.552533","article-title":"When Gesture \u201cTakes Over\u201d: Speech-Embedded Nonverbal Depictions in Multimodal Interaction","volume":"11","author":"Hsu","year":"2021","journal-title":"Front. Psychol."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Tambe, S., Galphat, Y., Rijhwani, N., Goythale, A., and Patil, J. (2020, January 16\u201317). Analysing and Enhancing Communication Platforms available for a Deaf-Blind user. Proceedings of the 2020 IEEE International Symposium on Sustainable Energy, Signal Processing and Cyber Security (iSSSC), Gunupur Odisha, India.","DOI":"10.1109\/iSSSC50941.2020.9358823"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Seebun, G.R., and Nagowah, L. (2020, January 25\u201327). Let\u2019s Talk: An Assistive Mobile Technology for Hearing and Speech Impaired Persons. Proceedings of the 2020 3rd International Conference on Emerging Trends in Electrical, Electronic and Communications Engineering (ELECOM), Balaclava, Mauritius.","DOI":"10.1109\/ELECOM49001.2020.9297026"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Maarif, H.A.Q., Akmeliawati, R., and Bilal, S. (2012, January 3\u20135). Malaysian Sign Language database for research. Proceedings of the 2012 International Conference on Computer and Communication Engineering (ICCCE), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ICCCE.2012.6271327"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ryumin, D., Ivanko, D., and Ryumina, E. (2023). Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices. Sensors, 23.","DOI":"10.3390\/s23042284"},{"key":"ref_27","unstructured":"Novopoltsev, M., Verkhovtsev, L., Murtazin, R., Milevich, D., and Zemtsova, I. (2023). Fine-tuning of sign language recognition models: A technical report. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Konaite, M., Owolawi, P.A., Mapayi, T., Malele, V., Odeyemi, K., Aiyetoro, G., and Ojo, J.S. (2021, January 9\u201310). Smart Hat for the blind with Real-Time Object Detection using Raspberry Pi and TensorFlow Lite. Proceedings of the International Conference on Artificial Intelligence and its Applications (icARTi \u201821), Bagatelle, Mauritius.","DOI":"10.1145\/3487923.3487929"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"012114","DOI":"10.1088\/1742-6596\/1651\/1\/012114","article-title":"Real-time and accurate object detection on edge device with TensorFlow Lite","volume":"1651","author":"Dai","year":"2020","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kannan, R., Jian, C.J., and Guo, X. (2020, January 8\u201310). Adversarial Evasion Noise Attacks Against TensorFlow Object Detection API. Proceedings of the 15th International Conference for Internet Technology and Secured Transactions (ICITST), London, UK.","DOI":"10.23919\/ICITST51030.2020.9351331"},{"key":"ref_31","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv."},{"key":"ref_32","unstructured":"Yu, T., Gande, S., and Yu, R. (2015, January 27\u201330). An Open-Source Based Speech Recognition Android Application for Helping Handicapped Students Writing Programs. Proceedings of the International Conference on Wireless Networks (ICWN), Las Vegas, NV, USA."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/6\/319\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:45:43Z","timestamp":1760125543000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/6\/319"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,31]]},"references-count":32,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["info14060319"],"URL":"https:\/\/doi.org\/10.3390\/info14060319","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,31]]}}}