{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,11]],"date-time":"2026-11-11T21:48:46Z","timestamp":1794433726843,"version":"4.9.1"},"reference-count":47,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,9]],"date-time":"2022-08-09T00:00:00Z","timestamp":1660003200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The Convolutional Neural Network (CNN) has demonstrated excellent performance in image recognition and has brought new opportunities for sign language recognition. However, the features undergo many nonlinear transformations while performing the convolutional operation and the traditional CNN models are insufficient in dealing with the correlation between images. In American Sign Language (ASL) recognition, J and Z with moving gestures bring recognition challenges. This paper proposes a novel Two-Stream Mixed (TSM) method with feature extraction and fusion operation to improve the correlation of feature expression between two time-consecutive images for the dynamic gestures. The proposed TSM-CNN system is composed of preprocessing, the TSM block, and CNN classifiers. Two consecutive images in the dynamic gesture are used as inputs of streams, and resizing, transformation, and augmentation are carried out in the preprocessing stage. The fusion feature map obtained by addition and concatenation in the TSM block is used as inputs of the classifiers. Finally, a classifier classifies images. The TSM-CNN model with the highest performance scores depending on three concatenation methods is selected as the definitive recognition model for ASL recognition. We design 4 CNN models with TSM: TSM-LeNet, TSM-AlexNet, TSM-ResNet18, and TSM-ResNet50. The experimental results show that the CNN models with the TSM are better than models without TSM. The TSM-ResNet50 has the best accuracy of 97.57% for MNIST and ASL datasets and is able to be applied to a RGB image sensing system for hearing-impaired people.<\/jats:p>","DOI":"10.3390\/s22165959","type":"journal-article","created":{"date-parts":[[2022,8,10]],"date-time":"2022-08-10T04:20:32Z","timestamp":1660105232000},"page":"5959","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":46,"title":["Two-Stream Mixed Convolutional Neural Network for American Sign Language Recognition"],"prefix":"10.3390","volume":"22","author":[{"given":"Ying","family":"Ma","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Chonnam National University, Yeosu 59626, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0455-3126","authenticated-orcid":false,"given":"Tianpei","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Chonnam National University, Yeosu 59626, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kangchul","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Chonnam National University, Yeosu 59626, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,9]]},"reference":[{"key":"ref_1","unstructured":"(2022, June 07). World Federation of the Deaf (WFD). Available online: https:\/\/wfdeaf.org."},{"key":"ref_2","unstructured":"(2022, June 07). National Institute on Deafness and Other Communication Disorders (NIDCD), Available online: https:\/\/www.nidcd.nih.gov\/health\/american-sign-language."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"113794","DOI":"10.1016\/j.eswa.2020.113794","article-title":"Sign language recognition: A deep survey","volume":"164","author":"Rastgoo","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wang, P., Li, W., Liu, S., Gao, Z., Tang, C., and Ogunbona, P. (2016, January 4\u20138). Large-scale isolated gesture recognition using convolutional neural networks. Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun, Mexico.","DOI":"10.1109\/ICPR.2016.7899599"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"112829","DOI":"10.1016\/j.eswa.2019.112829","article-title":"MultiD-CNN: A multi-dimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences","volume":"139","author":"Elboushaki","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_6","first-page":"526","article-title":"LeNet-5 improvement based on FPGA acceleration","volume":"2020","author":"Hou","year":"2020","journal-title":"J. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"79","DOI":"10.18280\/ts.380108","article-title":"Comparison of Plant Leaf Classification Using Modified AlexNet and Support Vector Machine","volume":"39","author":"Wagle","year":"2021","journal-title":"Trait. Signal."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zhou, Y., Ren, F., Nishide, S., and Kang, X. (2019, January 8\u201310). Facial sentiment classification based on resnet-18 model. Proceedings of the 2019 International Conference on Electronic Engineering and Informatics (EEI), Nanjing, China.","DOI":"10.1109\/EEI48997.2019.00106"},{"key":"ref_9","unstructured":"Xiao, T., and Chao, C. (2019, January 28\u201330). Modulation pattern recognition based on resnet50 neural network. Proceedings of the 2nd IEEE International Conference on Information Communication and Signal Processing, Weihai, China."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1109\/TNSRE.2015.2424371","article-title":"An analysis of intrinsic and extrinsic hand muscle EMG for improved pattern recognition control","volume":"24","author":"Adewuyi","year":"2015","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Huang, D., Zhang, X., Saponas, T.S., Fogarty, J., and Gollakota, S. (2015, January 11\u201315). Leveraging dual-observable input for fine-grained thumb interaction using forearm EMG. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, Charlotte, NC, USA.","DOI":"10.1145\/2807442.2807506"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1692","DOI":"10.1109\/TPAMI.2015.2461544","article-title":"Moddrop: Adaptive multi-modal gesture recognition","volume":"38","author":"Neverova","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"132306","DOI":"10.1016\/j.physd.2019.132306","article-title":"Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network","volume":"404","author":"Sherstinsky","year":"2020","journal-title":"Phys. D Nonlinear Phenom."},{"key":"ref_14","unstructured":"Cate, H., Dalvi, F., and Hussain, Z. (2017). Sign language recognition using temporal classification. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chai, X., Liu, Z., Yin, F., Liu, Z., and Chen, X. (2016, January 4\u20138). Two streams recurrent neural networks for large-scale continuous gesture recognition. Proceedings of the International Conference on Pattern Recognition (ICPR), Cancun, Mexico.","DOI":"10.1109\/ICPR.2016.7899603"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Li, X., Mao, C., Huang, S., and Ye, Z. (2017, January 28\u201329). Chinese sign language recognition based on shs descriptor and encoder-decoder lstm model. Proceedings of the Chinese Conference on Biometric Recognition, Shenzhen, China.","DOI":"10.1007\/978-3-319-69923-3_77"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lin, C., Wan, J., Liang, Y., and Li, S.Z. (2018, January 15\u201319). Large-scale isolated gesture recognition using a refined fused model based on masked Res-C3D network and skeleton LSTM. Proceedings of the 13th International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00018"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Pu, J., Zhou, W., and Li, H. (2018, January 13\u201319). Dilated convolutional network with iterative optimization for continuous sign language recognition. Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-18), Stockholm, Sweden.","DOI":"10.24963\/ijcai.2018\/123"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, S., Guo, D., Zhou, W.-G., Zha, Z.-J., and Wang, M. (2018, January 22\u201326). Connectionist temporal fusion for sign language translation. Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Korea.","DOI":"10.1145\/3240508.3240671"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kim, S., and Ji, Y. (February, January 31). An effective sign language learning with object detection based ROI segmentation. Proceedings of the 2018 Second IEEE International Conference on Robotic Computing (IRC), Laguna Hills, CA, USA.","DOI":"10.1109\/IRC.2018.00069"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Devineau, G., Moutarde, F., Xi, W., and Yang, J. (2018, January 15\u201319). Deep learning for hand gesture recognition on skeletal data. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00025"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"El-Sawy, A., Hazem, E.L.B., and Loey, M. (2016, January 24\u201326). CNN for handwritten Arabic digits recognition based on LeNet-5. Proceedings of the International Conference on Advanced Intelligent Systems and Informatics, Cairo, Egypt.","DOI":"10.1007\/978-3-319-48308-5_54"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lu\u010di\u0107, M., and Schmid, C. (2021, January 10\u201317). Vivit: A video vision transformer. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00676"},{"key":"ref_24","unstructured":"Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. (2020). Tabtransformer: Tabular data modeling using contextual embeddings. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"145929","DOI":"10.1109\/ACCESS.2021.3122921","article-title":"Incorporating Relative Position Information in Transformer-Based Sign Language Recognition and Translation","volume":"9","author":"Aloysius","year":"2021","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Man, K.L., Smith, J., Siddique, K., and Guan, S.U. (2020). Improved two-stream model for human action recognition. EURASIP J. Image Video Process., 24.","DOI":"10.1186\/s13640-020-00501-x"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chen, J.C., Lee, C.Y., Huang, P.Y., and Lin, C.R. (2020). Driver Behavior Analysis via Two-Stream Deep Convolutional Neural Network. Appl. Sci., 10.","DOI":"10.3390\/app10061908"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Huang, J., Zhou, W., Zhang, Q., Li, H., and Li, W. (2018, January 2\u20137). Video-based sign language recognition without temporal segmentation. Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11903"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Gao, Q., Ogenyi, U.E., Liu, J., Ju, Z., and Liu, H. (2019). A two-stream CNN framework for American sign language recognition based on multimodal data fusion. UK Workshop on Computational Intelligence, Springer.","DOI":"10.1007\/978-3-030-29933-0_9"},{"key":"ref_30","unstructured":"(2021, June 27). MNIST Dataset. Available online: https:\/\/www.kaggle.com\/datamunge\/sign-language-mnist."},{"key":"ref_31","unstructured":"(2021, June 27). ASL Dataset. Available online: https:\/\/www.kaggle.com\/grassknoted\/asl-alphabet."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Miko\u0142ajczyk, A., and Grochowski, M. (2018, January 9\u201312). Data augmentation for improving deep learning in image classification problem. Proceedings of the 2018 International Interdisciplinary PhD Workshop (IIPhDW), Swinoujscie, Poland.","DOI":"10.1109\/IIPHDW.2018.8388338"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Banerjee, C., Mukherjee, T., and Pasiliao, E. (2019, January 18\u201320). An empirical study on generalizations of the ReLU activation function. Proceedings of the 2019 ACM Southeast Conference, Kennesaw, GA, USA.","DOI":"10.1145\/3299815.3314450"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Dubey, A.K., and Jain, V. (2019). Comparative study of convolution neural network\u2019s relu and leaky-relu activation functions. Applications of Computing, Automation and Wireless Systems in Electrical Engineering, Springer.","DOI":"10.1007\/978-981-13-6772-4_76"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ballester, P., and Araujo, R.M. (2016, January 12\u201317). On the performance of GoogLeNet and AlexNet applied to sketches. Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, Arizona USA.","DOI":"10.1609\/aaai.v30i1.10171"},{"key":"ref_36","unstructured":"Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. (2018). How does batch normalization help optimization?. arXiv."},{"key":"ref_37","unstructured":"Balduzzi, D., Frean, M., Leary, L., Lewis, J.P., Ma, K.W.D., and McWilliams, B. (2017). The shattered gradients problem: If resnets are the answer, then what is the question. International Conference on Machine Learning, PMLR."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bartoli, A., and Fusiello, A. (2020). Phonologically-meaningful subunits for deep learning-based sign language recognition. Computer Vision\u2014ECCV 2020 Workshops, Springer. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-030-66096-3"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Hao, A., Min, Y., and Chen, X. (2021, January 10\u201317). Self-mutual distillation learning for continuous sign language recognition. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.01111"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1750","DOI":"10.1109\/TMM.2021.3070438","article-title":"A Comprehensive Study on Deep Learning-based Methods for Sign Language Recognition","volume":"24","author":"Adaloglou","year":"2022","journal-title":"IEEE Trans. Multimed."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/j.cag.2021.04.017","article-title":"Compact joints encoding for skeleton-based dynamic hand gesture recognition","volume":"97","author":"Li","year":"2021","journal-title":"Comput. Graph."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kothadiya, D., Bhatt, C., Sapariya, K., Patel, K., Gil-Gonz\u00e1lez, A.B., and Corchado, J.M. (2022). Deepsign: Sign Language Detection and Recognition Using Deep Learning. Electronics, 11.","DOI":"10.3390\/electronics11111780"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Das, A., Gawde, S., Suratwala, K., and Kalbande, D. (2018, January 5). Sign language recognitionusing deep learning on custom processed static gesture images. Proceedings of the 2018 International Conference on Smart City and Emerging Technology (ICSCET), Mumbai, India.","DOI":"10.1109\/ICSCET.2018.8537248"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Alashhab, S., Gallego, A.-J., and Lozano, M.\u00c1. (2018, January 20\u201322). Hand gesture detection with convolutional neural networks. Proceedings of the International Symposium on Distributed Computing and Artificial Intelligence, Toledo, Spain.","DOI":"10.1007\/978-3-319-94649-8_6"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Kania, K., and Markowska-Kaczmar, U. (2018, January 3\u20137). American sign language fingerspelling recognition using wide residual networks. Proceedings of the International Conference on Artificial Intelligence and Soft Computing, Zakopane, Poland.","DOI":"10.1007\/978-3-319-91253-0_10"},{"key":"ref_46","first-page":"225","article-title":"Real-time American sign language recognition with convolutional neural networks","volume":"2","author":"Garcia","year":"2016","journal-title":"Convolutional Neural Netw. Vis. Recognit."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Bousbai, K., and Merah, M. (2019, January 24\u201325). A comparative study of hand gestures recognition based on MobileNetV2 and ConvNet models. Proceedings of the 2019 6th International Conference on Image and Signal Processing and their Applications (ISPA), Mostaganem, Algeria.","DOI":"10.1109\/ISPA48434.2019.8966918"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/16\/5959\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:06:22Z","timestamp":1760141182000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/16\/5959"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,9]]},"references-count":47,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["s22165959"],"URL":"https:\/\/doi.org\/10.3390\/s22165959","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,9]]}}}