{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T14:50:42Z","timestamp":1781275842446,"version":"3.54.1"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"14","license":[{"start":{"date-parts":[[2023,2,28]],"date-time":"2023-02-28T00:00:00Z","timestamp":1677542400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,2,28]],"date-time":"2023-02-28T00:00:00Z","timestamp":1677542400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Sign language recognition is one of the fundamental ways to assist deaf people to communicate with others. An accurate vision-based sign language recognition system using deep learning is a fundamental goal for many researchers. Deep convolutional neural networks have been extensively considered in the last few years, and a slew of architectures have been proposed. Recently, Vision Transformer and other Transformers have shown apparent advantages in object recognition compared to traditional computer vision models such as Faster R-CNN, YOLO, SSD, and other deep learning models. In this paper, we propose a Vision Transformer-based sign language recognition method called DETR (Detection Transformer), aiming to improve the current state-of-the-art sign language recognition accuracy. The DETR method proposed in this paper is able to recognize sign language from digital videos with a high accuracy using a new deep learning model ResNet152\u2009+\u2009FPN (i.e., Feature Pyramid Network), which is based on Detection Transformer. Our experiments show that the method has excellent potential for improving sign language recognition accuracy. For instance, our newly proposed net ResNet152\u2009+\u2009FPN is able to enhance the detection accuracy up to 1.70% on the test dataset of sign language compared to the standard Detection Transformer models. Besides, an overall accuracy 96.45% was attained by using the proposed method.<\/jats:p>","DOI":"10.1007\/s11042-023-14646-0","type":"journal-article","created":{"date-parts":[[2023,3,2]],"date-time":"2023-03-02T12:01:41Z","timestamp":1677758501000},"page":"21673-21685","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":49,"title":["Sign language recognition from digital videos using feature pyramid network with detection transformer"],"prefix":"10.1007","volume":"82","author":[{"given":"Yu","family":"Liu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Parma","family":"Nand","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Md Akbar","family":"Hossain","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Minh","family":"Nguyen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei Qi","family":"Yan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,2,28]]},"reference":[{"key":"14646_CR1","doi-asserted-by":"crossref","unstructured":"Bastanfard A, Rezaei NA, Mottaghizadeh M, Fazel M (2010) A novel multimedia educational speech therapy system for hearing impaired children. Springer, pp. 705\u2013715","DOI":"10.1007\/978-3-642-15696-0_65"},{"key":"14646_CR2","doi-asserted-by":"crossref","unstructured":"Bauer B, Hienz H, Kraiss KF (2000) Video-based continuous sign language recognition using statistical methods. In: International Conference on Pattern Recognition (ICPR), pp. 463\u2013466","DOI":"10.1109\/ICPR.2000.906112"},{"key":"14646_CR3","doi-asserted-by":"crossref","unstructured":"Bauer, B., Hienz, H., Kraiss, K. (2000) Video-based continuous sign language recognition using statistical methods. In: International Conference on Pattern Recognition (ICPR)","DOI":"10.1007\/3-540-46616-9_17"},{"issue":"3","key":"14646_CR4","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1080\/17517575.2018.1557256","volume":"13","author":"UA Bhatti","year":"2019","unstructured":"Bhatti UA, Huang M, Wu D, Zhang Y, Mehmood A, Han H (2019) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterprise Inform Syst 13(3):329\u2013351","journal-title":"Enterprise Inform Syst"},{"key":"14646_CR5","doi-asserted-by":"crossref","unstructured":"Camgoz NC, Koller O, Hadfield S, Bowden R (2020) Sign language Transformers: Joint end-to-end sign language recognition and translation. arXiv: 2003.13830","DOI":"10.1109\/CVPR42600.2020.01004"},{"key":"14646_CR6","doi-asserted-by":"crossref","unstructured":"Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with Transformers. arXiv: 2005.12872","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"14646_CR7","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani M, Minderer M, Heigold G, Gelly S, et al. (2020) An image is worth 16 \u00d7 16 words: Transformers for image recognition at scale. arXiv:2010.11929"},{"key":"14646_CR8","doi-asserted-by":"crossref","unstructured":"Duarte A (2019) Cross-modal neural sign language translation. In: IEEE International Conference on Multimedia and Expo","DOI":"10.1145\/3343031.3352587"},{"key":"14646_CR9","unstructured":"Huang J, Zhou W, Li H, Li W (2015) Sign language recognition using 3D convolutional neural networks. In: IEEE International Conference on Multimedia and Expo"},{"issue":"13","key":"14646_CR10","doi-asserted-by":"publisher","first-page":"2683","DOI":"10.3390\/app9132683","volume":"9","author":"SK Ko","year":"2019","unstructured":"Ko SK, Kim CJ, Jung H, Cho C (2019) Neural sign language translation based on human keypoint estimation. Appl Sci 9(13):2683","journal-title":"Appl Sci"},{"key":"14646_CR11","doi-asserted-by":"crossref","unstructured":"Koller O, Ney H, Bowden R (2016) Deep hand: How to train a CNN on 1 million hand images when your data is continuous and weakly labelled. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3793\u20133802","DOI":"10.1109\/CVPR.2016.412"},{"key":"14646_CR12","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097\u20131105"},{"key":"14646_CR13","doi-asserted-by":"crossref","unstructured":"Liu J, Kuipers B, Savarese S. (2011) Recognizing human actions by attributes, In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3337\u20133344","DOI":"10.1109\/CVPR.2011.5995353"},{"key":"14646_CR14","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1016\/j.imavis.2016.04.004","volume":"55","author":"Z Liu","year":"2016","unstructured":"Liu Z, Zhang C, Tian Y (2016) 3D-based deep convolutional neural network for action recognition with depth sequences. Image Vis Comput 55:93\u2013100","journal-title":"Image Vis Comput"},{"key":"14646_CR15","doi-asserted-by":"publisher","first-page":"6389","DOI":"10.1007\/s11042-021-11806-y","volume":"81","author":"SAH Minoofam","year":"2022","unstructured":"Minoofam SAH, Bastanfard A, Keyvanpour MR (2022) RALF: an adaptive reinforcement learning framework for teaching dyslexic students. Multimed Tools Appl 81:6389\u20136412","journal-title":"Multimed Tools Appl"},{"key":"14646_CR16","doi-asserted-by":"crossref","unstructured":"Mishra A, Kumar V, Shiva M, Reddy K, Arulkumar S, Rai P, Mittal A (2018) A generative approach to zero-shot and few-shot action recognition. In: IEEE Winter Conference on Applications of Computer Vision. pp. 372\u2013380","DOI":"10.1109\/WACV.2018.00047"},{"key":"14646_CR17","doi-asserted-by":"crossref","unstructured":"Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4207\u20134215","DOI":"10.1109\/CVPR.2016.456"},{"key":"14646_CR18","doi-asserted-by":"crossref","unstructured":"Orbay A, Akarun L (2020) Neural sign language translation by learning tokenization. arXiv:2002.00479","DOI":"10.1109\/FG47880.2020.00002"},{"key":"14646_CR19","doi-asserted-by":"crossref","unstructured":"\u00d6zdemir O, Camg\u00f6z NC, Akarun L (2016) Isolated sign language recognition using improved dense trajectories. In: Sig Proc Commun Appl Conf (SIU)","DOI":"10.1109\/SIU.2016.7496151"},{"key":"14646_CR20","doi-asserted-by":"crossref","unstructured":"Qin J, Liu L, Shao L, Shen F, Ni B, Chen J, Wang Y (2017) Zero-shot action recognition with error-correcting output codes, In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2833\u20132842","DOI":"10.1109\/CVPR.2017.117"},{"key":"14646_CR21","doi-asserted-by":"crossref","unstructured":"Rastgoo R, Kiani K, Escalera S, Sabokrou M (2021) Multi-modal zero-shot sign language recognition. arXiv: 2109.00796","DOI":"10.1109\/CVPRW53098.2021.00384"},{"issue":"6","key":"14646_CR22","doi-asserted-by":"publisher","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","volume":"39","author":"S Ren","year":"2016","unstructured":"Ren S, He K, Girshick R, Sun J (2016) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137\u20131149","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"14646_CR23","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1007\/978-94-015-8935-2_10","volume-title":"Motion-based recognition. Computational Imaging and Vision","author":"T Starner","year":"1997","unstructured":"Starner T, Pentland A (1997) Real-time American sign language recognition from video using hidden Markov models. In: Shah M, Jain R (eds) Motion-based recognition. Computational Imaging and Vision, vol 9, pp 227\u2013243"},{"issue":"3","key":"14646_CR24","first-page":"75","volume":"11","author":"M S\u00fczg\u00fcn","year":"2015","unstructured":"S\u00fczg\u00fcn M et al (2015) Hospisign: an interactive sign language platform for hearing impaired. J Naval Sci Eng 11(3):75\u201392","journal-title":"J Naval Sci Eng"},{"issue":"4","key":"14646_CR25","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1016\/0031-3203(88)90048-9","volume":"21","author":"S Tamura","year":"1988","unstructured":"Tamura S, Kawasaki S (1988) Recognition of sign language motion images. Pattern Recogn 21(4):343\u2013353","journal-title":"Pattern Recogn"},{"key":"14646_CR26","unstructured":"Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, J\u00e9gou H (2021) Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning, pp. 10347\u201310357"},{"key":"14646_CR27","unstructured":"Vaswani A, Shazeer N, Parmar N, Yang L, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. arXiv: 1706.03762"},{"key":"14646_CR28","doi-asserted-by":"crossref","unstructured":"Wu J, Ishwar P, Konrad J (2016) Two-stream CNNs for gesture-based verification and identification: Learning user style. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 42\u201350","DOI":"10.1109\/CVPRW.2016.21"},{"key":"14646_CR29","doi-asserted-by":"crossref","unstructured":"Xiang N, Pan C, Li X (2021) An object algorithm combining FPN structure with DETR. In: ACM ICCCV, pp. 57\u201363","DOI":"10.1145\/3484274.3484284"},{"key":"14646_CR30","doi-asserted-by":"crossref","unstructured":"Xu T, Hospedales M, Gong S (2016) Multi-task zero-shot action recognition with prioritized data augmentation, In: European Conference on Computer Vision, pp. 343\u2013359","DOI":"10.1007\/978-3-319-46475-6_22"},{"key":"14646_CR31","doi-asserted-by":"crossref","unstructured":"Yin, K. (2020) Sign Language translation with Transformers. arXiv:2004.00588","DOI":"10.18653\/v1\/2020.coling-main.525"},{"key":"14646_CR32","doi-asserted-by":"crossref","unstructured":"Yin K, Read J (2020) Better sign language Translation with STMC-Transformer. In: International Conference on Computational Linguistics, pp. 5975\u20135989","DOI":"10.18653\/v1\/2020.coling-main.525"},{"key":"14646_CR33","unstructured":"Zhou D, Kang B, Jin X, Yang L, Lian X, Jiang Z, Hou Q, Jiashi FJ (2021) DeepViT: Towards deeper Vision Transformer. arXiv: 2103.11886"},{"key":"14646_CR34","doi-asserted-by":"crossref","unstructured":"Zhu Y, Long Y, Guan Y, Newsam S, Shao L(2018) Towards universal representation for unseen action recognition, In: IEEE Conference on Computer Vision and Pattern Recognition","DOI":"10.1109\/CVPR.2018.00983"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-14646-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-023-14646-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-14646-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T09:28:11Z","timestamp":1685006891000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-023-14646-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,28]]},"references-count":34,"journal-issue":{"issue":"14","published-print":{"date-parts":[[2023,6]]}},"alternative-id":["14646"],"URL":"https:\/\/doi.org\/10.1007\/s11042-023-14646-0","relation":{},"ISSN":["1380-7501","1573-7721"],"issn-type":[{"value":"1380-7501","type":"print"},{"value":"1573-7721","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,28]]},"assertion":[{"value":"11 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 June 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 February 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 February 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This work was supported by our school\u2019s summer research grant, it has not any conflicts of interests or competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}}]}}