{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T16:17:23Z","timestamp":1788279443456,"version":"build-2803163510"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,3,29]],"date-time":"2022-03-29T00:00:00Z","timestamp":1648512000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100003399","name":"Science and Technology Commission of Shanghai Municipality","doi-asserted-by":"crossref","award":["19DZ2283800"],"award-info":[{"award-number":["19DZ2283800"]}],"id":[{"id":"10.13039\/501100003399","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"NSFC","doi-asserted-by":"crossref","award":["62102245"],"award-info":[{"award-number":["62102245"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2022,3,29]]},"abstract":"<jats:p>Sign language translation (SLT) is considered as the core technology to break the communication barrier between the deaf and hearing people. However, most studies only focus on recognizing the sequence of sign gestures (sign language recognition (SLR)), ignoring the significant difference of linguistic structures between sign language and spoken language. In this paper, we approach SLT as a spatio-temporal machine translation task and propose a wearable-based system, WearSign, to enable direct translation from the sign-induced sensory signals into spoken texts. WearSign leverages a smartwatch and an armband of ElectroMyoGraphy (EMG) sensors to capture the sophisticated sign gestures. In the design of the translation network, considering the significant modality and linguistic gap between sensory signals and spoken language, we design a multi-task encoder-decoder framework which uses sign glosses (sign gesture labels) for intermediate supervision to guide the end-to-end training. In addition, due to the lack of sufficient training data, the performance of prior studies usually degrades drastically when it comes to sentences with complex structures or unseen in the training set. To tackle this, we borrow the idea of back-translation and leverage the much more available spoken language data to synthesize the paired sign language data. We include the synthetic pairs into the training process, which enables the network to learn better sequence-to-sequence mapping as well as generate more fluent spoken language sentences.<\/jats:p>\n          <jats:p>We construct an American sign language (ASL) dataset consisting of 250 commonly used sentences gathered from 15 volunteers. WearSign achieves 4.7% and 8.6% word error rate (WER) in user-independent tests and unseen sentence tests respectively. We also implement a real-time version of WearSign which runs fully on the smartphone with a low latency and energy overhead.<\/jats:p>","DOI":"10.1145\/3517257","type":"journal-article","created":{"date-parts":[[2022,3,29]],"date-time":"2022-03-29T13:42:46Z","timestamp":1648561366000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["WearSign"],"prefix":"10.1145","volume":"6","author":[{"given":"Qian","family":"Zhang","sequence":"first","affiliation":[{"name":"Shanghai Jiao Tong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"JiaZhen","family":"Jing","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dong","family":"Wang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Run","family":"Zhao","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,3,29]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"World Health Organization 2019. 2019. Deafness and hearing loss. https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/deafness-and-hearing-loss"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01505"},{"key":"e_1_2_2_3_1","volume-title":"International conference on machine learning. PMLR, 173--182","author":"Amodei Dario","year":"2016","unstructured":"Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning. PMLR, 173--182."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308561.3353774"},{"key":"e_1_2_2_5_1","doi-asserted-by":"crossref","unstructured":"Diane Brentari. 2009. Effects of language modality on word segmentation: An experimental study of phonological factors in a sign language. In Laboratory Phonology 8. De Gruyter Mouton 155--184.","DOI":"10.1515\/9783110197211.1.155"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00812"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01004"},{"key":"e_1_2_2_8_1","volume-title":"Ariel M Cohen-Goldberg, and Karen Emmorey.","author":"Caselli Naomi K","year":"2017","unstructured":"Naomi K Caselli, Zed Sevcikova Sehyr, Ariel M Cohen-Goldberg, and Karen Emmorey. 2017. ASL-LEX: A lexical database of American Sign Language. Behavior research methods 49, 2 (2017), 784--801."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCNC.2018.8376998"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58586-0_41"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1515\/jisys-2016-0009"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.332"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/2503308.2503313"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.175"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3427315"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.3115\/1289189.1289273"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3131672.3131693"},{"key":"e_1_2_2_18_1","unstructured":"Google. [n.d.]. Deploy machine learning models on mobile and IoT devices. https:\/\/www.tensorflow.org\/lite"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143891"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0277(77)90006-3"},{"key":"e_1_2_2_21_1","unstructured":"HandSpeak. [n.d.]. Learn sign language and Deaf culture. https:\/\/www.handspeak.com\/learn\/"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3300061.3300117"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494992"},{"key":"e_1_2_2_24_1","unstructured":"Inside Facebook Reality Labs. 2021. Wrist-based interaction for the next computing platform. https:\/\/tech.fb.com\/inside-facebook-reality-labs-wrist-based-interaction-for-the-next-computing-platform\/"},{"key":"e_1_2_2_25_1","unstructured":"Thalmic Labs. 2020. The Myo armband. https:\/\/github.com\/thalmiclabs"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430780"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3569476"},{"key":"e_1_2_2_28_1","volume-title":"Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81."},{"key":"e_1_2_2_29_1","first-page":"1","article-title":"BlinkListener: \" Listen\" to Your Eye Blink Using Your Smartphone","volume":"5","author":"Liu Jialin","year":"2021","unstructured":"Jialin Liu, Dong Li, Lei Wang, and Jie Xiong. 2021. BlinkListener: \" Listen\" to Your Eye Blink Using Your Smartphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1--27.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432208"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449890"},{"key":"e_1_2_2_32_1","first-page":"478","article-title":"Systems and methods for determining axial orientation and location of a user's wrist","volume":"10","author":"Lor Jason","year":"2019","unstructured":"Jason Lor and Siddharth Nangia. 2019. Systems and methods for determining axial orientation and location of a user's wrist. US Patent 10,478,099.","journal-title":"US Patent"},{"key":"e_1_2_2_33_1","volume-title":"Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025","author":"Luong Minh-Thang","year":"2015","unstructured":"Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3191755"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1366728917000724"},{"key":"e_1_2_2_36_1","volume-title":"Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.456"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2005.112"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/FG47880.2020.00002"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448110"},{"key":"e_1_2_2_41_1","volume-title":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311--318","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311--318."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594368.2594379"},{"key":"e_1_2_2_43_1","volume-title":"Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779","author":"Park Daniel S","year":"2019","unstructured":"Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 (2019)."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2856767.2856794"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3150974"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00429"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3381010"},{"key":"e_1_2_2_48_1","volume-title":"Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709","author":"Sennrich Rico","year":"2015","unstructured":"Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709 (2015)."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2906388.2906407"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242599"},{"key":"e_1_2_2_51_1","unstructured":"Ilya Sutskever Oriol Vinyals and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. 3104--3112."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-1815"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274783.3274854"},{"key":"e_1_2_2_54_1","unstructured":"ultraleap. [n.d.]. Leap Motion Controller. https:\/\/www.ultraleap.com\/product\/leap-motion-controller\/"},{"key":"e_1_2_2_55_1","unstructured":"American Sign Language University. [n.d.]. American Sign Language: Grammar. https:\/\/www.lifeprint.com\/asl101\/pages-layout\/grammar.htm"},{"key":"e_1_2_2_56_1","unstructured":"American Sign Language University. [n.d.]. BASIC ASL. http:\/\/lifeprint.com\/asl101\/pages-layout\/concepts.htm"},{"key":"e_1_2_2_57_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008."},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897735"},{"key":"e_1_2_2_59_1","volume-title":"Hear Sign Language: A Real-time End-to-End Sign Language Recognition System","author":"Wang Zhibo","year":"2020","unstructured":"Zhibo Wang, Tengda Zhao, Jinxin Ma, Hongkai Chen, Kaixin Liu, Huajie Shao, Qian Wang, and Ju Ren. 2020. Hear Sign Language: A Real-time End-to-End Sign Language Recognition System. IEEE Transactions on Mobile Computing (2020)."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2016.2598302"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448105"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46478-7_27"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.525"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3356250.3360045"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302296"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494990"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494990"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494990"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2021.3087006"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2019.2962760"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.7001"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3517257","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3517257","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:26:27Z","timestamp":1752467187000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3517257"}},"subtitle":["Pushing the Limit of Sign Language Translation Using Inertial and EMG Wearables"],"short-title":[],"issued":{"date-parts":[[2022,3,29]]},"references-count":72,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3,29]]}},"alternative-id":["10.1145\/3517257"],"URL":"https:\/\/doi.org\/10.1145\/3517257","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,29]]},"assertion":[{"value":"2022-03-29","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}