{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T15:44:03Z","timestamp":1781797443234,"version":"3.54.5"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,6,12]],"date-time":"2023-06-12T00:00:00Z","timestamp":1686528000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Shenzhen Holdfound Foundation"},{"name":"Guangdong Provincial Key Laboratory of Human Digital Twin","award":["022B1212010004"],"award-info":[{"award-number":["022B1212010004"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2023,6,12]]},"abstract":"<jats:p>Sign language builds up an important bridge between the d\/Deaf and hard-of-hearing (DHH) and hearing people. Regrettably, most hearing people face challenges in comprehending sign language, necessitating sign language translation. However, state-of-the-art wearable-based techniques mainly concentrate on recognizing manual markers (e.g., hand gestures), while frequently overlooking non-manual markers, such as negative head shaking, question markers, and mouthing. This oversight results in the loss of substantial grammatical and semantic information in sign language. To address this limitation, we introduce SmartASL, a novel proof-of-concept system that can 1) recognize both manual and non-manual markers simultaneously using a combination of earbuds and a wrist-worn IMU, and 2) translate the recognized American Sign Language (ASL) glosses into spoken language. Our experiments demonstrate the SmartASL system's significant potential to accurately recognize the manual and non-manual markers in ASL, effectively bridging the communication gaps between ASL signers and hearing people using commercially available devices.<\/jats:p>","DOI":"10.1145\/3596255","type":"journal-article","created":{"date-parts":[[2023,6,12]],"date-time":"2023-06-12T18:58:16Z","timestamp":1686596296000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["SmartASL"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1812-4234","authenticated-orcid":false,"given":"Yincheng","family":"Jin","sequence":"first","affiliation":[{"name":"University at Buffalo, Department of Computer Science and Engineering, Buffalo, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3054-9590","authenticated-orcid":false,"given":"Shibo","family":"Zhang","sequence":"additional","affiliation":[{"name":"HP Inc., USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6811-0183","authenticated-orcid":false,"given":"Yang","family":"Gao","sequence":"additional","affiliation":[{"name":"East China Normal University, Department of Computer Science, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5930-3899","authenticated-orcid":false,"given":"Xuhai","family":"Xu","sequence":"additional","affiliation":[{"name":"University of Washington, Information School, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4284-6728","authenticated-orcid":false,"given":"Seokmin","family":"Choi","sequence":"additional","affiliation":[{"name":"University at Buffalo, Department of Computer Science and Engineering, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1880-5096","authenticated-orcid":false,"given":"Zhengxiong","family":"Li","sequence":"additional","affiliation":[{"name":"University of Colorado Denver, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2166-1417","authenticated-orcid":false,"given":"Henry J.","family":"Adler","sequence":"additional","affiliation":[{"name":"University at Buffalo, Department of Communicative Disorders and Sciences, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3020-3736","authenticated-orcid":false,"given":"Zhanpeng","family":"Jin","sequence":"additional","affiliation":[{"name":"South China University of Technology, School of Future Technology, China and University at Buffalo, Department of Computer Science and Engineering, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,6,12]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Accessibility.com LLC. 2022. Is American Sign Language (ASL) a language? https:\/\/www.accessibility.com\/blog\/is-american-sign-language-asl-a-language\/."},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Ashwin Ahuja Andrea Ferlini and Cecilia Mascolo. 2021. PilotEar: Enabling In-ear Inertial Navigation. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers. 139--145.","DOI":"10.1145\/3460418.3479326"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341163.3347747"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0940-9602(11)80182-2"},{"key":"e_1_2_1_5_1","volume-title":"CSEar: Meta-learning for Head Gesture Recognition Using Earphones in Internet of Healthcare Things","author":"Bi Hongliang","year":"2022","unstructured":"Hongliang Bi and Jiajia Liu. 2022. CSEar: Meta-learning for Head Gesture Recognition Using Earphones in Internet of Healthcare Things. IEEE Internet of Things Journal (2022)."},{"key":"e_1_2_1_6_1","volume-title":"Seminars in Hearing","author":"Branda Eric","unstructured":"Eric Branda and Tobias Wurzbacher. 2021. Motion Sensors in Automatic Steering of Hearing Aids. In Seminars in Hearing, Vol. 42. Thieme Medical Publishers, Inc., 237--247."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3300061.3345454"},{"key":"e_1_2_1_8_1","volume-title":"Motion-resilient Heart Rate Monitoring with In-ear Microphones. arXiv preprint arXiv:2108.09393","author":"Butkow Kayla-Jade","year":"2021","unstructured":"Kayla-Jade Butkow, Ting Dang, Andrea Ferlini, Dong Ma, and Cecilia Mascolo. 2021. Motion-resilient Heart Rate Monitoring with In-ear Microphones. arXiv preprint arXiv:2108.09393 (2021)."},{"key":"e_1_2_1_9_1","volume-title":"Non-manual cues in automatic sign language recognition. Personal and ubiquitous computing 18, 1","author":"Caridakis George","year":"2014","unstructured":"George Caridakis, Stylianos Asteriadis, and Kostas Karpouzis. 2014. Non-manual cues in automatic sign language recognition. Personal and ubiquitous computing 18, 1 (2014), 37--46."},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3534597","article-title":"PPGface: Like What You Are Watching? Earphones Can","volume":"6","author":"Choi Seokmin","year":"2022","unstructured":"Seokmin Choi, Yang Gao, Yincheng Jin, Se jun Kim, Jiyang Li, Wenyao Xu, and Zhanpeng Jin. 2022. PPGface: Like What You Are Watching? Earphones Can\" Feel\" Your Facial Expressions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1--32.","journal-title":"Feel\" Your Facial Expressions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_11_1","volume-title":"https:\/\/www.startasl.com\/what-does-d-d-and-d-deaf-mean-in-the-deaf-community\/. [Updated","author":"Community Deaf","year":"2021","unstructured":"Deaf Community. 2021. Deaf Culture. https:\/\/www.startasl.com\/what-does-d-d-and-d-deaf-mean-in-the-deaf-community\/. [Updated May 13, 2021]."},{"key":"e_1_2_1_12_1","unstructured":"ASLLRP Continuous Signing Corpora. 2022. American Sign Language Linguistic Research Project. https:\/\/dai.cs.rutgers.edu\/dai\/s\/dai. [Online]."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3131672.3131693"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447993.3483240"},{"key":"e_1_2_1_15_1","first-page":"5","article-title":"Facial action coding system: a technique for the measurement of facial movement","volume":"3","author":"Friesen E","year":"1978","unstructured":"E Friesen and Paul Ekman. 1978. Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3, 2 (1978), 5.","journal-title":"Palo Alto"},{"key":"e_1_2_1_16_1","first-page":"1","article-title":"SonicFace: Tracking Facial Expressions Using a Commodity Microphone Array","volume":"5","author":"Gao Yang","year":"2021","unstructured":"Yang Gao, Yincheng Jin, Seokmin Choi, Jiyang Li, Junjie Pan, Lin Shu, Chi Zhou, and Zhanpeng Jin. 2021. SonicFace: Tracking Facial Expressions Using a Commodity Microphone Array. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1--33.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351239"},{"key":"e_1_2_1_18_1","unstructured":"Google. 2022. AR Glass. https:\/\/nerdist.com\/article\/google-ar-glasses-live-translation-real-time-transcription\/."},{"key":"e_1_2_1_19_1","unstructured":"Audien Hearing. 2023. Atom Pro. https:\/\/audienhearing.com\/products\/audien-atom-pro-pair?variant=39511193255999."},{"key":"e_1_2_1_20_1","unstructured":"Carl-Herman Hjortsj\u00f6. 1969. Man's face and mimic language. Studentlitteratur."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3300061.3300117"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11903"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3498361.3538935"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550321"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494992"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.measurement.2020.108471"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2017.7953075"},{"key":"e_1_2_1_28_1","volume-title":"Michael Syskind Pedersen, and Susanna L\u00f8ve Callaway","author":"Goff Nicolas Le","year":"2016","unstructured":"Nicolas Le Goff, Jesper Jensen, Michael Syskind Pedersen, and Susanna L\u00f8ve Callaway. 2016. An introduction to OpenSound Navigator\u2122. Oticon A\/S (2016)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1249\/MSS.0000000000000183"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432208"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450268.3453537"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.3390\/electronics10141739"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3191755"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogbrainres.2004.08.012"},{"key":"e_1_2_1_35_1","unstructured":"Meta. 2016. Binaural Audio for Narrative AR. https:\/\/www.oculus.com\/story-studio\/blog\/binaural-audio-for-narrative-vr\/."},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Nicholas Michael Peng Yang Qingshan Liu Dimitris N Metaxas Carol Neidle and CBIM Center. 2011. A Framework for the Recognition of Nonmanual Markers in Segmented Sequences of American Sign Language.. In BMVC. 1--12.","DOI":"10.5244\/C.25.124"},{"key":"e_1_2_1_37_1","volume-title":"Hearing Loss and Hearing Aid Use. https:\/\/www.mdcd.nih.gov\/news\/multimedia\/hearing-loss-and-hearing-aid-use-text-version. [Updated","author":"NIH.","year":"2017","unstructured":"NIH. 2008. Hearing Loss and Hearing Aid Use. https:\/\/www.mdcd.nih.gov\/news\/multimedia\/hearing-loss-and-hearing-aid-use-text-version. [Updated July 17, 2017]."},{"key":"e_1_2_1_38_1","volume-title":"5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon LREC.","author":"Othman Achraf","year":"2012","unstructured":"Achraf Othman and Mohamed Jemni. 2012. English-ASL gloss parallel corpus 2012: ASLG-PC12. In 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon LREC."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2009.191"},{"key":"e_1_2_1_40_1","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 140 (2020), 1--67.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_41_1","unstructured":"Grand Review Research. 2023. Grand Review Research. https:\/\/www.grandviewresearch.com\/industry-analysis\/earphone-and-headphone-market. [Online]."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550314"},{"key":"e_1_2_1_43_1","volume-title":"CVPR Workshops.","author":"Sabyrov Arman","year":"2019","unstructured":"Arman Sabyrov, Medet Mukushev, and Vadim Kimmelman. 2019. Towards Real-time Sign Language Interpreting Robot: Evaluation of Non-manual Components on Recognition Accuracy.. In CVPR Workshops."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3381010"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2016.12.003"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097620.3097624"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPSN54338.2022.00019"},{"key":"e_1_2_1_48_1","volume-title":"ASL Dictionary -- Learn Essential Vocabulary. https:\/\/www.handspeak.com\/word\/. [Updated","author":"ASL.","year":"2020","unstructured":"StartASL. 2020. ASL Dictionary -- Learn Essential Vocabulary. https:\/\/www.handspeak.com\/word\/. [Updated April 28, 2020]."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compeleceng.2019.08.006"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432199"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0224365"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478085"},{"key":"e_1_2_1_53_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3448098","article-title":"EarDynamic: An Ear Canal Deformation Based Continuous User Authentication Using In-Ear Wearables","volume":"5","author":"Wang Zi","year":"2021","unstructured":"Zi Wang, Sheng Tan, Linghan Zhang, Yili Ren, Zhi Wang, and Jie Yang. 2021. EarDynamic: An Ear Canal Deformation Based Continuous User Authentication Using In-Ear Wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1--27.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_54_1","volume-title":"WH-questions in American Sign Language: Contributions of non-manual marking to structure and meaning","author":"Watson Katharine L","unstructured":"Katharine L Watson. 2010. WH-questions in American Sign Language: Contributions of non-manual marking to structure and meaning. Purdue University."},{"key":"e_1_2_1_55_1","volume-title":"Questions in American Sign Language: A quantitative analysis of raised and lowered eyebrows","author":"Weast Traci Patricia","unstructured":"Traci Patricia Weast. 2008. Questions in American Sign Language: A quantitative analysis of raised and lowered eyebrows. The University of Texas at Arlington."},{"key":"e_1_2_1_56_1","unstructured":"WHO. 2022. Deafness and hearing loss. https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/deafness-and-hearing-loss. [Online]."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2016.2598302"},{"key":"e_1_2_1_58_1","volume-title":"Sign language translation with transformers. arXiv preprint arXiv:2004.00588 2","author":"Yin Kayo","year":"2020","unstructured":"Kayo Yin. 2020. Sign language translation with transformers. arXiv preprint arXiv:2004.00588 2 (2020)."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070481.2070532"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3569482"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302296"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41928-020-0428-6"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3596255","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3596255","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:47:18Z","timestamp":1752468438000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3596255"}},"subtitle":["\"Point-of-Care\" Comprehensive ASL Interpreter Using Wearables"],"short-title":[],"issued":{"date-parts":[[2023,6,12]]},"references-count":62,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,12]]}},"alternative-id":["10.1145\/3596255"],"URL":"https:\/\/doi.org\/10.1145\/3596255","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,12]]},"assertion":[{"value":"2023-06-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}