{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T07:00:16Z","timestamp":1781593216577,"version":"3.54.5"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T00:00:00Z","timestamp":1724284800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No. 62072004"],"award-info":[{"award-number":["No. 62072004"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2024,8,22]]},"abstract":"<jats:p>Earphones have become a popular voice input and interaction device. However, airborne speech is susceptible to ambient noise, making it necessary to improve the quality and intelligibility of speech on earphones in noisy conditions. As the dual-microphone structure (i.e., outer and in-ear microphones) has been widely adopted in earphones (especially ANC earphones), we design EarSpeech which exploits in-ear acoustic sensory as the complementary modality to enable airborne speech enhancement. The key idea of EarSpeech is that in-ear speech is less sensitive to ambient noise and exhibits a correlation with airborne speech. However, due to the occlusion effect, in-ear speech has limited bandwidth, making it challenging to directly correlate with full-band airborne speech. Therefore, we exploit the occlusion effect to carry out theoretical modeling and quantitative analysis of this cross-channel correlation and study how to leverage such cross-channel correlation for speech enhancement. Specifically, we design a series of methodologies including data augmentation, deep learning-based fusion, and noise mixture scheme, to improve the generalization, effectiveness, and robustness of EarSpeech, respectively. Lastly, we conduct real-world experiments to evaluate the performance of our system. Specifically, EarSpeech achieves an average improvement ratio of 27.23% and 13.92% in terms of PESQ and STOI, respectively, and significantly improves SI-SDR by 8.91 dB. Benefiting from data augmentation, EarSpeech can achieve comparable performance with a small-scale dataset that is 40 times less than the original dataset. In addition, we validate the generalization of different users, speech content, and language types, respectively, as well as robustness in the real world via comprehensive experiments. The audio demo of EarSpeech is available on https:\/\/github.com\/EarSpeech\/earspeech.github.io\/.<\/jats:p>","DOI":"10.1145\/3678594","type":"journal-article","created":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T14:36:21Z","timestamp":1725892581000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["EarSpeech: Exploring In-Ear Occlusion Effect on Earphones for Data-efficient Airborne Speech Enhancement"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7555-1232","authenticated-orcid":false,"given":"Feiyu","family":"Han","sequence":"first","affiliation":[{"name":"Nanjing University of Information Science and Technology, Nanjing, China and University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1057-2793","authenticated-orcid":false,"given":"Panlong","family":"Yang","sequence":"additional","affiliation":[{"name":"Nanjing University of Information Science and Technology, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-0769-2924","authenticated-orcid":false,"given":"You","family":"Zuo","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5495-8869","authenticated-orcid":false,"given":"Fei","family":"Shang","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5454-157X","authenticated-orcid":false,"given":"Fenglei","family":"Xu","sequence":"additional","affiliation":[{"name":"Suzhou University of Science and Technology, Jiangsu Industrial Intelligent and Low-carbon Technology Engineering Center, Suzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6070-6625","authenticated-orcid":false,"given":"Xiang-Yang","family":"Li","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,9,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10772-013-9205-5"},{"key":"e_1_2_1_2_1","volume-title":"Common Voice: A Massively-Multilingual Speech Corpus. arXiv:1912.06670 [cs.CL]","author":"Ardila Rosana","year":"2020","unstructured":"Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common Voice: A Massively-Multilingual Speech Corpus. arXiv:1912.06670 [cs.CL]"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8683799"},{"key":"e_1_2_1_4_1","first-page":"S2","article-title":"HeartPrint","volume":"50","author":"Cao Yetong","year":"2023","unstructured":"Yetong Cao, Chao Cai, Fan Li, Zhe Chen, and Jun Luo. 2023. HeartPrint: Passive Heart Sounds Authentication Exploiting In-Ear Microphones. Heart 50, S1 (2023), S2.","journal-title":"Passive Heart Sounds Authentication Exploiting In-Ear Microphones. Heart"},{"key":"e_1_2_1_5_1","first-page":"1","article-title":"EarAce: Empowering Versatile Acoustic Sensing via Earable Active Noise Cancellation Platform","volume":"7","author":"Cao Yetong","year":"2023","unstructured":"Yetong Cao, Chao Cai, Anbo Yu, Fan Li, and Jun Luo. 2023. EarAce: Empowering Versatile Acoustic Sensing via Earable Active Noise Cancellation Platform. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 2 (2023), 1--23.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1121\/10.0001237"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1051\/aacus\/2021029"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3498361.3538933"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.3390\/a16030153"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3446382.3450216"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3550303","article-title":"UltraSpeech: Speech Enhancement by Interaction between Ultrasound and Speech","volume":"6","author":"Ding Han","year":"2022","unstructured":"Han Ding, Yizhan Wang, Hao Li, Cui Zhao, Ge Wang, Wei Xi, and Jizhong Zhao. 2022. UltraSpeech: Speech Enhancement by Interaction between Ultrasound and Speech. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1--25.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/89.397090"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3570361.3613281"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447993.3483240"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSIPA.2017.8281993"},{"key":"e_1_2_1_16_1","first-page":"1","article-title":"Voice in ear: Spoofing-resistant and passphrase-independent body sound authentication","volume":"5","author":"Gao Yang","year":"2021","unstructured":"Yang Gao, Yincheng Jin, Jagmohan Chauhan, Seokmin Choi, Jiyang Li, and Zhanpeng Jin. 2021. Voice in ear: Spoofing-resistant and passphrase-independent body sound authentication. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1--25.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.IR.4930"},{"key":"e_1_2_1_18_1","volume-title":"A method for silence removal and segmentation of speech signals, implemented in Matlab","author":"Giannakopoulos Theodoros","year":"2009","unstructured":"Theodoros Giannakopoulos. 2009. A method for silence removal and segmentation of speech signals, implemented in Matlab. University of Athens, Athens 2 (2009)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3560905.3568522"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2023.3314837"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3687048"},{"key":"e_1_2_1_22_1","volume-title":"BreathSign: Transparent and Continuous In-ear Authentication Using Bone-conducted Breathing Biometrics. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications. IEEE, 1--10","author":"Han Feiyu","year":"2023","unstructured":"Feiyu Han, Panlong Yang, Shaojie Yan, Haohua Du, and Yuanhao Feng. 2023. BreathSign: Transparent and Continuous In-ear Authentication Using Bone-conducted Breathing Biometrics. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications. IEEE, 1--10."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3581791.3596832"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2002.5745591"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition. 7482--7491","author":"Kendall Alex","year":"2018","unstructured":"Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7482--7491."},{"key":"e_1_2_1_26_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_1_27_1","volume-title":"Real-time digital signal processing: fundamentals, implementations and applications","author":"Kuo Sen M","unstructured":"Sen M Kuo, Bob H Lee, and Wenshun Tian. 2013. Real-time digital signal processing: fundamentals, implementations and applications. John Wiley & Sons."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2022.3193847"},{"key":"e_1_2_1_29_1","volume-title":"Towards understanding the importance of shortcut connections in residual networks. Advances in neural information processing systems 32","author":"Liu Tianyi","year":"2019","unstructured":"Tianyi Liu, Minshuo Chen, Mo Zhou, Simon S Du, Enlu Zhou, and Tuo Zhao. 2019. Towards understanding the importance of shortcut connections in residual networks. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485730.3485945"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458864.3467680"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2021.3066303"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581392"},{"key":"e_1_2_1_34_1","unstructured":"Online. 2022. Basic English Speaking. https:\/\/basicenglishspeaking.com"},{"key":"e_1_2_1_35_1","unstructured":"Online. 2023. Alango Technologies - Making Digital Sound Better. Technologies. Voice Enhancement. OnlyVoice\u2122 with In-Ear Microphone. https:\/\/www.alango.com\/technologies-onlyvoice-in-ear-microphone.php."},{"key":"e_1_2_1_36_1","volume-title":"Share & Trends Analysis Report By Price Band (&gt;100, 50--100, &lt;50), By Product (Earphones And Headphones), By Technology, By Application, By Region, And Segment Forecasts","year":"2023","unstructured":"Online. 2023. Earphones And Headphones Market Size, Share & Trends Analysis Report By Price Band (&gt;100, 50--100, &lt;50), By Product (Earphones And Headphones), By Technology, By Application, By Region, And Segment Forecasts, 2023 - 2030. https:\/\/www.researchandmarkets.com\/reports\/4118850\/earphones-and-headphones-market-size-share-and. (Accessed on 10\/30\/2023)."},{"key":"e_1_2_1_37_1","unstructured":"Online. 2023. PyTorch. https:\/\/pytorch.org\/."},{"key":"e_1_2_1_38_1","unstructured":"Online. 2023. Transcribe Audio to Text | Notta. https:\/\/www.notta.ai\/en. (Accessed on 11\/09\/2023)."},{"key":"e_1_2_1_39_1","unstructured":"Online. 2024. AS-B6027AL30-RC microphone. http:\/\/www.aospow.com\/Products\/znjqry6mmm.html. (Accessed on 04\/21\/2024)."},{"key":"e_1_2_1_40_1","unstructured":"Online. 2024. WH-1000XM4 | Help Guide | Speaking with someone while wearing the headset (Speak-to-Chat). https:\/\/helpguide.sony.net\/mdr\/wh1000xm4\/v1\/en\/contents\/TP0002754732.html. (Accessed on 04\/17\/2024)."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2023.3250846"},{"key":"e_1_2_1_42_1","volume-title":"The importance of phase in speech enhancement. speech communication 53, 4","author":"Paliwal Kuldip","year":"2011","unstructured":"Kuldip Paliwal, Kamil W\u00f3jcicki, and Benjamin Shannon. 2011. The importance of phase in speech enhancement. speech communication 53, 4 (2011), 465--494."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178964"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8683634"},{"key":"e_1_2_1_45_1","volume-title":"A fully convolutional neural network for speech enhancement. arXiv preprint arXiv:1609.07132","author":"Park Se Rim","year":"2016","unstructured":"Se Rim Park and Jinwon Lee. 2016. A fully convolutional neural network for speech enhancement. arXiv preprint arXiv:1609.07132 (2016)."},{"key":"e_1_2_1_46_1","volume-title":"SEGAN: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452","author":"Pascual Santiago","year":"2017","unstructured":"Santiago Pascual, Antonio Bonafonte, and Joan Serra. 2017. SEGAN: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452 (2017)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2733373.2806390"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1080\/2050571X.2021.1913300"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460421.3480432"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2873728"},{"key":"e_1_2_1_51_1","volume-title":"Intelligent Human Computer Interaction: 10th International Conference, IHCI 2018, Allahabad, India, December 7-9, 2018, Proceedings 10","author":"Singh Premjeet","year":"2018","unstructured":"Premjeet Singh, Manoj Kumar Mukul, and Rajkishore Prasad. 2018. Bone conducted speech signal enhancement using LPC and MFCC. In Intelligent Human Computer Interaction: 10th International Conference, IHCI 2018, Allahabad, India, December 7-9, 2018, Proceedings 10. Springer, 148--158."},{"key":"e_1_2_1_52_1","volume-title":"MUSAN: A Music, Speech, and Noise Corpus. arXiv:1510.08484 arXiv:1510.08484v1.","author":"Snyder David","year":"2015","unstructured":"David Snyder, Guoguo Chen, and Daniel Povey. 2015. MUSAN: A Music, Speech, and Noise Corpus. arXiv:1510.08484 arXiv:1510.08484v1."},{"key":"e_1_2_1_53_1","volume-title":"Transcranial attenuation of bone-conducted sound when stimulation is at the mastoid and at the bone conduction hearing aid position. Otology & neurotology 33, 2","author":"Stenfelt Stefan","year":"2012","unstructured":"Stefan Stenfelt. 2012. Transcranial attenuation of bone-conducted sound when stimulation is at the mastoid and at the bone conduction hearing aid position. Otology & neurotology 33, 2 (2012), 105--114."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1080\/14992020701545880"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447993.3448626"},{"key":"e_1_2_1_56_1","first-page":"1","article-title":"A survey on techniques for enhancing speech","volume":"179","author":"Taha Tayseer MF","year":"2018","unstructured":"Tayseer MF Taha, Ahsan Adeel, and Amir Hussain. 2018. A survey on techniques for enhancing speech. International Journal of Computer Applications 179, 17 (2018), 1--14.","journal-title":"International Journal of Computer Applications"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASL.2007.907344"},{"key":"e_1_2_1_58_1","volume-title":"Marc Proesmans, Dengxin Dai, and Luc Van Gool.","author":"Vandenhende Simon","year":"2021","unstructured":"Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. 2021. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence 44, 7 (2021), 3614--3633."},{"key":"e_1_2_1_59_1","volume-title":"Measuring speech quality for text-to-speech systems: development and assessment of a modified mean opinion score (MOS) scale. Computer speech & language 19, 1","author":"Viswanathan Mahesh","year":"2005","unstructured":"Mahesh Viswanathan and Madhubalan Viswanathan. 2005. Measuring speech quality for text-to-speech systems: development and assessment of a modified mean opinion score (MOS) scale. Computer speech & language 19, 1 (2005), 55--83."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2022.3209943"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2022.3224285"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM48880.2022.9796951"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3307334.3326073"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6489"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494990"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2016.2536478"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1587\/transfun.E102.A.2001"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678594","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678594","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T14:45:31Z","timestamp":1755787531000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678594"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,22]]},"references-count":67,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,8,22]]}},"alternative-id":["10.1145\/3678594"],"URL":"https:\/\/doi.org\/10.1145\/3678594","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,22]]},"assertion":[{"value":"2024-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}