{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T06:33:04Z","timestamp":1773901984472,"version":"3.50.1"},"reference-count":32,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,9,6]],"date-time":"2022-09-06T00:00:00Z","timestamp":1662422400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61902320"],"award-info":[{"award-number":["61902320"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2018AAA0100500"],"award-info":[{"award-number":["2018AAA0100500"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2022,9,6]]},"abstract":"<jats:p>The built-in loudspeakers of mobile devices (e.g., smartphones, smartwatches, and tablets) play significant roles in human-machine interaction, such as playing music, making phone calls, and enabling voice-based interaction. Prior studies have pointed out that it is feasible to eavesdrop on the speaker via motion sensors, but whether it is possible to synthesize speech from non-acoustic signals with sub-Nyquist sampling frequency has not been studied. In this paper, we present an end-to-end model to reconstruct the acoustic waveforms that are playing on the loudspeaker through the vibration captured by the built-in accelerometer. Specifically, we present an end-to-end speech synthesis framework dubbed AccMyrinx to eavesdrop on the speaker using the built-in low-resolution accelerometer of mobile devices. AccMyrinx takes advantage of the coexistence of an accelerometer with the loudspeaker on the same motherboard and compromises the loudspeaker by the solid-borne vibrations captured by the accelerometer. Low-resolution vibration signals are fed to a wavelet-based MelGAN to generate intelligible acoustic waveforms. We conducted extensive experiments on a large-scale dataset created based on audio clips downloaded from Voice of America (VOA). The experimental results show that AccMyrinx is capable of reconstructing intelligible acoustic signals that are playing on the loudspeaker with a smoothed word error rate (SWER) of 42.67%. The quality of synthesized speeches could be severely affected by several factors including gender, speech rate, and volume.<\/jats:p>","DOI":"10.1145\/3550338","type":"journal-article","created":{"date-parts":[[2022,9,7]],"date-time":"2022-09-07T14:54:27Z","timestamp":1662562467000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["AccMyrinx"],"prefix":"10.1145","volume":"6","author":[{"given":"Yunji","family":"Liang","sequence":"first","affiliation":[{"name":"Northwestern Polytechnical University, Xi'an, ShaanXi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuchen","family":"Qin","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, Xi'an, ShaanXi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qi","family":"Li","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, Xi'an, ShaanXi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaokai","family":"Yan","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, Xi'an, ShaanXi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiwen","family":"Yu","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, Xi'an, ShaanXi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bin","family":"Guo","sequence":"additional","affiliation":[{"name":"Northwestern Polytechnical University, Xi'an, ShaanXi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sagar","family":"Samtani","sequence":"additional","affiliation":[{"name":"Indiana University, Bloomington, Indiana, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yanyong","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, AnHui, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,9,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00004"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448300.3468499"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems","author":"Arik Sercan","year":"2017","unstructured":"Sercan \u00d6. Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017. Deep Voice 2: Multi-Speaker Neural Text-to-Speech. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 2966--2974."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372224.3417323"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372224.3417323"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3417986"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2021-414"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494986"},{"key":"e_1_2_1_9_1","volume-title":"2017 16th ACM\/IEEE International Conference on Information Processing in Sensor Networks (IPSN). 181--192","author":"Han Jun","year":"2017","unstructured":"Jun Han, Albert Jin Chung, and Patrick Tague. 2017. PitchIn: Eavesdropping via Intelligible Speech Reconstruction Using Non-acoustic Sensor Fusion. In 2017 16th ACM\/IEEE International Conference on Information Processing in Sensor Networks (IPSN). 181--192."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","unstructured":"Yan Huang Lei He Wenning Wei William Gale Jinyu Li and Yifan Gong. 2020. Using Personalized Speech Synthesis and Neural Language Generator for Rapid Speaker Adaptation. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP). 7399--7403. https:\/\/doi.org\/10.1109\/ICASSP40776.2020.9053104","DOI":"10.1109\/ICASSP40776.2020.9053104"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2017.2740004"},{"key":"e_1_2_1_12_1","volume-title":"Advances in Neural Information Processing Systems","author":"Kong Jungil","year":"2020","unstructured":"Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 17022--17033. https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/c5d736809766d46260d816d8dbc9eb44-Paper.pdf"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/PACRIM.1993.407206"},{"key":"e_1_2_1_14_1","volume-title":"Jose Sotelo, Alexandre de Br\u00e9bisson, Yoshua Bengio, and Aaron C Courville.","author":"Kumar Kundan","year":"2019","unstructured":"Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Br\u00e9bisson, Yoshua Bengio, and Aaron C Courville. 2019. MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\u00e9-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2019\/file\/6804c9bca0a615bdb9374d00a9fcba59-Paper.pdf"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICP46580.2020.9206494"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3133964"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287058"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 23rd USENIX Conference on Security Symposium","author":"Michalevsky Yan","year":"2014","unstructured":"Yan Michalevsky, Dan Boneh, and Gabi Nakibly. 2014. Gyrophone: Recognizing Speech from Gyroscope Signals. In Proceedings of the 23rd USENIX Conference on Security Symposium (San Diego, CA) (SEC'14). USENIX Association, USA, 1053--1067."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2804258"},{"key":"e_1_2_1_20_1","volume-title":"Evaluating Voice Quality and Speech Synthesis Using Crowdsourcing","author":"Parson Jeanne","unstructured":"Jeanne Parson, Daniela Braga, Michael Tjalve, and Jieun Oh. 2013. Evaluating Voice Quality and Speech Synthesis Using Crowdsourcing. In Text, Speech, and Dialogue, Ivan Habernal and V\u00e1clav Matou\u0161ek (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 233--240."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2014.2344856"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2011.5946971"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPWRD.2005.852367"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461368"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3494969"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1017\/ATSIP.2019.2"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130969"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2789168.2790119"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411836"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/SLT48900.2021.9383551"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-1238"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2742647.2742658"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550338","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3550338","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:40:47Z","timestamp":1752468047000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550338"}},"subtitle":["Speech Synthesis with Non-Acoustic Sensor"],"short-title":[],"issued":{"date-parts":[[2022,9,6]]},"references-count":32,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,9,6]]}},"alternative-id":["10.1145\/3550338"],"URL":"https:\/\/doi.org\/10.1145\/3550338","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,6]]},"assertion":[{"value":"2022-09-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}