{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T05:37:52Z","timestamp":1781674672901,"version":"3.54.5"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T00:00:00Z","timestamp":1732147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2024,11,21]]},"abstract":"<jats:p>We propose TRAMBA, a hybrid transformer and Mamba architecture for acoustic and bone conduction speech enhancement, suitable for mobile and wearable platforms. Bone conduction speech enhancement has been impractical to adopt in mobile and wearable platforms for several reasons: (i) data collection is labor-intensive, resulting in scarcity; (ii) there exists a performance gap between state-of-art models with memory footprints of hundreds of MBs and methods better suited for resource-constrained systems. To adapt TRAMBA to vibration-based sensing modalities, we pre-train TRAMBA with audio speech datasets that are widely available. Then, users fine-tune with a small amount of bone conduction data. TRAMBA outperforms state-of-art GANs by up to 7.3% in Perceptual Evaluation of Speech Quality (PESQ) and 1.8% in Short-Time Objective Intelligibility (STOI), with an order of magnitude smaller memory footprint and an inference speed up of up to 465 times. We integrate TRAMBA into real systems and show that TRAMBA (i) improves battery life of wearables by up to 160% by requiring less data sampling and transmission; (ii) generates higher quality voice in noisy environments than over-the-air speech; (iii) requires a memory footprint of less than 20.0 MB.<\/jats:p>","DOI":"10.1145\/3699757","type":"journal-article","created":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T12:23:32Z","timestamp":1732191812000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-1602-5359","authenticated-orcid":false,"given":"Yueyuan","family":"Sui","sequence":"first","affiliation":[{"name":"Northwestern University, Evanston, Illinois, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1999-0078","authenticated-orcid":false,"given":"Minghui","family":"Zhao","sequence":"additional","affiliation":[{"name":"Columbia University, New York, New York, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7356-1348","authenticated-orcid":false,"given":"Junxi","family":"Xia","sequence":"additional","affiliation":[{"name":"Northwestern University, Evanston, Illinois, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6480-0299","authenticated-orcid":false,"given":"Xiaofan","family":"Jiang","sequence":"additional","affiliation":[{"name":"Columbia University, New York, New York, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5713-8885","authenticated-orcid":false,"given":"Stephen","family":"Xia","sequence":"additional","affiliation":[{"name":"Northwestern University, Evanston, Illinois, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,11,21]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"[n.d.]. MAX78000. https:\/\/www.analog.com\/en\/products\/max78000.html. Accessed: 2024-05-01."},{"key":"e_1_2_1_2_1","unstructured":"[n. d.]. STM32 32-bit ARM Cortex-MCUs. https:\/\/www.st.com\/en\/microcontrollers-microprocessors\/stm32-32-bit-arm-cortex-mcus.html. Accessed: 2024-05-01."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2011.11.005"},{"key":"e_1_2_1_4_1","unstructured":"Apple Inc. 2023. Create a Personal Voice on your Phone iPad or Mac. https:\/\/support.apple.com\/en-us\/104993 Last accessed on 2024-04-27."},{"key":"e_1_2_1_5_1","volume-title":"Pang Wei W Koh, and Stefano Ermon","author":"Birnbaum Sawyer","year":"2019","unstructured":"Sawyer Birnbaum, Volodymyr Kuleshov, Zayd Enam, Pang Wei W Koh, and Stefano Ermon. 2019. Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations. Advances in Neural Information Processing Systems 32 (2019)."},{"key":"e_1_2_1_6_1","unstructured":"Carl Gene Fordham. 2015. Basic Spoken English. https:\/\/www.carlgene.com\/Basic_Spoken_English_carlgene.com.pdf Last accessed on 2024-04-30."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3498361.3538933"},{"key":"e_1_2_1_8_1","volume-title":"International Conference on Learning Representations.","author":"Choi Hyeong-Seok","year":"2018","unstructured":"Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Ha, and Kyogu Lee. 2018. Phase-aware speech enhancement with deep complex u-net. In International Conference on Learning Representations."},{"key":"e_1_2_1_9_1","unstructured":"Krzysztof Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane Tamas Sarlos Peter Hawkins Jared Davis Afroz Mohiuddin Lukasz Kaiser et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 (2020)."},{"key":"e_1_2_1_10_1","volume-title":"Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847","author":"Defossez Alexandre","year":"2020","unstructured":"Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. 2020. Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847 (2020)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3631447"},{"key":"e_1_2_1_12_1","volume-title":"Generative adversarial nets. Advances in neural information processing systems 27","author":"Goodfellow Ian","year":"2014","unstructured":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014)."},{"key":"e_1_2_1_13_1","unstructured":"Grand View Research Inc. 2023. Smart Glasses Market Size & Share Analysis Report 2030. https:\/\/www.grandviewresearch.com\/industry-analysis\/smart-glasses-market-report Last accessed on 2024-04-25."},{"key":"e_1_2_1_14_1","unstructured":"Grand View Research Inc. 2024. Earphones And Headphones Market Size Report 2030. https:\/\/www.grandviewresearch.com\/industry-analysis\/earphone-and-headphone-market ."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASSP.1976.1162849"},{"key":"e_1_2_1_16_1","volume-title":"Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752","author":"Gu Albert","year":"2023","unstructured":"Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)."},{"key":"e_1_2_1_17_1","volume-title":"Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396","author":"Gu Albert","year":"2021","unstructured":"Albert Gu, Karan Goel, and Christopher R\u00e9. 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 (2021)."},{"key":"e_1_2_1_18_1","volume-title":"Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems 34","author":"Gu Albert","year":"2021","unstructured":"Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R\u00e9. 2021. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems 34 (2021), 572--585."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP49357.2023.10096301"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3581791.3596832"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TETCI.2017.2784878"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539490.3539598"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP43922.2022.9746303"},{"key":"e_1_2_1_24_1","unstructured":"Invensense. 2021. IIM-42352. https:\/\/invensense.tdk.com\/products\/smartindustrial\/iim-42352\/ Last accessed on 2024-04-27."},{"key":"e_1_2_1_25_1","volume-title":"Likert scale: Explored and explained. British journal of applied science & technology 7, 4","author":"Joshi Ankur","year":"2015","unstructured":"Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015. Likert scale: Explored and explained. British journal of applied science & technology 7, 4 (2015), 396--403."},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Knowles Electronics LLC. 2024. Knowles V2S. https:\/\/www.knowles.com\/V2S Last accessed on 2024-04-27.","DOI":"10.1016\/j.softx.2024.101811"},{"key":"e_1_2_1_27_1","unstructured":"Volodymyr Kuleshov S Zayd Enam and Stefano Ermon. 2017. Audio super-resolution using neural nets. In ICLR (Workshop Track)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178801"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP39728.2021.9413439"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.3390\/s23010035"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3550338","article-title":"AccMyrinx: Speech Synthesis with Non-Acoustic Sensor","volume":"6","author":"Liang Yunji","year":"2022","unstructured":"Yunji Liang, Yuchen Qin, Qi Li, Xiaokai Yan, Zhiwen Yu, Bin Guo, Sagar Samtani, and Yanyong Zhang. 2022. AccMyrinx: Speech Synthesis with Non-Acoustic Sensor. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1--24.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462049"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2018.06.002"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2013-130"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462116"},{"key":"e_1_2_1_37_1","volume-title":"Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation","author":"Luo Yi","year":"2019","unstructured":"Yi Luo and Nima Mesgarani. 2019. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE\/ACM transactions on audio, speech, and language processing 27, 8 (2019), 1256--1266."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP49357.2023.10095382"},{"key":"e_1_2_1_39_1","unstructured":"Market.us. 2024. Wearable Technology Market Size Share. https:\/\/market.us\/report\/wearable-technology-market\/ Last accessed on 2024-04-25."},{"key":"e_1_2_1_40_1","volume-title":"Jose A Gonzalez, and Antonio M Peinado.","author":"Martin-Donas Juan Manuel","year":"2018","unstructured":"Juan Manuel Martin-Donas, Angel Manuel Gomez, Jose A Gonzalez, and Antonio M Peinado. 2018. A deep learning loss function based on the perceptual evaluation of the speech quality. IEEE Signal processing letters 25, 11 (2018), 1680--1684."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287058"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2021.3066303"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP43922.2022.9747699"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2016.2580946"},{"key":"e_1_2_1_45_1","unstructured":"NVIDIA Corporation. 2023. NVIDIA L40 GPU for Data Center. https:\/\/www.nvidia.com\/en-us\/data-center\/l40\/ Last accessed on 2024-04-27."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11671"},{"key":"e_1_2_1_47_1","volume-title":"International Conference on Machine Learning. PMLR, 28492--28518","author":"Radford Alec","year":"2023","unstructured":"Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning. PMLR, 28492--28518."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/MLSP52302.2021.9596082"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462417"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2001.941023"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.207"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550281"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2010.5495701"},{"key":"e_1_2_1_54_1","volume-title":"Attention is all you need. Advances in neural information processing systems 30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_1_55_1","volume-title":"Audio source separation and speech enhancement","author":"Vincent Emmanuel","unstructured":"Emmanuel Vincent, Tuomas Virtanen, and Sharon Gannot. 2018. Audio source separation and speech enhancement. John Wiley & Sons."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2022.3209943"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3417313.3429383"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3412382.3458784"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPSN54338.2022.00017"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3412382.3458267"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","unstructured":"Junichi Yamagishi Christophe Veaux and Kirsten MacDonald. 2019. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit. https:\/\/doi.org\/10.7488\/ds\/2645.","DOI":"10.7488\/ds\/2645"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9053795"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00805"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3699757","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3699757","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T16:29:53Z","timestamp":1758817793000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3699757"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,21]]},"references-count":63,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,11,21]]}},"alternative-id":["10.1145\/3699757"],"URL":"https:\/\/doi.org\/10.1145\/3699757","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,21]]},"assertion":[{"value":"2024-11-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}