{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T19:22:07Z","timestamp":1776885727089,"version":"3.51.2"},"reference-count":36,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T00:00:00Z","timestamp":1726704000000},"content-version":"vor","delay-in-days":262,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,9,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Parameter-efficient fine-tuning (PEFT) for personalizing automatic speech recognition (ASR) has recently shown promise for adapting general population models to atypical speech. However, these approaches assume a priori knowledge of the atypical speech disorder being adapted for\u2014the diagnosis of which requires expert knowledge that is not always available. Even given this knowledge, data scarcity and high inter-\/intra-speaker variability further limit the effectiveness of traditional fine-tuning. To circumvent these challenges, we first identify the minimal set of model parameters required for ASR adaptation. Our analysis of each individual parameter\u2019s effect on adaptation performance allows us to reduce Word Error Rate (WER) by half while adapting 0.03% of all weights. Alleviating the need for cohort-specific models, we next propose the novel use of a meta-learned hypernetwork to generate highly individualized, utterance-level adaptations on-the-fly for a diverse set of atypical speech characteristics. Evaluating adaptation at the global, cohort, and individual-level, we show that hypernetworks generalize better to out-of-distribution speakers, while maintaining an overall relative WER reduction of 75.2% using 0.1% of the full parameter budget.<\/jats:p>","DOI":"10.1162\/tacl_a_00696","type":"journal-article","created":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T19:44:52Z","timestamp":1726775092000},"page":"1182-1196","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Hypernetworks for Personalizing ASR to Atypical Speech"],"prefix":"10.1162","volume":"12","author":[{"given":"Max","family":"M\u00fcller-Eberstein","sequence":"first","affiliation":[{"name":"IT University of Copenhagen, Denmark mamy@itu.dk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dianna","family":"Yee","sequence":"additional","affiliation":[{"name":"Apple, USA dianna_yee@apple.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karren","family":"Yang","sequence":"additional","affiliation":[{"name":"Apple, USA karren_yang@apple.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gautam Varma","family":"Mantena","sequence":"additional","affiliation":[{"name":"Apple, USA gmantena@apple.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Colin","family":"Lea","sequence":"additional","affiliation":[{"name":"Apple, USA colin_lea@apple.com"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,9,18]]},"reference":[{"key":"2024091919443763700_bib1","author":"Speech Accessibility Project"},{"key":"2024091919443763700_bib2","doi-asserted-by":"publisher","first-page":"4762","DOI":"10.18653\/v1\/2021.findings-emnlp.410","article-title":"MAD-G: Multilingual adapter generation for efficient cross- lingual transfer","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Ansell","year":"2021"},{"key":"2024091919443763700_bib3","doi-asserted-by":"publisher","first-page":"1086","DOI":"10.21437\/Interspeech.2018-1929","article-title":"VoxCeleb2: Deep Speaker Recognition","volume-title":"Proceedings of Interspeech 2018","author":"Chung","year":"2018"},{"key":"2024091919443763700_bib4","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-2650","article-title":"ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification","volume-title":"Proceedings of Interspeech 2020","author":"Desplanques","year":"2020"},{"key":"2024091919443763700_bib5","volume-title":"Motor Speech Disorders: Substrates, Differential Diagnosis, and Management","author":"Duffy","year":"1995"},{"key":"2024091919443763700_bib6","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-10602-6_17","volume-title":"Uniform Manifold Approximation and Projection (UMAP)","author":"Ghojogh","year":"2023"},{"key":"2024091919443763700_bib7","doi-asserted-by":"publisher","first-page":"4778","DOI":"10.21437\/Interspeech.2021-1384","article-title":"Automatic speech recognition of disordered speech: Personalized models outperforming human listeners on short phrases","volume-title":"Proceedings of Interspeech 2021","author":"Green","year":"2021"},{"key":"2024091919443763700_bib8","article-title":"Hypernetworks","volume-title":"International Conference on Learning Representations","author":"Ha","year":"2017"},{"key":"2024091919443763700_bib9","article-title":"LoRA: Low-rank adaptation of large language models","author":"Edward","year":"2021","journal-title":"Computing Research Repository (CoRR), arXiv e-prints, 2106.09685"},{"key":"2024091919443763700_bib10","doi-asserted-by":"publisher","first-page":"565","DOI":"10.18653\/v1\/2021.acl-long.47","article-title":"Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Mahabadi","year":"2021"},{"issue":"6","key":"2024091919443763700_bib11","doi-asserted-by":"publisher","first-page":"2008","DOI":"10.1137\/S1064827500377332","article-title":"Principal angles between subspaces in an A-based scalar product: Algorithms and perturbation estimates","volume":"23","author":"Knyazev","year":"2002","journal-title":"SIAM Journal on Scientific Computing"},{"key":"2024091919443763700_bib12","doi-asserted-by":"publisher","first-page":"817","DOI":"10.18653\/v1\/2021.acl-short.103","article-title":"Lightweight adapter tuning for multilingual speech translation","volume-title":"International Joint Conference on Natural Language Processing (IJCNLP)","author":"Le","year":"2021"},{"key":"2024091919443763700_bib13","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581224","article-title":"From user perceptions to technical improvement: Enabling people who stutter to better use speech recognition","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems","author":"Lea","year":"2023"},{"key":"2024091919443763700_bib14","doi-asserted-by":"publisher","first-page":"4833","DOI":"10.21437\/Interspeech.2021-697","article-title":"Disordered speech data collection: Lessons learned at 1 million utterances from Project Euphonia","volume-title":"Proceedings of Interspeech 2021","author":"MacDonald","year":"2021"},{"key":"2024091919443763700_bib15","doi-asserted-by":"publisher","first-page":"1565","DOI":"10.21437\/Interspeech.2007-126","article-title":"Modelling confusion matrices to improve speech recognition accuracy, with an application to dysarthric speech","volume-title":"Proceedings of Interspeech 2007","author":"Morales","year":"2007"},{"key":"2024091919443763700_bib16","doi-asserted-by":"publisher","first-page":"2765","DOI":"10.21437\/Interspeech.2004-668","article-title":"From WER and RIL to MER and WIL: Improved evaluation measures for connected speech recognition","volume-title":"Proceedings Interspeech 2004","author":"Morris","year":"2004"},{"issue":"C","key":"2024091919443763700_bib17","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2019.101027","article-title":"Voxceleb: Large-scale speaker verification in the wild","volume":"60","author":"Nagrani","year":"2020","journal-title":"Computer Science and Language"},{"key":"2024091919443763700_bib18","doi-asserted-by":"publisher","first-page":"2616","DOI":"10.21437\/Interspeech.2017-950","article-title":"VoxCeleb: A large-scale speaker identification dataset","volume-title":"Proceedings of Interspeech 2017","author":"Nagrani","year":"2017"},{"key":"2024091919443763700_bib19","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.39","article-title":"Adapterfusion: Non-destructive task composition for transfer learning","author":"Pfeiffer","year":"2021"},{"key":"2024091919443763700_bib20","first-page":"27854","article-title":"HyperTuning: Toward adapting large language models without back-propagation","volume-title":"Proceedings of the 40th International Conference on Machine Learning","author":"Phang","year":"2023"},{"key":"2024091919443763700_bib21","doi-asserted-by":"publisher","first-page":"151","DOI":"10.21437\/Interspeech.2023-1627","article-title":"Parameter-efficient dysarthric speech recognition using adapter fusion and householder transformation","volume-title":"Proceedings of INTERSPEECH 2023","author":"Qi","year":"2023"},{"key":"2024091919443763700_bib22","first-page":"28492","article-title":"Robust speech recognition via large-scale weak supervision","volume-title":"Proceedings of the 40th International Conference on Machine Learning","author":"Radford","year":"2023"},{"key":"2024091919443763700_bib23","article-title":"SpeechBrain: A general-purpose speech toolkit","volume":"arxiv:2106.04624","author":"Ravanelli","year":"2021","journal-title":"Computing Research Repository"},{"key":"2024091919443763700_bib24","article-title":"Learning multiple visual domains with residual adapters","volume-title":"Advances in Neural Information Processing Systems","author":"Rebuffi","year":"2017"},{"key":"2024091919443763700_bib25","volume-title":"SSI-4: Stuttering Severity Instrument Fourth Edition","author":"Riley","year":"2009"},{"key":"2024091919443763700_bib26","doi-asserted-by":"publisher","DOI":"10.3389\/fcomp.2022.770210","article-title":"Characterizing dysarthria diversity for automatic speech recognition: A tutorial from the clinical perspective","volume":"4","author":"Rowe","year":"2022","journal-title":"Frontiers in Computer Science"},{"issue":"1","key":"2024091919443763700_bib27","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1044\/jshd.2801.19","article-title":"Frequency of syllable repetition and \u2018stutterer\u2019 judgments","volume":"28","author":"Sander","year":"1963","journal-title":"Journal of Speech and Hearing Disorders"},{"issue":"2","key":"2024091919443763700_bib28","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1044\/2015_JSLHR-S-15-0086","article-title":"Dysarthria in adults with cerebral palsy: Clinical presentation and impacts on communication","volume":"59","author":"Sch\u00f6lderle","year":"2016","journal-title":"Journal of Speech, Language, and Hearing Research"},{"key":"2024091919443763700_bib29","doi-asserted-by":"publisher","first-page":"784","DOI":"10.21437\/Interspeech.2019-1427","article-title":"Personalizing ASR for Dysarthric and Accented Speech with Limited Data","volume-title":"Proceedings of Interspeech 2019","author":"Shor","year":"2019"},{"issue":"2","key":"2024091919443763700_bib30","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1097\/01.TGR.0000318899.87690.44","article-title":"Speech and swallowing in parkinson\u2019s disease","volume":"24","author":"Tjaden","year":"2008","journal-title":"Topics in Geriatric Rehabilitation"},{"key":"2024091919443763700_bib31","article-title":"On-device personalization of automatic speech recognition models for disordered speech","volume":"arxiv:2106.10259","author":"Tomanek","year":"2021","journal-title":"Computing Research Repository"},{"key":"2024091919443763700_bib32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/ICASSP49357.2023.10097195","article-title":"An analysis of degenerating speech due to progressive dysarthria on asr performance","volume-title":"International Conference on Acoustics, Speech, & Signal Processing (ICASSP)","author":"Tomanek","year":"2023"},{"key":"2024091919443763700_bib33","doi-asserted-by":"publisher","first-page":"6751","DOI":"10.18653\/v1\/2021.emnlp-main.541","article-title":"Residual adapters for parameter-efficient ASR adaptation to atypical and accented speech","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Tomanek","year":"2021"},{"issue":"3","key":"2024091919443763700_bib34","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1162\/coli_a_00443","article-title":"UDapter: Typology-based language adapters for multilingual dependency parsing and sequence labeling","volume":"48","author":"\u00dcst\u00fcn","year":"2022","journal-title":"Computational Linguistics"},{"key":"2024091919443763700_bib35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.541","article-title":"Hyper-X: A unified hypernetwork for multi-task multilingual transfer","author":"\u00dcst\u00fcn","year":"2022"},{"key":"2024091919443763700_bib36","doi-asserted-by":"publisher","first-page":"161","DOI":"10.21437\/Interspeech.2023-1921","article-title":"Latent phrase matching for dysarthric speech","volume-title":"Proceedings INTERSPEECH 2023","author":"Yee","year":"2023"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00696\/2470797\/tacl_a_00696.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00696\/2470797\/tacl_a_00696.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T19:44:56Z","timestamp":1726775096000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00696\/124461\/Hypernetworks-for-Personalizing-ASR-to-Atypical"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":36,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00696","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}