{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T16:36:04Z","timestamp":1780418164087,"version":"3.54.1"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,5,10]],"date-time":"2024-05-10T00:00:00Z","timestamp":1715299200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>Voice cloning in text-to-speech (TTS) is the process of replicating the voice of a target speaker with limited data. Among various voice cloning techniques, this article focuses on zero-shot voice cloning. Although existing TTS models can generate high-quality speech for seen speakers, cloning the voice of an unseen speaker remains a challenging task. The key aspect of zero-shot voice cloning is to obtain a speaker embedding from the target speaker. Previous works have used a speaker encoder to obtain a fixed-size speaker embedding from a single reference audio unsupervised, but they suffer from insufficient speaker information and content information leakage in speaker embedding. To address these issues, this article proposes MRMI-TTS, a FastSpeech2-based framework that uses speaker embedding as a conditioning variable to provide speaker information. The MRMI-TTS extracts speaker embedding and content embedding from multi-reference audios using a speaker encoder and a content encoder. To obtain sufficient speaker information, multi-reference audios are selected based on sentence similarity. The proposed model applies mutual information minimization on the two embeddings to remove entangled information within each embedding. Experiments on the public English dataset VCTK show that our method can improve synthesized speech in terms of both similarity and naturalness, even for unseen speakers. Compared to state-of-the-art reference embedding learned methods, our method achieves the best performance on the zero-shot voice cloning task. Furthermore, we demonstrate that the proposed method has a better capability of maintaining the speaker embedding in different languages. Sample outputs are available on the demo page.<jats:xref ref-type=\"fn\"><jats:sup>1<\/jats:sup><\/jats:xref><\/jats:p>","DOI":"10.1145\/3649501","type":"journal-article","created":{"date-parts":[[2024,3,30]],"date-time":"2024-03-30T09:24:01Z","timestamp":1711790641000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["MRMI-TTS: Multi-Reference Audios and Mutual Information Driven Zero-Shot Voice Cloning"],"prefix":"10.1145","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6748-6832","authenticated-orcid":false,"given":"Yi Ting","family":"Chen","sequence":"first","affiliation":[{"name":"Harbin Institute of Technology Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8101-5567","authenticated-orcid":false,"given":"Wanting","family":"Li","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0271-8246","authenticated-orcid":false,"given":"Buzhou","family":"Tang","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,5,10]]},"reference":[{"key":"e_1_3_3_2_1","article-title":"Neural voice cloning with a few samples","author":"Arik Sercan","year":"2018","unstructured":"Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. 2018. Neural voice cloning with a few samples. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems"},{"key":"e_1_3_3_3_1","unstructured":"Yanyao Bian Changbin Chen Yongguo Kang and Zhenglin Pan. 2019. Multi-reference Tacotron by intercross training for style disentangling transfer and control in speech synthesis. arXiv:1904.02373. Retrieved from https:\/\/arxiv.org\/abs\/1904.02373"},{"key":"e_1_3_3_4_1","unstructured":"Weicheng Cai Jinkun Chen and Ming Li. 2018. Exploring the encoding layer and loss function in end-to-end speaker and language recognition system. In Proceedings of the Speaker and Language Recognition Workshop (Odyssey\u201918) 74\u201381. Retrieved from https:\/\/www.isca-archive.org\/odyssey_2018\/cai18_odyssey.html"},{"key":"e_1_3_3_5_1","doi-asserted-by":"crossref","unstructured":"Zexin Cai Chuxiong Zhang and Ming Li. 2020. From speaker verification to multispeaker speech synthesis deep transfer with feedback constraint. In Proceedings of Interspeech 3974\u20133978. Retrieved from https:\/\/www.isca-archive.org\/interspeech_2020\/cai20c_interspeech.html","DOI":"10.21437\/Interspeech.2020-1032"},{"key":"e_1_3_3_6_1","doi-asserted-by":"crossref","unstructured":"Edresson Casanova Christopher Shulby Eren G\u00f6lge Nicolas Michael M\u00fcller Frederico Santos de Oliveira Arnaldo Candido Junior Anderson da Silva Soares Sandra Maria Aluisio and Moacir Antonelli Ponti. 2021. SC-GlowTTS: an efficient zero-shot multi-speaker text-to-speech model. In Proceedings of Interspeech 3645\u20133949. Retrieved from https:\/\/www.isca-archive.org\/interspeech_2021\/casanova21b_interspeech.pdf","DOI":"10.21437\/Interspeech.2021-1774"},{"key":"e_1_3_3_7_1","unstructured":"Jiawei Chen Xu Tan Jian Luan Tao Qin and Tie-Yan Liu. 2020. HiFiSinger: Towards high-fidelity neural singing voice synthesis. arXiv:2009.01776. Retrieved from https:\/\/arxiv.org\/abs\/2009.01776"},{"key":"e_1_3_3_8_1","first-page":"1779","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Cheng Pengyu","year":"2020","unstructured":"Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. 2020. Club: A contrastive log-ratio upper bound of mutual information. In Proceedings of the International Conference on Machine Learning. PMLR, 1779\u20131788."},{"key":"e_1_3_3_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP39728.2021.9413880"},{"key":"e_1_3_3_10_1","doi-asserted-by":"crossref","unstructured":"Seungwoo Choi Seungju Han Dongyoung Kim and Sungjoo Ha. 2020. Attentron: Few-shot text-to-speech utilizing attention-based variable-length embedding. In Proceedings of Interspeech 2007\u20132011. https:\/\/www.isca-archive.org\/interspeech_2020\/choi20c_interspeech.html","DOI":"10.21437\/Interspeech.2020-2096"},{"key":"e_1_3_3_11_1","doi-asserted-by":"crossref","unstructured":"Ju-chieh Chou Cheng-chieh Yeh and Hung-yi Lee. 2019. One-shot voice conversion by separating speaker and content representations with instance normalization. In Proceedings of Interspeech 664\u2013668. https:\/\/www.isca-archive.org\/interspeech_2019\/chou19_interspeech.html","DOI":"10.21437\/Interspeech.2019-2663"},{"key":"e_1_3_3_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9054535"},{"key":"e_1_3_3_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP43922.2022.9747801"},{"key":"e_1_3_3_14_1","doi-asserted-by":"crossref","unstructured":"Anmol Gulati James Qin Chung-Cheng Chiu Niki Parmar Yu Zhang Jiahui Yu Wei Han Shibo Wang Zhengdong Zhang Yonghui Wu et\u00a0al. 2020. Conformer: Convolution-augmented transformer for speech recognition. arXiv:2005.08100. Retrieved from https:\/\/arxiv.org\/abs\/2005.08100","DOI":"10.21437\/Interspeech.2020-3015"},{"key":"e_1_3_3_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2022.3167258"},{"key":"e_1_3_3_16_1","article-title":"Transfer learning from speaker verification to multispeaker text-to-speech synthesis","author":"Jia Ye","year":"2018","unstructured":"Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et\u00a0al. 2018. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. InProceedings of the 32nd International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems"},{"key":"e_1_3_3_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9052942"},{"key":"e_1_3_3_18_1","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR\u201915) San Diego CA Conference Track Proceedings."},{"key":"e_1_3_3_19_1","first-page":"17022","article-title":"HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis","author":"Kong Jungil","year":"2020","unstructured":"Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. InProceedings of the 34th International Conference on Neural Information Processing Systems. 17022\u201317033.","journal-title":"Proceedings of the 34th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_3_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2018.8639672"},{"key":"e_1_3_3_21_1","unstructured":"Yanqing Liu Zhihang Xu Gang Wang Kuan Chen Bohan Li Xu Tan Jinzhu Li Lei He and Sheng Zhao. 2021. DelightfulTTS: The microsoft speech synthesis system for blizzard challenge 2021. arXiv:2110.12612. Retrieved from https:\/\/arxiv.org\/abs\/2110.12612"},{"key":"e_1_3_3_22_1","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Ma Shuang","year":"2018","unstructured":"Shuang Ma, Daniel Mcduff, and Yale Song. 2018. Neural TTS stylization with adversarial and collaborative games. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_3_23_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2017-1386"},{"key":"e_1_3_3_24_1","first-page":"7748","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Min Dongchan","year":"2021","unstructured":"Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. 2021. Meta-stylespeech: Multi-speaker adaptive text-to-speech generation. In Proceedings of the International Conference on Machine Learning. PMLR, 7748\u20137759."},{"key":"e_1_3_3_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9054301"},{"key":"e_1_3_3_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8683519"},{"key":"e_1_3_3_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-019-01265-2"},{"key":"e_1_3_3_28_1","unstructured":"Yi Ren Chenxu Hu Xu Tan Tao Qin Sheng Zhao Zhou Zhao and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. In Proceedings of the 1st International Conference on Learning Representations (ICLR\u201921). https:\/\/openreview.net\/forum?id=piLPYqxtWuA"},{"key":"e_1_3_3_29_1","doi-asserted-by":"crossref","unstructured":"Yi Ren Xu Tan Tao Qin Zhou Zhao and Tie-Yan Liu. 2022. Revisiting over-smoothness in text to speech. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 8197\u20138213. Retrieved from https:\/\/aclanthology.org\/2022.acl-long.564","DOI":"10.18653\/v1\/2022.acl-long.564"},{"key":"e_1_3_3_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9054548"},{"key":"e_1_3_3_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461368"},{"key":"e_1_3_3_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461368"},{"key":"e_1_3_3_33_1","doi-asserted-by":"crossref","unstructured":"Yao Shi Hui Bu Xin Xu Shaoji Zhang and Ming Li. 2021. AISHELL-3: A multi-speaker Mandarin TTS corpus and the baselines. In Proceedings of Interspeech. Retrieved from https:\/\/sites.duke.edu\/dkusmiip\/files\/2022\/11\/shi21c_interspeech.pdf","DOI":"10.21437\/Interspeech.2021-755"},{"key":"e_1_3_3_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461375"},{"key":"e_1_3_3_35_1","unstructured":"Dmitry Ulyanov Andrea Vedaldi and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv:1607.08022. Retrieved from https:\/\/arxiv.org\/abs\/1607.08022"},{"key":"e_1_3_3_36_1","article-title":"Neural discrete representation learning","author":"Oord Aaron Van Den","year":"2017","unstructured":"Aaron Van Den Oord, Oriol Vinyals, et\u00a0al. 2017. Neural discrete representation learning. InProceedings of the 31st International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems"},{"key":"e_1_3_3_37_1","article-title":"Attention is all you need","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems"},{"key":"e_1_3_3_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462665"},{"key":"e_1_3_3_39_1","doi-asserted-by":"crossref","unstructured":"Disong Wang Liqun Deng Yu Ting Yeung Xiao Chen Xunying Liu and Helen Meng. 2021. VQMIVC: Vector quantization and mutual information-based unsupervised speech representation disentanglement for one-shot voice conversion. In Proceedings of Interspeech 1344\u20131348. https:\/\/www.isca-archive.org\/interspeech_2021\/wang21n_interspeech.pdf","DOI":"10.21437\/Interspeech.2021-283"},{"key":"e_1_3_3_40_1","doi-asserted-by":"crossref","unstructured":"Yuxuan Wang R. J. Skerry-Ryan Daisy Stanton Yonghui Wu Ron J. Weiss Navdeep Jaitly Zongheng Yang Ying Xiao Zhifeng Chen Samy Bengio et\u00a0al. 2017. Tacotron: Towards end-to-end speech synthesis. In Proceedings of Interspeech 4006\u20134010. https:\/\/www.isca-archive.org\/interspeech_2017\/wang17n_interspeech.html","DOI":"10.21437\/Interspeech.2017-1452"},{"key":"e_1_3_3_41_1","first-page":"5180","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Wang Yuxuan","year":"2018","unstructured":"Yuxuan Wang, Daisy Stanton, Yu Zhang, R. J.-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A. Saurous. 2018a. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Proceedings of the International Conference on Machine Learning. PMLR, 5180\u20135189."},{"key":"e_1_3_3_42_1","first-page":"5180","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Wang Yuxuan","year":"2018","unstructured":"Yuxuan Wang, Daisy Stanton, Yu Zhang, R. J.-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A. Saurous. 2018b. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Proceedings of the International Conference on Machine Learning. PMLR, 5180\u20135189."},{"key":"e_1_3_3_43_1","doi-asserted-by":"crossref","unstructured":"Yihan Wu Xu Tan Bohan Li Lei He Sheng Zhao Ruihua Song Tao Qin and Tie-Yan Liu. 2022. AdaSpeech 4: Adaptive text to speech in zero-shot scenarios. In The Proceedings of Interspeech 2568\u20132572. https:\/\/www.isca-archive.org\/interspeech_2022\/wu22f_interspeech.html","DOI":"10.21437\/Interspeech.2022-901"},{"key":"e_1_3_3_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP39728.2021.9414226"},{"key":"e_1_3_3_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2021.03.005"},{"key":"e_1_3_3_46_1","doi-asserted-by":"crossref","unstructured":"Heiga Zen Viet Dang Rob Clark Yu Zhang Ron J. Weiss Ye Jia Zhifeng Chen and Yonghui Wu. 2019. LibriTTS: A corpus derived from LibriSpeech for text-to-speech. In Proceedings of Interspeech 1526\u20131530. https:\/\/www.isca-archive.org\/interspeech_2019\/zen19_interspeech.html","DOI":"10.21437\/Interspeech.2019-2441"}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649501","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649501","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T23:56:53Z","timestamp":1750291013000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649501"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,10]]},"references-count":45,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3649501"],"URL":"https:\/\/doi.org\/10.1145\/3649501","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"value":"2375-4699","type":"print"},{"value":"2375-4702","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,10]]},"assertion":[{"value":"2022-07-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-17","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}