{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T13:17:49Z","timestamp":1740143869811,"version":"3.37.3"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,11,24]],"date-time":"2021-11-24T00:00:00Z","timestamp":1637712000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,24]],"date-time":"2021-11-24T00:00:00Z","timestamp":1637712000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"national natural science foundation of china","doi-asserted-by":"publisher","award":["61631016"],"award-info":[{"award-number":["61631016"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"national key r&d program of china","award":["SQ2020YFF0426386"],"award-info":[{"award-number":["SQ2020YFF0426386"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J AUDIO SPEECH MUSIC PROC."],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Voice conversion is to transform a source speaker to the target one, while keeping the linguistic content unchanged. Recently, one-shot voice conversion gradually becomes a hot topic for its potentially wide range of applications, where it has the capability to convert the voice from any source speaker to any other target speaker even when both the source speaker and the target speaker are unseen during training. Although a great progress has been made in one-shot voice conversion, the naturalness of the converted speech remains a challenging problem. To further improve the naturalness of the converted speech, this paper proposes a two-level nested U-structure (U<jats:sup>2<\/jats:sup>-Net) voice conversion algorithm called U<jats:sup>2<\/jats:sup>-VC. The U<jats:sup>2<\/jats:sup>-Net can extract both local feature and multi-scale feature of log-mel spectrogram, which can help to learn the time-frequency structures of the source speech and the target speech. Moreover, we adopt sandwich adaptive instance normalization (SaAdaIN) in decoder for speaker identity transformation to retain more content information of the source speech while maintaining the speaker similarity between the converted speech and the target speech. Experiments on VCTK dataset show that U<jats:sup>2<\/jats:sup>-VC outperforms many SOTA approaches including AGAIN-VC and AdaIN-VC in terms of both objective and subjective measurements.<\/jats:p>","DOI":"10.1186\/s13636-021-00226-3","type":"journal-article","created":{"date-parts":[[2021,11,24]],"date-time":"2021-11-24T10:02:44Z","timestamp":1637748164000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["U2-VC: one-shot voice conversion using two-level nested U-structure"],"prefix":"10.1186","volume":"2021","author":[{"given":"Fangkun","family":"Liu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hui","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Renhua","family":"Peng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chengshi","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaodong","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,11,24]]},"reference":[{"key":"226_CR1","unstructured":"K. Qian, Y. Zhang, S. Chang, D. Cox, M. Hasegawa-Johnson, Unsupervised speech decomposition via triple information bottleneck (2021). http:\/\/arxiv.org\/abs\/2004.11284. Accessed 2020."},{"key":"226_CR2","doi-asserted-by":"publisher","unstructured":"L. W. Chen, H. Y. Lee, T. Yu, in Interspeech 2019. Generative adversarial networks for unpaired voice transformation on impaired speech (ISCA, 2019). https:\/\/doi.org\/10.21437\/interspeech.2019-1265.","DOI":"10.21437\/interspeech.2019-1265"},{"issue":"1","key":"226_CR3","doi-asserted-by":"publisher","first-page":"134","DOI":"10.1016\/j.specom.2011.07.007","volume":"54","author":"K. Nakamura","year":"2012","unstructured":"K. Nakamura, T. Toda, H. Saruwatari, K. Shikano, Speaking-aid systems using GMM-based voice conversion for electrolaryngeal speech. Speech Commun.54(1), 134\u2013146 (2012).","journal-title":"Speech Commun."},{"key":"226_CR4","doi-asserted-by":"publisher","unstructured":"M. Zhang, X. Wang, F. Fang, H. Li, J. Yamagishi, in Interspeech 2019. Joint training framework for text-to-speech and voice conversion using multi-source tacotron and wavenet (ISCA, 2019). https:\/\/doi.org\/10.21437\/interspeech.2019-1357.","DOI":"10.21437\/interspeech.2019-1357"},{"key":"226_CR5","doi-asserted-by":"publisher","unstructured":"S. Zhao, T. H. Nguyen, H. Wang, B. Ma, in Proc. Interspeech 2020. Towards natural bilingual and code-switched speech synthesis based on mix of monolingual recordings and cross-lingual voice conversion, (2020), pp. 2927\u20132931. https:\/\/doi.org\/10.21437\/Interspeech.2020-1163.","DOI":"10.21437\/Interspeech.2020-1163"},{"key":"226_CR6","doi-asserted-by":"publisher","unstructured":"L. Sun, S. Kang, K. Li, H. Meng, in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Voice conversion using deep bidirectional long short-term memory based recurrent neural networks (IEEE, 2015). https:\/\/doi.org\/10.1109\/icassp.2015.7178896.","DOI":"10.1109\/icassp.2015.7178896"},{"key":"226_CR7","doi-asserted-by":"publisher","unstructured":"T. Kaneko, H. Kameoka, K. Hiramatsu, K. Kashino, in Interspeech 2017. Sequence-to-sequence voice conversion with similarity metric learned using generative adversarial networks (ISCA, 2017). https:\/\/doi.org\/10.21437\/interspeech.2017-970.","DOI":"10.21437\/interspeech.2017-970"},{"key":"226_CR8","doi-asserted-by":"publisher","unstructured":"T. Kou, H. Kameoka, T. Kaneko, N. Hojo, in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Atts2s-vc: sequence-to-sequence voice conversion with attention and context preservation mechanisms (IEEE, 2019). https:\/\/doi.org\/10.1109\/icassp.2019.8683282.","DOI":"10.1109\/icassp.2019.8683282"},{"issue":"3","key":"226_CR9","doi-asserted-by":"publisher","first-page":"631","DOI":"10.1109\/TASLP.2019.2892235","volume":"27","author":"J. -X. Zhang","year":"2019","unstructured":"J. -X. Zhang, Z. -H. Ling, L. -J. Liu, Y. Jiang, L. -R. Dai, Sequence-to-sequence acoustic modeling for voice conversion. IEEE\/ACM Trans. Audio Speech Lang. Process.27(3), 631\u2013644 (2019). https:\/\/doi.org\/10.1109\/TASLP.2019.2892235.","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"226_CR10","doi-asserted-by":"publisher","unstructured":"H. Kameoka, T. Kaneko, K. Tanaka, N. Hojo, in 2018 IEEE Spoken Language Technology Workshop (SLT). Stargan-vc: non-parallel many-to-many voice conversion using star generative adversarial networks, (2018), pp. 266\u2013273. https:\/\/doi.org\/10.1109\/SLT.2018.8639535.","DOI":"10.1109\/SLT.2018.8639535"},{"key":"226_CR11","doi-asserted-by":"publisher","unstructured":"T. Kaneko, H. Kameoka, in 2018 26th European Signal Processing Conference (EUSIPCO). Cyclegan-vc: non-parallel voice conversion using cycle-consistent adversarial networks, (2018), pp. 2100\u20132104. https:\/\/doi.org\/10.23919\/EUSIPCO.2018.8553236.","DOI":"10.23919\/EUSIPCO.2018.8553236"},{"key":"226_CR12","doi-asserted-by":"publisher","unstructured":"T. Kaneko, H. Kameoka, K. Tanaka, N. Hojo, in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Cyclegan-vc2: improved cyclegan-based non-parallel voice conversion, (2019), pp. 6820\u20136824. https:\/\/doi.org\/10.1109\/ICASSP.2019.8682897.","DOI":"10.1109\/ICASSP.2019.8682897"},{"key":"226_CR13","doi-asserted-by":"publisher","unstructured":"T. Kaneko, H. Kameoka, K. Tanaka, N. Hojo, in Proc. Interspeech 2020. CycleGAN-VC3: examining and improving CycleGAN-VCs for mel-spectrogram conversion, (2020), pp. 2017\u20132021. https:\/\/doi.org\/10.21437\/Interspeech.2020-2280.","DOI":"10.21437\/Interspeech.2020-2280"},{"key":"226_CR14","doi-asserted-by":"publisher","unstructured":"W. -C. Huang, H. -T. Hwang, Y. -H. Peng, Y. Tsao, H. -M. Wang, in 2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP). Voice conversion based on cross-domain features using variational auto encoders, (2018), pp. 51\u201355. https:\/\/doi.org\/10.1109\/ISCSLP.2018.8706604.","DOI":"10.1109\/ISCSLP.2018.8706604"},{"key":"226_CR15","doi-asserted-by":"publisher","unstructured":"R. Levy-Leshem, R. Giryes, in 2020 28th European Signal Processing Conference (EUSIPCO). Taco-vc: a single speaker tacotron based voice conversion with limited data, (2021), pp. 391\u2013395. https:\/\/doi.org\/10.23919\/Eusipco47968.2020.9287448.","DOI":"10.23919\/Eusipco47968.2020.9287448"},{"key":"226_CR16","doi-asserted-by":"publisher","unstructured":"T. Li, Y. Liu, C. Hu, H. Zhao, Cvc: contrastive learning for non-parallel voice conversion (2021). https:\/\/doi.org\/10.21437\/interspeech.2021-137.","DOI":"10.21437\/interspeech.2021-137"},{"key":"226_CR17","doi-asserted-by":"publisher","first-page":"656","DOI":"10.1109\/TASLP.2020.3047262","volume":"29","author":"H. Kameoka","year":"2021","unstructured":"H. Kameoka, W. -C. Huang, K. Tanaka, T. Kaneko, N. Hojo, T. Toda, Many-to-many voice transformer network. IEEE\/ACM Trans. Audio Speech Lang. Process.29:, 656\u2013670 (2021). https:\/\/doi.org\/10.1109\/TASLP.2020.3047262.","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"226_CR18","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1109\/TASLP.2020.3038524","volume":"29","author":"B. Sisman","year":"2021","unstructured":"B. Sisman, J. Yamagishi, S. King, H. Li, An overview of voice conversion and its challenges: from statistical modeling to deep learning. IEEE\/ACM Trans. Audio Speech Lang. Process.29:, 132\u2013157 (2021). https:\/\/doi.org\/10.1109\/TASLP.2020.3038524.","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"226_CR19","unstructured":"K. Qian, Y. Zhang, S. Chang, X. Yang, M. Hasegawa-Johnson, AUTOVC: zero-shot voice style transfer with only autoencoder loss (2019). http:\/\/arxiv.org\/abs\/1905.05879. Accessed 2019."},{"key":"226_CR20","unstructured":"D. -Y. Wu, Y. -H. Chen, H. -Y. Lee, VQVC+: one-shot voice conversion by vector quantization and U-Net architecture (2020). http:\/\/arxiv.org\/abs\/2006.04154. Accessed 2020."},{"key":"226_CR21","doi-asserted-by":"publisher","unstructured":"J. -C. Chou, H. -Y. Lee, in Proc. Interspeech 2019. One-shot voice conversion by separating speaker and content representations with instance normalization, (2019), pp. 664\u2013668. https:\/\/doi.org\/10.21437\/Interspeech.2019-2663.","DOI":"10.21437\/Interspeech.2019-2663"},{"key":"226_CR22","doi-asserted-by":"publisher","unstructured":"Y. -H. Chen, D. -Y. Wu, T. -H. Wu, H. -Y. Lee, in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Again-vc: a one-shot voice conversion using activation guidance and adaptive instance normalization, (2021), pp. 5954\u20135958. https:\/\/doi.org\/10.1109\/ICASSP39728.2021.9414257.","DOI":"10.1109\/ICASSP39728.2021.9414257"},{"key":"226_CR23","doi-asserted-by":"publisher","first-page":"107404","DOI":"10.1016\/j.patcog.2020.107404","volume":"106","author":"X. Qin","year":"2020","unstructured":"X. Qin, Z. Zhang, C. Huang, M. Dehghan, O. R. Zaiane, M. Jagersand, U 2-net: going deeper with nested u-structure for salient object detection. Patt. Recog.106:, 107404 (2020). https:\/\/doi.org\/10.1016\/j.patcog.2020.107404.","journal-title":"Patt. Recog."},{"key":"226_CR24","volume-title":"U-net: convolutional networks for biomedical image segmentation","author":"O. Ronneberger","year":"2015","unstructured":"O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for biomedical image segmentation (Springer, Cham, 2015)."},{"key":"226_CR25","unstructured":"X. Gong, W. Chen, T. Chen, Z. Wang, Sandwich batch normalization (2021). http:\/\/arxiv.org\/abs\/2102.11382. Accessed 2021."},{"key":"226_CR26","doi-asserted-by":"publisher","unstructured":"R. Kubichek, in Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, 1. Mel-cepstral distance measure for objective speech quality assessment, (1993), pp. 125\u20131281. https:\/\/doi.org\/10.1109\/PACRIM.1993.407206.","DOI":"10.1109\/PACRIM.1993.407206"},{"key":"226_CR27","doi-asserted-by":"crossref","unstructured":"G. Mittag, B. Naderi, A. Chehadi, S. M\u00f6ller, NISQA: a deep CNN-self-attention model for multidimensional speech quality prediction with crowdsourced datasets (2021). http:\/\/arxiv.org\/abs\/2104.09494. Accessed 2021.","DOI":"10.21437\/Interspeech.2021-299"},{"key":"226_CR28","doi-asserted-by":"publisher","unstructured":"L. Wan, Q. Wang, A. Papir, I. L. Moreno, in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Generalized end-to-end loss for speaker verification, (2018), pp. 4879\u20134883. https:\/\/doi.org\/10.1109\/ICASSP.2018.8462665.","DOI":"10.1109\/ICASSP.2018.8462665"},{"key":"226_CR29","unstructured":"A. van den Oord, O. Vinyals, K. Kavukcuoglu, Neural discrete representation learning (2017). http:\/\/arxiv.org\/abs\/1711.00937. Accessed 2018."},{"key":"226_CR30","doi-asserted-by":"publisher","unstructured":"X. Huang, S. Belongie, in 2017 IEEE International Conference on Computer Vision (ICCV). Arbitrary style transfer in real-time with adaptive instance normalization, (2017), pp. 1510\u20131519. https:\/\/doi.org\/10.1109\/ICCV.2017.167.","DOI":"10.1109\/ICCV.2017.167"},{"key":"226_CR31","unstructured":"Y. Kwon, S. -W. Chung, H. -S. Heo, H. -G. Kang, Learning in your voice: non-parallel voice conversion based on speaker consistency loss (2020). http:\/\/arxiv.org\/abs\/2011.02168. Accessed 2020."},{"key":"226_CR32","unstructured":"K. -W. Kim, S. -W. Park, M. -C. Joe, Assem-VC: realistic voice conversion by assembling modern speech synthesis techniques (2021). http:\/\/arxiv.org\/abs\/2104.00931. Accessed 2021."},{"key":"226_CR33","doi-asserted-by":"publisher","first-page":"745","DOI":"10.1109\/TASLP.2021.3049336","volume":"29","author":"W. -C. Huang","year":"2021","unstructured":"W. -C. Huang, T. Hayashi, Y. -C. Wu, H. Kameoka, T. Toda, Pretraining techniques for sequence-to-sequence voice conversion. IEEE\/ACM Trans. Audio Speech Lang. Process.29:, 745\u2013755 (2021). https:\/\/doi.org\/10.1109\/TASLP.2021.3049336.","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"226_CR34","doi-asserted-by":"publisher","unstructured":"C. Veaux, J. Yamagishi, K. MacDonald, CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit, [sound]. University of Edinburgh. The Centre for Speech Technology Research (CSTR) (2017). https:\/\/doi.org\/10.7488\/ds\/1994.","DOI":"10.7488\/ds\/1994"},{"key":"226_CR35","unstructured":"K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brebisson, Y. Bengio, A. Courville, MelGAN: generative adversarial networks for conditional waveform synthesis (2019). http:\/\/arxiv.org\/abs\/1910.06711. Accessed 2019."},{"key":"226_CR36","unstructured":"I. Loshchilov, F. Hutter, Decoupled weight decay regularization (2017). http:\/\/arxiv.org\/abs\/1711.05101. Accessed 2019."},{"key":"226_CR37","doi-asserted-by":"publisher","unstructured":"A. W. Black, H. T. Bunnell, Y. Dou, P. Kumar Muthukumar, F. Metze, D. Perry, T. Polzehl, K. Prahallad, S. Steidl, C. Vaughn, in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Articulatory features for expressive speech synthesis, (2012), pp. 4005\u20134008. https:\/\/doi.org\/10.1109\/ICASSP.2012.6288796.","DOI":"10.1109\/ICASSP.2012.6288796"},{"key":"226_CR38","doi-asserted-by":"publisher","unstructured":"T. -H. Huang, J. -H. Lin, H. -Y. Lee, in 2021 IEEE Spoken Language Technology Workshop (SLT). How far are we from robust voice conversion: a survey, (2021), pp. 514\u2013521. https:\/\/doi.org\/10.1109\/SLT48900.2021.9383498.","DOI":"10.1109\/SLT48900.2021.9383498"}],"container-title":["EURASIP Journal on Audio, Speech, and Music Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13636-021-00226-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13636-021-00226-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13636-021-00226-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,24]],"date-time":"2021-11-24T10:06:45Z","timestamp":1637748405000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13636-021-00226-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,24]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["226"],"URL":"https:\/\/doi.org\/10.1186\/s13636-021-00226-3","relation":{},"ISSN":["1687-4722"],"issn-type":[{"type":"electronic","value":"1687-4722"}],"subject":[],"published":{"date-parts":[[2021,11,24]]},"assertion":[{"value":"20 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 October 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"40"}}