{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T16:59:53Z","timestamp":1777654793784,"version":"3.51.4"},"reference-count":62,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T00:00:00Z","timestamp":1683763200000},"content-version":"vor","delay-in-days":130,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,5,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as transfer of semantic properties such as emphasis and emotion, in automatic dubbing systems.<\/jats:p>","DOI":"10.1162\/tacl_a_00551","type":"journal-article","created":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T20:21:36Z","timestamp":1683836496000},"page":"419-435","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":14,"title":["Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing"],"prefix":"10.1162","volume":"11","author":[{"given":"William","family":"Brannon","sequence":"first","affiliation":[{"name":"MIT Media Lab, USA. wbrannon@mit.edu"}]},{"given":"Yogesh","family":"Virkar","sequence":"additional","affiliation":[{"name":"AWS AI Labs, USA. yvvirkar@amazon.com"}]},{"given":"Brian","family":"Thompson","sequence":"additional","affiliation":[{"name":"AWS AI Labs, USA. brianjt@amazon.com"}]}],"member":"281","published-online":{"date-parts":[[2023,5,11]]},"reference":[{"key":"2023051120213145800_bib1","doi-asserted-by":"publisher","first-page":"98","DOI":"10.18653\/v1\/2022.iwslt-1.10","article-title":"Findings of the IWSLT 2022 Evaluation Campaign","volume-title":"Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)","author":"Anastasopoulos","year":"2022"},{"key":"2023051120213145800_bib2","doi-asserted-by":"publisher","first-page":"789","DOI":"10.18653\/v1\/P18-1073","article-title":"A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Artetxe","year":"2018"},{"key":"2023051120213145800_bib3","unstructured":"Paul\n              Boersma\n             and DavidWeenink. 2022. Praat: Doing phonetics by computer [Computer program]. Version 6.2.14, retrieved July 6, 2022 from https:\/\/www.praat.org, https:\/\/www.fon.hum.uva.nl\/praat\/"},{"key":"2023051120213145800_bib4","doi-asserted-by":"crossref","DOI":"10.4324\/9781315717166-4","article-title":"Dubbing","volume-title":"The Routledge Handbook of Audiovisual Translation Studies","author":"Bosseaux","year":"2018"},{"key":"2023051120213145800_bib5","doi-asserted-by":"publisher","first-page":"3111","DOI":"10.21437\/Interspeech.2021-560","article-title":"End-to-end speaker segmentation for overlap-aware resegmentation","volume-title":"Interspeech 2021","author":"Bredin","year":"2021"},{"key":"2023051120213145800_bib6","doi-asserted-by":"publisher","first-page":"7124","DOI":"10.1109\/ICASSP40776.2020.9052974","article-title":"Pyannote.Audio: Neural building blocks for speaker diarization","volume-title":"ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Bredin","year":"2020"},{"key":"2023051120213145800_bib7","volume-title":"Audiovisual Translation: Dubbing","author":"Chaume","year":"2012","edition":"1st"},{"key":"2023051120213145800_bib8","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/978-3-030-42105-2_6","article-title":"Dubbing","volume-title":"The Palgrave Handbook of Audiovisual Translation and Media Accessibility","author":"Chaume","year":"2020"},{"key":"2023051120213145800_bib9","first-page":"25","article-title":"Issues in audiovisual translation","volume-title":"The Routledge Companion to Translation Studies","author":"Chiaro","year":"2008","edition":"1st"},{"key":"2023051120213145800_bib10","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1075\/btl.78.24chi","article-title":"Issues of quality in screen translation: Problems and solutions","volume-title":"Benjamins Translation Library","author":"Chiaro","year":"2008"},{"key":"2023051120213145800_bib11","doi-asserted-by":"publisher","DOI":"10.1163\/9789401207881","volume-title":"Audiovisual Translation through a Gender Lens","author":"De Marco","year":"2012"},{"key":"2023051120213145800_bib12","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1075\/btl.148.06di","article-title":"Chapter 6. Are we all together across languages? An eye tracking study of original and dubbed films","volume-title":"Benjamins Translation Library","author":"Di Giovanni","year":"2019"},{"key":"2023051120213145800_bib13","first-page":"644","article-title":"A simple, fast, and effective reparameterization of IBM Model 2","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Dyer","year":"2013"},{"key":"2023051120213145800_bib14","doi-asserted-by":"publisher","first-page":"257","DOI":"10.18653\/v1\/2020.iwslt-1.31","article-title":"From speech-to-speech translation to automatic dubbing","volume-title":"Proceedings of the 17th International Conference on Spoken Language Translation","author":"Federico","year":"2020"},{"key":"2023051120213145800_bib15","doi-asserted-by":"publisher","first-page":"1481","DOI":"10.21437\/Interspeech.2020-2983","article-title":"Evaluating and optimizing prosodic alignment for automatic dubbing","volume-title":"Interspeech 2020","author":"Federico","year":"2020"},{"issue":"4","key":"2023051120213145800_bib16","doi-asserted-by":"publisher","first-page":"796","DOI":"10.1044\/jshr.1104.796","article-title":"Confusions among visually perceived consonants","volume":"11","author":"Fisher","year":"1968","journal-title":"Journal of Speech and Hearing Research"},{"key":"2023051120213145800_bib17","volume-title":"Film Dubbing: Phonetic, Semiotic, Esthetic and Psychological Aspects","author":"Fodor","year":"1976","edition":"1st"},{"key":"2023051120213145800_bib18","volume-title":"Analysing Audiovisual Dialogue: Linguistic and Translational Insights","author":"Freddi","year":"2009"},{"key":"2023051120213145800_bib19","first-page":"733","article-title":"Results of the WMT21 Metrics Shared Task: Evaluating metrics with expert-based human evaluations on TED and news domain","volume-title":"Proceedings of the Sixth Conference on Machine Translation","author":"Freitag","year":"2021"},{"issue":"1","key":"2023051120213145800_bib20","doi-asserted-by":"publisher","first-page":"49","DOI":"10.7202\/029793ar","article-title":"Naturalness in the Spanish dubbing language: A case of not-so-close friends","volume":"54","author":"Fresco","year":"2009","journal-title":"Meta"},{"issue":"3","key":"2023051120213145800_bib21","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1037\/0033-2909.97.3.412","article-title":"Communicating emotion: The role of prosodic features.","volume":"97","author":"Frick","year":"1985","journal-title":"Psychological Bulletin"},{"key":"2023051120213145800_bib22","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1075\/btl.26.21her","article-title":"Dubbing and the dubbed text\u2014style and cohesion","volume-title":"Benjamins Translation Library","author":"Herbst","year":"1997"},{"key":"2023051120213145800_bib23","first-page":"16582","article-title":"Neural Dubber: Dubbing for videos according to scripts","volume-title":"Advances in Neural Information Processing Systems","author":"Chenxu","year":"2021"},{"key":"2023051120213145800_bib24","doi-asserted-by":"publisher","first-page":"4327","DOI":"10.18653\/v1\/2020.coling-main.382","article-title":"The two shades of dubbing in neural machine translation","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Karakanta","year":"2020"},{"issue":"6","key":"2023051120213145800_bib25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3355089.3356500","article-title":"Neural style-preserving visual dubbing","volume":"38","author":"Kim","year":"2019","journal-title":"ACM Transactions on Graphics"},{"key":"2023051120213145800_bib26","doi-asserted-by":"publisher","first-page":"7538","DOI":"10.1109\/ICASSP39728.2021.9414411","article-title":"Machine translation verbosity control for automatic dubbing","volume-title":"ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Lakew","year":"2021"},{"key":"2023051120213145800_bib27","doi-asserted-by":"publisher","first-page":"6242","DOI":"10.1109\/ICASSP43922.2022.9747023","article-title":"ISOMETRIC MT: Neural machine translation for automatic dubbing","volume-title":"ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Lakew","year":"2022"},{"key":"2023051120213145800_bib28","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139166621","volume-title":"Principles of Phonetics","author":"Laver","year":"1994"},{"key":"2023051120213145800_bib29","volume-title":"Cine independiente y traducci\u00f3n","author":"Ferriol","year":"2010"},{"issue":"2","key":"2023051120213145800_bib30","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1075\/babel.56.2.01mat","article-title":"Translations for dubbing as dynamic texts: Strategies in film synchronisation","volume":"56","author":"Matamala","year":"2010","journal-title":"Babel. Revue internationale de la traduction \/ International Journal of Translation"},{"issue":"3","key":"2023051120213145800_bib31","doi-asserted-by":"publisher","first-page":"356","DOI":"10.7202\/003608ar","article-title":"Concept of constrained translation. Non-linguistic perspectives of translation","volume":"33","author":"Mayoral","year":"1988","journal-title":"Meta"},{"key":"2023051120213145800_bib32","doi-asserted-by":"publisher","first-page":"498","DOI":"10.21437\/Interspeech.2017-1386","article-title":"Montreal forced aligner: Trainable text-speech alignment using Kaldi","volume-title":"Interspeech 2017","author":"McAuliffe","year":"2017"},{"key":"2023051120213145800_bib33","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-04966-9","volume-title":"Dialogue Writing for Dubbing: An Insider\u2019s Perspective","author":"Miggiani","year":"2019","edition":"1st"},{"issue":"6","key":"2023051120213145800_bib34","first-page":"56","article-title":"The art of dubbing","volume":"6","author":"Myers","year":"1973","journal-title":"Filmmakers\u2019 Newsletter"},{"key":"2023051120213145800_bib35","first-page":"49","article-title":"Parlato-parlato, parlato-scritto, parlato-recitato","volume":"29","author":"Nencioni","year":"1976","journal-title":"Strumenti critici"},{"key":"2023051120213145800_bib36","doi-asserted-by":"publisher","first-page":"20","DOI":"10.21437\/IberSPEECH.2018-5","article-title":"Bilingual prosodic dataset compilation for spoken language translation","volume-title":"IberSPEECH 2018","author":"\u00d6ktem","year":"2018"},{"key":"2023051120213145800_bib37","doi-asserted-by":"publisher","first-page":"4215","DOI":"10.21437\/Interspeech.2019-1621","article-title":"Prosodic phrase alignment for machine dubbing","volume-title":"Interspeech 2019","author":"\u00d6ktem","year":"2019"},{"key":"2023051120213145800_bib38","volume-title":"Tradurre per il doppiaggio: la trasposizione linguistica dell\u2019audiovisivo: teoria e pratica di un\u2019arte imperfetta","author":"Paolinelli","year":"2009","edition":"4th"},{"key":"2023051120213145800_bib39","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2023051120213145800_bib40","first-page":"117","article-title":"L\u2019allocuzione nel doppiaggio dall\u2019inglese all\u2019italiano","volume-title":"Traduzione multimediale per il cinema, la televisione e la scena : atti del Convegno internazionale : Forl\u00ec, 26\u201328 ottobre 1995 = Multimediale \u00dcbersetzung f\u00fcr film ... = Multimedia translation for film ... \/ a cura di Christine Heiss, Rosa Maria Bollettieri Bosinelli","author":"Pavesi","year":"1996"},{"key":"2023051120213145800_bib41","doi-asserted-by":"publisher","first-page":"197","DOI":"10.21832\/9781847691552-016","article-title":"Chapter 14: Dubbing English into Italian: A closer look at the translation of spoken language","volume-title":"New Trends in Audiovisual Translation","author":"Pavesi","year":"2009"},{"key":"2023051120213145800_bib42","first-page":"255","article-title":"An empirical take on the dubbing vs. subtitling debate: An eye movement study","volume":"19","author":"Perego","year":"2016","journal-title":"Lingue e Linguaggi"},{"key":"2023051120213145800_bib43","article-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2019"},{"key":"2023051120213145800_bib44","article-title":"SpeechBrain: A general-purpose speech toolkit","author":"Ravanelli","year":"2021","journal-title":"arXiv preprint arXiv:2106.04624"},{"key":"2023051120213145800_bib45","doi-asserted-by":"crossref","first-page":"2685","DOI":"10.18653\/v1\/2020.emnlp-main.213","article-title":"COMET: A neural framework for MT Evaluation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Rei","year":"2020"},{"key":"2023051120213145800_bib46","doi-asserted-by":"publisher","first-page":"94","DOI":"10.18653\/v1\/W19-5210","article-title":"Integration of dubbing constraints into machine translation","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)","author":"Saboo","year":"2019"},{"key":"2023051120213145800_bib47","first-page":"19","volume-title":"Dubbing and Prosody at the Interface","author":"S\u00e1nchez-Mompe\u00e1n","year":"2020"},{"key":"2023051120213145800_bib48","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-35521-0","volume-title":"The Prosody of Dubbed Speech: Beyond the Character\u2019s Words","author":"S\u00e1nchez-Mompe\u00e1n","year":"2020"},{"key":"2023051120213145800_bib49","article-title":"Filmmaking: Cultural referents, terminology and identity","volume-title":"Traducci\u00f3n, Interpretaci\u00f3n y Estudios Interculturales","author":"Santamaria","year":"2016"},{"key":"2023051120213145800_bib50","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2022-11136","article-title":"Isochrony-aware neural machine translation for automatic dubbing","volume-title":"Interspeech 2022","author":"Tam","year":"2022"},{"key":"2023051120213145800_bib51","doi-asserted-by":"publisher","first-page":"4904","DOI":"10.1109\/ICASSP.2015.7178903","article-title":"A mouth full of words: Visually consistent acoustic redubbing","volume-title":"2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Taylor","year":"2015"},{"key":"2023051120213145800_bib52","doi-asserted-by":"publisher","first-page":"1342","DOI":"10.18653\/v1\/D19-1136","article-title":"Vecalign: Improved sentence alignment in linear time and space","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Thompson","year":"2019"},{"key":"2023051120213145800_bib53","doi-asserted-by":"publisher","first-page":"5997","DOI":"10.18653\/v1\/2020.emnlp-main.483","article-title":"Exploiting sentence order in document alignment","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Thompson","year":"2020"},{"key":"2023051120213145800_bib54","doi-asserted-by":"publisher","first-page":"90","DOI":"10.18653\/v1\/2020.emnlp-main.8","article-title":"Automatic machine translation evaluation in many languages via zero-shot paraphrasing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Thompson","year":"2020"},{"key":"2023051120213145800_bib55","first-page":"561","article-title":"Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity","volume-title":"Proceedings of the Fifth Conference on Machine Translation","author":"Thompson","year":"2020"},{"issue":"3","key":"2023051120213145800_bib56","first-page":"113","article-title":"Sub-titling-Constrained Translation","volume":"27","author":"Titford","year":"1982","journal-title":"Lebende Sprachen"},{"key":"2023051120213145800_bib57","doi-asserted-by":"publisher","first-page":"652","DOI":"10.1109\/SLT48900.2021.9383459","article-title":"VOXLINGUA107: A dataset for spoken language recognition","volume-title":"2021 IEEE Spoken Language Technology Workshop (SLT)","author":"Valk","year":"2021"},{"key":"2023051120213145800_bib58","doi-asserted-by":"publisher","first-page":"7543","DOI":"10.1109\/ICASSP39728.2021.9414966","article-title":"Improvements to prosodic alignment for automatic dubbing","volume-title":"ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Virkar","year":"2021"},{"key":"2023051120213145800_bib59","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2022-11089","article-title":"Prosodic alignment for off-screen automatic dubbing","author":"Virkar","year":"2022"},{"key":"2023051120213145800_bib60","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1075\/btl.17.26zab","article-title":"Dubbing and the nonverbal dimension of translation","volume-title":"Benjamins Translation Library","author":"Zabalbeascoa","year":"1997"},{"key":"2023051120213145800_bib61","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1075\/btl.77.05zab","article-title":"The nature of the audiovisual text and its parameters","volume-title":"Benjamins Translation Library","author":"Zabalbeascoa","year":"2008"},{"key":"2023051120213145800_bib62","first-page":"109","article-title":"Translation and transcreation in the dubbing process: A genetic approach","volume":"7","author":"Zanotti","year":"2014","journal-title":"Cultus"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00551\/2093728\/tacl_a_00551.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00551\/2093728\/tacl_a_00551.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T20:22:01Z","timestamp":1683836521000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00551\/115968\/Dubbing-in-Practice-A-Large-Scale-Study-of-Human"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":62,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00551","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}