{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T20:09:23Z","timestamp":1776888563078,"version":"3.51.2"},"reference-count":41,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> The conventional paradigm in speech translation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applications, which often display transcripts alongside the translations to users. <\/jats:p><jats:p> We make this common requirement explicit and explore the task of jointly transcribing and translating speech. Although high accuracy of transcript and translation are crucial, even highly accurate systems can suffer from inconsistencies between both outputs that degrade the user experience. We introduce a methodology to evaluate consistency and compare several modeling approaches, including the traditional cascaded approach and end-to-end models. We find that direct models are poorly suited to the joint transcription\/translation task, but that end-to-end models that feature a coupled inference procedure are able to achieve strong consistency. We further introduce simple techniques for directly optimizing for consistency, and analyze the resulting trade-offs between consistency, transcription accuracy, and translation accuracy. <jats:sup>1<\/jats:sup> <\/jats:p>","DOI":"10.1162\/tacl_a_00340","type":"journal-article","created":{"date-parts":[[2020,11,13]],"date-time":"2020-11-13T20:25:25Z","timestamp":1605299125000},"page":"695-709","source":"Crossref","is-referenced-by-count":6,"title":["Consistent Transcription and Translation of Speech"],"prefix":"10.1162","volume":"8","author":[{"given":"Matthias","family":"Sperber","sequence":"first","affiliation":[{"name":"Apple."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hendra","family":"Setiawan","sequence":"additional","affiliation":[{"name":"Apple."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian","family":"Gollan","sequence":"additional","affiliation":[{"name":"Apple."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Udhyakumar","family":"Nallasamy","sequence":"additional","affiliation":[{"name":"Apple."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthias","family":"Paulik","sequence":"additional","affiliation":[{"name":"Apple."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"bib1","volume-title":"North American Chapter of the Association for Computational Linguistics (NAACL)","author":"Anastasopoulos Antonios","year":"2018"},{"key":"bib2","volume-title":"International Conference on Representation Learning (ICLR)","author":"Bahdanau Dzmitry","year":"2015"},{"issue":"4","key":"bib3","volume":"1","author":"Bates Douglas","year":"2013","journal-title":"R package version"},{"key":"bib4","first-page":"4774","volume-title":"International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Chiu Chung-Cheng","year":"2018"},{"key":"bib5","first-page":"1133","volume-title":"Annual Conference of the International Speech Communication Association (InterSpeech)","author":"Gangi Mattia A. Di","year":"2019"},{"key":"bib6","first-page":"1799","volume-title":"Association for Computational Linguistic (ACL)","author":"Domhan Tobias","year":"2018"},{"key":"bib7","first-page":"644","volume-title":"North American Chapter of the Association for Computational Linguistics (NAACL)","author":"Dyer Chris","year":"2013"},{"key":"bib8","volume-title":"Conference on Machine Translation (WMT)","author":"Fonseca Erick","year":"2019"},{"key":"bib9","unstructured":"Christian F\u00fcgen. 2008. A System for Simultaneous Translation of Lectures and Speeches. PhD thesis, University of Karlsruhe."},{"key":"bib10","first-page":"1019","volume-title":"Neural Information Processing Systems Conference (NIPS)","author":"Gal Yarin","year":"2016"},{"key":"bib11","volume-title":"North American Chapter of the Association for Computational Linguistics (NAACL)","author":"Gangi Antonino Mattia di","year":"2019"},{"issue":"8","key":"bib12","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1111\/j.1749-818X.2011.00295.x","volume":"5","author":"Gibson Edward","year":"2011","journal-title":"Language and Linguistics Compass"},{"key":"bib13","volume-title":"International Conference on Artificial Intelligence and Statistics (AISTATS)","author":"Glorot Xavier","year":"2010"},{"key":"bib14","volume-title":"Spoken Language Technology Workshop (SLT)","author":"Haghani Parisa","year":"2018"},{"issue":"5","key":"bib15","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1109\/MSP.2011.941852","volume":"28","author":"He Xiaodong","year":"2011","journal-title":"IEEE Signal Processing Magazine"},{"key":"bib16","first-page":"5632","volume-title":"International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"He Xiaodong","year":"2011"},{"key":"bib17","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"bib18","first-page":"765","volume-title":"Annual Conference of the International Speech Communication Association (InterSpeech)","author":"Hsiao Roger","year":"2006"},{"key":"bib19","first-page":"2630","volume-title":"Annual Conference of the International Speech Communication Association (InterSpeech)","author":"Kano Takatomo","year":"2017"},{"key":"bib20","volume-title":"International Conference on Learning Representations (ICLR)","author":"Kingma Diederik P.","year":"2014"},{"key":"bib21","first-page":"388","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Koehn Philipp","year":"2004"},{"key":"bib22","first-page":"66","author":"Kudo Taku","year":"2018","journal-title":"Empirical Methods in Natural Language Processing (EMNLP)"},{"key":"bib23","volume-title":"International Workshop on Spoken Language Translation (IWSLT)","author":"Lardilleux Adrien","year":"2017"},{"key":"bib24","volume-title":"Conference on Artificial Intelligence (AAAI)","author":"Liu Yuchen","year":"2020"},{"key":"bib25","volume-title":"Conference of the Association for Machine Translation in the Americas (AMTA) Open Source Software Showcase","author":"Neubig Graham","year":"2018"},{"key":"bib26","volume-title":"Approaches to Translation","author":"Newmark Peter","year":"1988"},{"key":"bib27","first-page":"517","volume-title":"International Conference on Acoustics, Speech, and Signal Processing (ICASSP)","author":"Ney Hermann","year":"1999"},{"issue":"5","key":"bib28","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1007\/s10459-010-9222-y","volume":"15","author":"Norman Geoff","year":"2010","journal-title":"Advances in Health Sciences Education"},{"key":"bib29","first-page":"311","volume-title":"Association for Computational Linguistic (ACL)","author":"Papineni Kishore","year":"2002"},{"key":"bib30","volume-title":"Advances in Neural Information Processing Systems (NeuIPS)","author":"Paszke Adam","year":"2019"},{"key":"bib31","first-page":"99","volume-title":"Proceedings of the Sixth Workshop on Statistical Machine Translation","author":"Popovi\u0107 Maja","year":"2011"},{"key":"bib32","first-page":"186","volume-title":"Conference on Machine Translation (WMT)","author":"Post Matt","year":"2019"},{"key":"bib33","first-page":"6174","volume-title":"Association for Computational Linguistic (ACL)","author":"Ribeiro Marco Tulio","year":"2019"},{"key":"bib34","volume-title":"Conference on Human factors in computing systems (CHI)","author":"Smith-Renner Alison","year":"2020"},{"key":"bib35","first-page":"1380","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Sperber Matthias","year":"2017"},{"key":"bib36","author":"Sperber Matthias","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics (TACL)"},{"issue":"2","key":"bib37","first-page":"116","volume":"6","author":"Stentiford Fred W. M.","year":"1988","journal-title":"British Telecom Technology Journal"},{"key":"bib38","first-page":"2818","volume-title":"Computer Vision and Pattern Recognition (CVPR)","author":"Szegedy Christian","year":"2016"},{"key":"bib39","first-page":"5998","volume-title":"Neural Information Processing Systems Conference (NIPS)","author":"Vaswani Ashish","year":"2017"},{"key":"bib40","first-page":"793","volume-title":"International Conference on Acoustics, Speech, and Signal Processing (ICASSP)","author":"Waibel Alex","year":"1991"},{"key":"bib41","volume-title":"Annual Conference of the International Speech Communication Association (InterSpeech)","author":"Weiss Ron J.","year":"2017"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00340","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:46Z","timestamp":1615585186000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96466"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":41,"alternative-id":["10.1162\/tacl_a_00340"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00340","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}