{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T05:21:49Z","timestamp":1790832109809,"version":"4.1.0"},"reference-count":54,"publisher":"MIT Press","issue":"1","license":[{"start":{"date-parts":[[2021,12,22]],"date-time":"2021-12-22T00:00:00Z","timestamp":1640131200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,4,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>As more users across the world are interacting with dialog agents in their daily life, there is a need for better speech understanding that calls for renewed attention to the dynamics between research in automatic speech recognition (ASR) and natural language understanding (NLU). We briefly review these research areas and lay out the current relationship between them. In light of the observations we make in this article, we argue that (1) NLU should be cognizant of the presence of ASR models being used upstream in a dialog system\u2019s pipeline, (2) ASR should be able to learn from errors found in NLU, (3) there is a need for end-to-end data sets that provide semantic annotations on spoken input, (4) there should be stronger collaboration between ASR and NLU research communities.<\/jats:p>","DOI":"10.1162\/coli_a_00430","type":"journal-article","created":{"date-parts":[[2021,12,22]],"date-time":"2021-12-22T19:27:10Z","timestamp":1640201230000},"page":"221-232","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":14,"title":["Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems"],"prefix":"10.1162","volume":"48","author":[{"given":"Manaal","family":"Faruqui","sequence":"first","affiliation":[{"name":"Google Assistant. mfaruqui@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dilek","family":"Hakkani-T\u00fcr","sequence":"additional","affiliation":[{"name":"Amazon Alexa AI. hakkanit@amazon.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,4,4]]},"reference":[{"key":"2022040614221656200_bib1","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1007\/11880561_9","article-title":"The intention behind web queries","volume-title":"String Processing and Information Retrieval","author":"Baeza-Yates","year":"2006"},{"key":"2022040614221656200_bib2","first-page":"178","article-title":"Abstract Meaning Representation for sembanking","volume-title":"Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse","author":"Banarescu","year":"2013"},{"key":"2022040614221656200_bib3","doi-asserted-by":"publisher","first-page":"1021","DOI":"10.3115\/1613715.1613848","article-title":"The linguistic structure of English web-search queries","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Barr","year":"2008"},{"key":"2022040614221656200_bib4","first-page":"819","article-title":"Learning noun phrase query segmentation","volume-title":"Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)","author":"Bergsma","year":"2007"},{"key":"2022040614221656200_bib5","doi-asserted-by":"publisher","first-page":"6189","DOI":"10.1109\/ICASSP.2018.8461718","article-title":"Spoken language understanding without speech recognition","volume-title":"Proceedings of ICASSP","author":"Chen","year":"2018"},{"issue":"3","key":"2022040614221656200_bib6","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1109\/MSP.2008.918413","article-title":"Spoken language understanding","volume":"25","author":"De Mori","year":"2008","journal-title":"IEEE Signal Processing Magazine"},{"key":"2022040614221656200_bib7","doi-asserted-by":"publisher","first-page":"1067","DOI":"10.21437\/Interspeech.2012-324","article-title":"Joint decoding for speech recognition and semantic tagging","volume-title":"INTERSPEECH 2012","author":"Deoras","year":"2012"},{"key":"2022040614221656200_bib8","article-title":"Contextual error correction in automatic speech recognition","author":"Faruqui","year":"2020","journal-title":"Technical Disclosure Commons"},{"key":"2022040614221656200_bib9","article-title":"Investigation of error simulation techniques for learning dialog policies for conversational error recovery","author":"Fazel-Zarandi","year":"2019","journal-title":"arXiv preprint arXiv:1911.03378"},{"key":"2022040614221656200_bib10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/W18-2501","article-title":"AllenNLP: A deep semantic natural language processing platform","volume-title":"Proceedings of Workshop for NLP Open Source Software (NLP-OSS)","author":"Gardner","year":"2018"},{"key":"2022040614221656200_bib11","article-title":"Limits of end-to-end learning","author":"Glasmachers","year":"2017","journal-title":"CoRR"},{"key":"2022040614221656200_bib12","article-title":"Switchboard-1 Release 2 LDC97S62","author":"Godfrey","year":"1993"},{"key":"2022040614221656200_bib13","doi-asserted-by":"publisher","first-page":"720","DOI":"10.1109\/SLT.2018.8639043","article-title":"From audio to semantics: Approaches to end-to-end spoken language understanding","volume-title":"Spoken Language Technology Workshop (SLT)","author":"Haghani","year":"2018"},{"issue":"4","key":"2022040614221656200_bib14","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1016\/j.csl.2005.07.005","article-title":"Beyond ASR 1-best: Using word confusion networks in spoken language understanding","volume":"20","author":"Hakkani-T\u00fcr","year":"2006","journal-title":"Computer Speech & Language"},{"key":"2022040614221656200_bib15","doi-asserted-by":"publisher","first-page":"5632","DOI":"10.1109\/ICASSP.2011.5947637","article-title":"Why word error rate is not a good metric for speech recognizer training for the speech translation task?","volume-title":"2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"He","year":"2011"},{"key":"2022040614221656200_bib16","doi-asserted-by":"publisher","first-page":"263","DOI":"10.3115\/v1\/W14-4337","article-title":"The second dialog state tracking challenge","volume-title":"Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)","author":"Henderson","year":"2014"},{"issue":"10","key":"2022040614221656200_bib17","doi-asserted-by":"publisher","first-page":"1670","DOI":"10.3390\/electronics9101670","article-title":"Survey of automatic spelling correction","volume":"9","author":"Hl\u00e1dek","year":"2020","journal-title":"Electronics"},{"key":"2022040614221656200_bib18","article-title":"A survey on semantic parsing","volume-title":"Automated Knowledge Base Construction (AKBC)","author":"Kamath","year":"2019"},{"key":"2022040614221656200_bib19","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-2924","article-title":"End-to-end spoken language understanding without full transcripts","author":"Kuo","year":"2020","journal-title":"CoRR"},{"key":"2022040614221656200_bib20","doi-asserted-by":"publisher","first-page":"695","DOI":"10.21437\/Interspeech.2016-1583","article-title":"LatticeRnn: Recurrent neural networks over lattices","volume-title":"Proceedings of Interspeech","author":"Ladhak","year":"2016"},{"key":"2022040614221656200_bib21","doi-asserted-by":"publisher","first-page":"949","DOI":"10.1109\/SLT.2018.8639505","article-title":"ODSQA: Open-domain spoken question answering dataset","volume-title":"2018 IEEE Spoken Language Technology Workshop (SLT)","author":"Lee","year":"2018"},{"key":"2022040614221656200_bib22","first-page":"3459","article-title":"Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension","volume-title":"Proceedings of Interspeech","author":"Li","year":"2018"},{"key":"2022040614221656200_bib23","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-2846","article-title":"On the comparison of popular end-to-end models for large scale speech recognition","author":"Li","year":"2020","journal-title":"arXiv preprint arXiv: 2005.14327"},{"key":"2022040614221656200_bib24","article-title":"Improving spoken language understanding by exploiting ASR n-best hypotheses","author":"Li","year":"2020","journal-title":"arXiv preprint arXiv:2001.05284"},{"key":"2022040614221656200_bib25","doi-asserted-by":"publisher","first-page":"3291","DOI":"10.18653\/v1\/N19-1333","article-title":"Corpora generation for grammatical error correction","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Lichtarge","year":"2019"},{"key":"2022040614221656200_bib26","doi-asserted-by":"publisher","first-page":"2467","DOI":"10.18653\/v1\/2021.acl-long.192","article-title":"Robustness testing of language understanding in task-oriented dialog","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Liu","year":"2021"},{"issue":"5","key":"2022040614221656200_bib27","doi-asserted-by":"publisher","first-page":"1526","DOI":"10.1109\/TASL.2006.878255","article-title":"Enriching speech recognition with automatic detection of sentence boundaries and disfluencies","volume":"14","author":"Liu","year":"2006","journal-title":"IEEE Transactions on Audio, Speech, and Language Processing"},{"issue":"5","key":"2022040614221656200_bib28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0196391","article-title":"The ryerson audio-visual database of emotional speech and song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English","volume":"13","author":"Livingstone","year":"2018","journal-title":"PLOS ONE"},{"key":"2022040614221656200_bib29","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2019-2396","article-title":"Speech model pre-training for end-to-end spoken language understanding","author":"Lugosch","year":"2019","journal-title":"arXiv preprint arXiv:1904.03670"},{"key":"2022040614221656200_bib30","doi-asserted-by":"publisher","first-page":"861","DOI":"10.3115\/1690219.1690267","article-title":"Semantic tagging of web search queries","volume-title":"Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP","author":"Manshadi","year":"2009"},{"key":"2022040614221656200_bib31","doi-asserted-by":"publisher","DOI":"10.1145\/1963192.1963239","article-title":"Unsupervised query segmentation using only query logs","volume-title":"Proceedings of WWW","author":"Mishra","year":"2011"},{"issue":"3","key":"2022040614221656200_bib32","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1109\/MSP.2008.918023","article-title":"Speech segmentation and spoken document processing","volume":"25","author":"Ostendorf","year":"2008","journal-title":"IEEE Signal Processing Magazine"},{"key":"2022040614221656200_bib33","doi-asserted-by":"publisher","first-page":"4418","DOI":"10.18653\/v1\/2021.acl-long.341","article-title":"RADDLE: An evaluation benchmark and analysis platform for robust task-oriented dialog systems","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Peng","year":"2021"},{"key":"2022040614221656200_bib34","doi-asserted-by":"publisher","DOI":"10.3115\/116580.116612","article-title":"Evaluation of spoken language systems: The ATIS domain","volume-title":"Speech and Natural Language: Proceedings of a Workshop","author":"Price","year":"1990"},{"key":"2022040614221656200_bib35","doi-asserted-by":"publisher","first-page":"2383","DOI":"10.18653\/v1\/D16-1264","article-title":"SQuAD: 100,000+ questions for machine comprehension of text","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Rajpurkar","year":"2016"},{"key":"2022040614221656200_bib36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3115\/1622064.1622066","article-title":"Optimizing endpointing thresholds using dialogue features in a spoken dialogue system","volume-title":"Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue","author":"Raux","year":"2008"},{"key":"2022040614221656200_bib37","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.259","article-title":"NoiseQA: Challenge Set Evaluation for User-Centric Question Answering","volume-title":"Conference of the European Chapter of the Association for Computational Linguistics (EACL)","author":"Ravichander","year":"2021"},{"key":"2022040614221656200_bib38","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1162\/tacl_a_00266","article-title":"CoQA: A conversational question answering challenge","volume":"7","author":"Reddy","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022040614221656200_bib39","unstructured":"Schriberg, Elisabeth\n          . 1994. Preliminaries to a theory of speech disfluencies. Technical report, University of California at Berkeley."},{"key":"2022040614221656200_bib40","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461785","article-title":"Towards end-to-end spoken language understanding","author":"Serdyuk","year":"2018","journal-title":"CoRR"},{"key":"2022040614221656200_bib41","doi-asserted-by":"publisher","first-page":"3830","DOI":"10.21437\/Interspeech.2019-2962","article-title":"Contextual recovery of out-of-lattice named entities in automatic speech recognition","volume-title":"Proceedings of Interspeech","author":"Serrino","year":"2019"},{"key":"2022040614221656200_bib42","article-title":"DRCD: A Chinese machine reading comprehension dataset","author":"Shao","year":"2018","journal-title":"arXiv preprint arXiv:1806.00920"},{"key":"2022040614221656200_bib43","doi-asserted-by":"publisher","first-page":"1011","DOI":"10.1145\/2600428.2609497","article-title":"Mobile query reformulations","volume-title":"Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR \u201914","author":"Shokouhi","year":"2014"},{"key":"2022040614221656200_bib44","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1007\/BF00134555","article-title":"Combinatory grammars and parasitic gaps","volume":"5","author":"Steedman","year":"1987","journal-title":"Natural Language and Linguistic Theory"},{"key":"2022040614221656200_bib45","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1145\/3176349.3176387","article-title":"Informing the design of spoken conversational search: Perspective paper","volume-title":"Proceedings of the 2018 Conference on Human Information Interaction and Retrieval, CHIIR \u201918","author":"Trippas","year":"2018"},{"key":"2022040614221656200_bib46","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1145\/3020165.3022144","article-title":"How do people interact in conversational speech-only search tasks: A preliminary analysis","volume-title":"Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR \u201917","author":"Trippas","year":"2017"},{"key":"2022040614221656200_bib47","doi-asserted-by":"publisher","DOI":"10.1002\/9781119992691","volume-title":"Spoken Language Understanding: Systems for Extracting Semantic Information from Speech","author":"Tur","year":"2011"},{"key":"2022040614221656200_bib48","doi-asserted-by":"publisher","first-page":"2579","DOI":"10.21437\/Interspeech.2013-580","article-title":"Semantic parsing using word confusion networks with conditional random fields","volume-title":"INTERSPEECH","author":"T\u00fcr","year":"2013"},{"key":"2022040614221656200_bib49","doi-asserted-by":"publisher","first-page":"2222","DOI":"10.21437\/Interspeech.2018-2453","article-title":"Semantic lattice processing in contextual automatic speech recognition for Google Assistant","volume-title":"Proceedings of Interspeech","author":"Velikovich","year":"2018"},{"key":"2022040614221656200_bib50","unstructured":"Walker, Willie, PaulLamere, PhilipKwok, BhikshaRaj, RitaSingh, EvandroGouvea, PeterWolf, and JoeWoelfel. 2004. Sphinx-4: A flexible open source framework for speech recognition."},{"key":"2022040614221656200_bib51","doi-asserted-by":"publisher","first-page":"63","DOI":"10.18653\/v1\/2020.nlp4convai-1.8","article-title":"Data augmentation for training dialog models robust to speech recognition errors","volume-title":"Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI","author":"Wang","year":"2020"},{"key":"2022040614221656200_bib52","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9053213","article-title":"Joint contextual modeling for ASR correction and language understanding","author":"Weng","year":"2020","journal-title":"CoRR"},{"key":"2022040614221656200_bib53","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2021-120","article-title":"Towards data distillation for end-to-end spoken conversational question answering","author":"You","year":"2021","journal-title":"arXiv preprint arXiv:2010.08923"},{"issue":"3","key":"2022040614221656200_bib54","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1016\/j.websem.2009.07.005","article-title":"From keywords to semantic queries\u2014Incremental query construction on the semantic web","volume":"7","author":"Zenz","year":"2009","journal-title":"Journal of Web Semantics"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/48\/1\/221\/2006612\/coli_a_00430.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/48\/1\/221\/2006612\/coli_a_00430.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,6]],"date-time":"2022-04-06T14:22:54Z","timestamp":1649254974000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/48\/1\/221\/108846\/Revisiting-the-Boundary-between-ASR-and-NLU-in-the"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":54,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,4,4]]},"published-print":{"date-parts":[[2022,4,4]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00430","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}