{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T15:38:46Z","timestamp":1774539526618,"version":"3.50.1"},"reference-count":48,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"vor","delay-in-days":336,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11,27]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Training a task-oriented dialogue policy using deep reinforcement learning is promising but requires extensive environment exploration. The amount of wasted invalid exploration makes policy learning inefficient. In this paper, we define and argue that dead-end states are important reasons for invalid exploration. When a conversation enters a dead-end state, regardless of the actions taken afterward, it will continue in a dead-end trajectory until the agent reaches a termination state or maximum turn. We propose a Dead-end Detection and Resurrection (DDR) method that detects dead-end states in an efficient manner and provides a rescue action to guide and correct the exploration direction. To prevent dialogue policies from repeating errors, DDR also performs dialogue data augmentation by adding relevant experiences that include dead-end states and penalties into the experience pool. We first validate the dead-end detection reliability and then demonstrate the effectiveness and generality of the method across various domains through experiments on four public dialogue datasets.<\/jats:p>","DOI":"10.1162\/tacl_a_00717","type":"journal-article","created":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T16:32:07Z","timestamp":1733157127000},"page":"1578-1596","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":2,"title":["Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization"],"prefix":"10.1162","volume":"12","author":[{"given":"Yangyang","family":"Zhao","sequence":"first","affiliation":[{"name":"Changsha University of Science and Technology, China. yyz@csust.edu.cn"},{"name":"Utrecht University, the Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mehdi","family":"Dastani","sequence":"additional","affiliation":[{"name":"Utrecht University, the Netherlands, M.M.Dastani@uu.nl"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jinchuan","family":"Long","sequence":"additional","affiliation":[{"name":"Central South University, China. longjc1226@163.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhenyu","family":"Wang","sequence":"additional","affiliation":[{"name":"South China University of Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shihan","family":"Wang","sequence":"additional","affiliation":[{"name":"Utrecht University, the Netherlands. s.wang2@uu.nl"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,11,27]]},"reference":[{"key":"2024120216320180100_bib1","article-title":"Hindsight experience replay","volume":"30","author":"Andrychowicz","year":"2017","journal-title":"NeurIPS"},{"issue":"5","key":"2024120216320180100_bib2","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1006\/ijhc.1997.0185","article-title":"A plan-based model of misunderstandings in cooperative dialogue","volume":"48","author":"Ardissono","year":"1998","journal-title":"International Journal of Human-Computer Studies"},{"key":"2024120216320180100_bib3","doi-asserted-by":"publisher","first-page":"5016","DOI":"10.18653\/v1\/D18-1547","article-title":"Multiwoz - A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling","volume-title":"EMNLP","author":"Budzianowski","year":"2018"},{"key":"2024120216320180100_bib4","doi-asserted-by":"publisher","first-page":"579","DOI":"10.1007\/978-3-540-30120-2_73","article-title":"Rapid dialogue prototyping methodology","volume-title":"Text, Speech and Dialogue, 7th International Conference, TSD 2004, Brno, Czech Republic, September 8\u201311, 2004, Proceedings","author":"Bui","year":"2004"},{"key":"2024120216320180100_bib5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.sigdial-1.40","article-title":"Adaptive dialog policy learning with hindsight and user modeling","author":"Cao","year":"2020","journal-title":"arXiv preprint arXiv:2005.03299"},{"key":"2024120216320180100_bib6","doi-asserted-by":"publisher","first-page":"198","DOI":"10.18653\/v1\/E17-2032","article-title":"On-line dialogue policy learning with companion teaching","volume-title":"EACL","author":"Chen","year":"2017"},{"key":"2024120216320180100_bib7","doi-asserted-by":"publisher","first-page":"2454","DOI":"10.18653\/v1\/D17-1260","article-title":"Agent-aware dropout dqn for safe and efficient on-line dialogue policy learning","volume-title":"EMNLP","author":"Chen","year":"2017"},{"key":"2024120216320180100_bib8","doi-asserted-by":"publisher","first-page":"1431","DOI":"10.1145\/3404835.3462913","article-title":"Unified conversational recommendation policy learning via graph-based reinforcement learning","volume-title":"SIGIR \u201921: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11\u201315, 2021","author":"Deng","year":"2021"},{"key":"2024120216320180100_bib9","volume-title":"The Psychology of Consciousness","author":"William Farthing","year":"1992"},{"key":"2024120216320180100_bib10","doi-asserted-by":"publisher","first-page":"101","DOI":"10.18653\/v1\/W16-3613","article-title":"Policy networks with two-stage training for dialogue systems","volume-title":"Proceedings of the SIGDIAL 2016 Conference, The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 13\u201315 September 2016, Los Angeles, CA, USA","author":"Fatemi","year":"2016"},{"key":"2024120216320180100_bib11","article-title":"Dead-ends and secure exploration in reinforcement learning","volume-title":"ICML","author":"Fatemi","year":"2019"},{"key":"2024120216320180100_bib12","doi-asserted-by":"publisher","DOI":"10.1109\/ASRU51503.2021.9687856","article-title":"What does the user want? Information gain for hierarchical dialogue policy optimisation","volume-title":"ASRU","author":"Geishauser","year":"2021"},{"key":"2024120216320180100_bib13","first-page":"5438","article-title":"Off-policy evaluation via off-policy classification","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\u201314, 2019, Vancouver, BC, Canada","author":"Irpan","year":"2019"},{"key":"2024120216320180100_bib14","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1613\/jair.301","article-title":"Reinforcement learning: A survey","volume":"4","author":"Kaelbling","year":"1996","journal-title":"Journal of artificial intelligence research"},{"issue":"1","key":"2024120216320180100_bib15","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1093\/biomet\/70.1.163","article-title":"Information gain and a general measure of correlation","volume":"70","author":"Kent","year":"1983","journal-title":"Biometrika"},{"key":"2024120216320180100_bib16","doi-asserted-by":"publisher","first-page":"1386","DOI":"10.1145\/3543507.3583536","article-title":"Confident action decision via hierarchical policy learning for conversational recommendation","volume-title":"Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 \u2013 4 May 2023","author":"Kim","year":"2023"},{"key":"2024120216320180100_bib17","first-page":"438","article-title":"A theory of goal-oriented mdps with dead ends","volume-title":"Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, August 14\u201318, 2012","author":"Kolobov","year":"2012"},{"key":"2024120216320180100_bib18","article-title":"A survey on recent advances and challenges in reinforcement learning methods for task-oriented dialogue policy learning","author":"Kwan","year":"2022","journal-title":"arXiv preprint arXiv:2202.13675"},{"key":"2024120216320180100_bib19","doi-asserted-by":"publisher","DOI":"10.1016\/0196-8858(85)90002-8","article-title":"Asymptotically efficient adaptive allocation rules","volume":"6","author":"Lai","year":"1985","journal-title":"Advanced in Applied Mathematics"},{"key":"2024120216320180100_bib20","doi-asserted-by":"publisher","first-page":"2073","DOI":"10.1145\/3394486.3403258","article-title":"Interactive path reasoning on graph for conversational recommendation","volume-title":"KDD \u201920: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23\u201327, 2020","author":"Lei","year":"2020"},{"key":"2024120216320180100_bib21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1259","article-title":"Deal or no deal? End-to-end learning for negotiation dialogues","author":"Lewis","year":"2017","journal-title":"CoRR"},{"key":"2024120216320180100_bib22","article-title":"A user simulator for task-completion dialogues","author":"Li","year":"2016","journal-title":"arXiv preprint arXiv:1612.05688"},{"key":"2024120216320180100_bib23","article-title":"Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems","author":"Li","year":"2018","journal-title":"arXiv preprint arXiv:1807.11125"},{"key":"2024120216320180100_bib24","doi-asserted-by":"publisher","first-page":"447","DOI":"10.18653\/v1\/2021.findings-acl.39","article-title":"Retrieve & memorize: Dialog policy learning with multi-action memory","volume-title":"Findings of the ACL","author":"Li","year":"2021"},{"key":"2024120216320180100_bib25","article-title":"Continuous control with deep reinforcement learning","author":"Lillicrap","year":"2015","journal-title":"arXiv preprint arXiv:1509.02971"},{"key":"2024120216320180100_bib26","doi-asserted-by":"publisher","first-page":"1091","DOI":"10.18653\/v1\/2021.findings-acl.94","article-title":"Scheduled dialog policy learning: An automatic curriculum learning framework for task-oriented dialog system","volume-title":"Findings of the Association for Computational Linguistics: ACL\/IJCNLP","author":"Liu","year":"2021"},{"key":"2024120216320180100_bib27","doi-asserted-by":"publisher","first-page":"2596","DOI":"10.1609\/aaai.v33i01.33012596","article-title":"Goal-oriented dialogue policy learning from failures","volume-title":"AAAI","author":"Keting","year":"2019"},{"key":"2024120216320180100_bib28","doi-asserted-by":"publisher","first-page":"60","DOI":"10.3115\/1654595.1654609","article-title":"Tracing actions helps in understanding interactions","volume-title":"Proceedings of the SIGDIAL 2006 Workshop, The 7th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 15\u201316 July 2006, Sydney, Australia","author":"Ludwig","year":"2006"},{"issue":"7540","key":"2024120216320180100_bib29","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"2024120216320180100_bib30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2022.sigdial-1.1","article-title":"Post-processing networks: Method for optimizing pipeline task-oriented dialogue systems using reinforcement learning","volume-title":"Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2022, Edinburgh, UK, 07\u201309 September 2022","author":"Ohashi","year":"2022"},{"key":"2024120216320180100_bib31","doi-asserted-by":"publisher","first-page":"13604","DOI":"10.1609\/aaai.v35i15.17604","article-title":"Dialog policy learning for joint clarification and active learning queries","volume-title":"AAAI","author":"Padmakumar","year":"2021"},{"key":"2024120216320180100_bib32","doi-asserted-by":"publisher","first-page":"6149","DOI":"10.1109\/ICASSP.2018.8461918","article-title":"Adversarial advantage actor-critic model for task-completion dialogue policy learning","volume-title":"ICASSP","author":"Peng","year":"2018"},{"key":"2024120216320180100_bib33","doi-asserted-by":"publisher","first-page":"748","DOI":"10.1109\/TASLP.2023.3235202","article-title":"Hierarchical reinforcement learning with guidance for multi-domain dialogue policy","volume":"31","author":"Rohmatillah","year":"2023","journal-title":"IEEE ACM Transactions on Audio, Speech, and Language Processing"},{"key":"2024120216320180100_bib34","article-title":"Generalization in reinforcement learning: Successful examples using sparse coarse coding","volume":"8","author":"Sutton","year":"1995","journal-title":"NeurIPS"},{"key":"2024120216320180100_bib35","doi-asserted-by":"publisher","first-page":"625","DOI":"10.18653\/v1\/2020.acl-main.59","article-title":"Multi-agent task-oriented dialog policy learning with role-aware reward decomposition","volume-title":"ACL","author":"Takanobu","year":"2020"},{"key":"2024120216320180100_bib36","doi-asserted-by":"publisher","first-page":"2298","DOI":"10.18653\/v1\/D18-1253","article-title":"Subgoal discovery for hierarchical dialogue policy learning","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 \u2013 November 4, 2018","author":"Da","year":"2018"},{"issue":"3","key":"2024120216320180100_bib37","doi-asserted-by":"publisher","first-page":"1019","DOI":"10.1007\/s12559-022-09996-0","article-title":"A review of plan-based approaches for dialogue management","volume":"14","author":"Teixeira","year":"2022","journal-title":"Cognitive Computing"},{"key":"2024120216320180100_bib38","article-title":"Deep reinforcement learning with double q-learning","author":"van Hasselt","year":"2015","journal-title":"CoRR"},{"key":"2024120216320180100_bib39","doi-asserted-by":"publisher","first-page":"6355","DOI":"10.18653\/v1\/2020.acl-main.566","article-title":"Learning efficient dialogue policy from demonstrations through shaping","volume-title":"ACL","author":"Wang","year":"2020"},{"key":"2024120216320180100_bib40","first-page":"1995","article-title":"Dueling network architectures for deep reinforcement learning","volume-title":"Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19\u201324, 2016","author":"Wang","year":"2016"},{"key":"2024120216320180100_bib41","doi-asserted-by":"publisher","first-page":"563","DOI":"10.18653\/v1\/2021.emnlp-main.44","article-title":"Open-domain clarification question generation without question examples","volume-title":"EMNLP","author":"White","year":"2021"},{"key":"2024120216320180100_bib42","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Machine Learning"},{"key":"2024120216320180100_bib43","doi-asserted-by":"publisher","first-page":"1786","DOI":"10.18653\/v1\/2021.findings-acl.156","article-title":"Gaussian process based deep dyna-q approach for dialogue policy learning","volume-title":"Findings of the ACL","author":"Guanlin","year":"2021"},{"key":"2024120216320180100_bib44","doi-asserted-by":"publisher","first-page":"3420","DOI":"10.18653\/v1\/2021.naacl-main.267","article-title":"Clipping loops for sample-efficient dialogue policy optimisation","volume-title":"NAACL","author":"Yen-Chen","year":"2021"},{"key":"2024120216320180100_bib45","article-title":"Tam: Using trainable-action-mask to improve sample- efficiency in reinforcement learning for dialogue systems","volume-title":"NeurIPS","author":"Yen-Chen","year":"2019"},{"key":"2024120216320180100_bib46","first-page":"1208","article-title":"Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers)","author":"Zhao","year":"2019"},{"key":"2024120216320180100_bib47","doi-asserted-by":"publisher","first-page":"9676","DOI":"10.1609\/aaai.v34i05.6516","article-title":"Dynamic reward-based dueling deep dyna-q: Robust policy learning in noisy environments","volume-title":"AAAI","author":"Zhao","year":"2020"},{"key":"2024120216320180100_bib48","doi-asserted-by":"publisher","first-page":"4311","DOI":"10.18653\/v1\/2021.emnlp-main.354","article-title":"Efficient dialogue complementary policy learning via deep q-network policy and episodic memory policy","volume-title":"EMNLP","author":"Zhao","year":"2021"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00717\/2482041\/tacl_a_00717.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00717\/2482041\/tacl_a_00717.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T16:32:14Z","timestamp":1733157134000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00717\/125484\/Rescue-Conversations-from-Dead-ends-Efficient"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":48,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00717","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}