{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T15:05:17Z","timestamp":1787497517498,"version":"build-2736575974"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2011,5,1]],"date-time":"2011-05-01T00:00:00Z","timestamp":1304208000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/F013930\/1"],"award-info":[{"award-number":["EP\/F013930\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004963","name":"Seventh Framework Programme","doi-asserted-by":"publisher","award":["216594"],"award-info":[{"award-number":["216594"]}],"id":[{"id":"10.13039\/501100004963","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2011,5]]},"abstract":"<jats:p>This article presents a novel algorithm for learning parameters in statistical dialogue systems which are modeled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy that selects the system's responses based on the inferred state; and a reward function that specifies the desired behavior of the system. Ideally both the model parameters and the policy would be designed to maximize the cumulative reward. However, while there are many techniques available for learning the optimal policy, no good ways of learning the optimal model parameters that scale to real-world dialogue systems have been found yet.<\/jats:p>\n                  <jats:p>The presented algorithm, called the Natural Actor and Belief Critic (NABC), is a policy gradient method that offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected cumulative reward. The resulting gradient is then used to adapt both the prior distribution of the dialogue model parameters and the policy parameters. In addition, the article presents a variant of the NABC algorithm, called the Natural Belief Critic (NBC), which assumes that the policy is fixed and only the model parameters need to be estimated. The algorithms are evaluated on a spoken dialogue system in the tourist information domain. The experiments show that model parameters estimated to maximize the expected cumulative reward result in significantly improved performance compared to the baseline hand-crafted model parameters. The algorithms are also compared to optimization techniques using plain gradients and state-of-the-art random search algorithms. In all cases, the algorithms based on the natural gradient work significantly better.<\/jats:p>","DOI":"10.1145\/1966407.1966411","type":"journal-article","created":{"date-parts":[[2011,6,6]],"date-time":"2011-06-06T07:51:38Z","timestamp":1307346698000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Natural actor and belief critic"],"prefix":"10.1145","volume":"7","author":[{"given":"Filip","family":"Jur\u010d\u00ed\u010dek","sequence":"first","affiliation":[{"name":"University of Cambridge, Cambridge, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Blaise","family":"Thomson","sequence":"additional","affiliation":[{"name":"University of Cambridge, Cambridge, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steve","family":"Young","sequence":"additional","affiliation":[{"name":"University of Cambridge, Cambridge, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2011,6,6]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Aberdeen D. A. 2003. Policy-gradient algorithms for partially observable Markov decision processes. Ph.D. thesis Australian National University.  Aberdeen D. A. 2003. Policy-gradient algorithms for partially observable Markov decision processes. Ph.D. thesis Australian National University."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017746"},{"key":"e_1_2_1_3_1","unstructured":"Bishop C. 2006. Pattern Recognition and Machine Learning. Springer.   Bishop C. 2006. Pattern Recognition and Machine Learning. Springer."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324908005032"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1228716.1228726"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102377"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL'09)","author":"Geist M."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue.","author":"Georgila K."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1162\/106365601750190398"},{"key":"e_1_2_1_10_1","first-page":"55","article-title":"Ridge Regression","volume":"12","author":"Hoerl A. E.","year":"1970","journal-title":"Biased Estimation for Nonorthogonal Problems. Technometrics"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of Interspeech.","author":"Jur\u010d\u00ed\u010dek F."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(98)00023-X"},{"key":"e_1_2_1_13_1","unstructured":"Kapadia S. 1998. Discriminative training of hidden Markov models. Ph.D. dissertation. Cambridge University.  Kapadia S. 1998. Discriminative training of hidden Markov models. Ph.D. dissertation. Cambridge University."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of Interspeech.","author":"Kim D."},{"key":"e_1_2_1_15_1","unstructured":"Konda V. and Tsitsiklis J. 2000. Actor-critic algorithms. In Advances in Neural Information Processing Systems 12. MIT Press.  Konda V. and Tsitsiklis J. 2000. Actor-critic algorithms. In Advances in Neural Information Processing Systems 12. MIT Press."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2007.58"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2007.11.026"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2008.02.003"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 3rd IEEE-RAS International Conference on Humanoid Robots.","author":"Peters J."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/11564096_29"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.18626"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of Interspeech.","author":"Raux A."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3115\/1075218.1075231"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding.","author":"Schatzmann J."},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Spall J. C. 2003. Introduction to Stochastic Search and Optimization. John Wiley &amp; Sons Inc. New York NY.   Spall J. C. 2003. Introduction to Stochastic Search and Optimization. John Wiley &amp; Sons Inc. New York NY.","DOI":"10.1002\/0471722138"},{"key":"e_1_2_1_26_1","volume-title":"Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning","author":"Sutton R.","year":"1998"},{"key":"e_1_2_1_27_1","unstructured":"Sutton R. McAllester D. Singh S. and Mansour Y. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12 MIT Press 1057--1063.  Sutton R. McAllester D. Singh S. and Mansour Y. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12 MIT Press 1057--1063."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the Annual Meeting of the Association for Computational Linguistics on Human Language Technologies. 121--124","author":"Syed U."},{"key":"e_1_2_1_29_1","unstructured":"Thomson B. 2010. Statistical methods for spoken dialogue management. Ph.D. thesis University of Cambridge.  Thomson B. 2010. Statistical methods for spoken dialogue management. Ph.D. thesis University of Cambridge."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2009.07.003"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the International Conference on Artificial Neural Networks.","author":"Wierstra D."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1093\/jigpal\/jzp049"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/1564144.1564145"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the AAAI-08 Workshop on Advancements in POMDP Solvers.","author":"Williams J. D.","year":"2008"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).","author":"Williams J. D."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992696"},{"key":"e_1_2_1_37_1","unstructured":"Young S. 2007. CUED standard dialogue acts: http:\/\/mi.eng.cam.ac.uk\/research\/dialogue\/LocalDocs\/dastd.pdf. Tech. rep. Engineering Department Cambridge University.  Young S. 2007. CUED standard dialogue acts: http:\/\/mi.eng.cam.ac.uk\/research\/dialogue\/LocalDocs\/dastd.pdf. Tech. rep. Engineering Department Cambridge University."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2009.04.001"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (UAI).","author":"Zhang B."}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1966407.1966411","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1966407.1966411","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:22:26Z","timestamp":1750231346000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1966407.1966411"}},"subtitle":["Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs"],"short-title":[],"issued":{"date-parts":[[2011,5]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2011,5]]}},"alternative-id":["10.1145\/1966407.1966411"],"URL":"https:\/\/doi.org\/10.1145\/1966407.1966411","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"value":"1550-4875","type":"print"},{"value":"1550-4883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2011,5]]},"assertion":[{"value":"2010-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2010-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-06-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}