{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T15:06:26Z","timestamp":1791299186476,"version":"4.1.0"},"reference-count":55,"publisher":"MIT Press","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2014,12]]},"abstract":"<jats:p>We address the problem of dynamically modeling and adapting to unknown users in resource-scarce domains in the context of interactive spoken dialogue systems. As an example, we show how a system can learn to choose referring expressions to refer to domain entities for users with different levels of domain expertise, and whose domain knowledge is initially unknown to the system. We approach this problem using a three step process: collecting data using a Wizard-of-Oz method, building simulated users, and learning to model and adapt to users using Reinforcement Learning techniques.<\/jats:p><jats:p>We show that by using only a small corpus of non-adaptive dialogues and user knowledge profiles it is possible to learn an adaptive user modeling policy using a sense-predict-adapt approach. Our evaluation results show that the learned user modeling and adaptation strategies performed better in terms of adaptation than some simple hand-coded baseline policies, with both simulated and real users. With real users, the learned policy produced around a 20% increase in adaptation in comparison to an adaptive hand-coded baseline. We also show that adaptation to users' domain knowledge results in improving task success (99.47% for the learned policy vs. 84.7% for a hand-coded baseline) and reducing dialogue time of the conversation (11% relative difference). We also compared the learned policy to a variety of carefully hand-crafted adaptive policies that employ the user knowledge profiles to adapt their choices of referring expressions throughout a conversation. We show that the learned policy generalises better to unseen user profiles than these hand-coded policies, while having comparable performance on known user profiles.<\/jats:p><jats:p>We discuss the overall advantages of this method and how it can be extended to other levels of adaptation such as content selection and dialogue management, and to other domains where adapting to users' domain knowledge is useful, such as travel and healthcare.<\/jats:p>","DOI":"10.1162\/coli_a_00203","type":"journal-article","created":{"date-parts":[[2014,6,18]],"date-time":"2014-06-18T15:12:36Z","timestamp":1403104356000},"page":"883-920","source":"Crossref","is-referenced-by-count":29,"title":["Adaptive Generation in Dialogue Systems Using Dynamic User Modeling"],"prefix":"10.1162","volume":"40","author":[{"given":"Srinivasan","family":"Janarthanam","sequence":"first","affiliation":[{"name":"Heriot-Watt University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oliver","family":"Lemon","sequence":"additional","affiliation":[{"name":"Heriot-Watt University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"R1","doi-asserted-by":"crossref","unstructured":"Ai, H. and D. Litman. 2007. Knowledge consistent user simulations for dialog systems. In Proceedings of Interspeech 2007, pages 2,697\u20132,700, Antwerp.","DOI":"10.21437\/Interspeech.2007-708"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.3115\/991250.991348"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1017\/S004740450001037X"},{"key":"R5","unstructured":"Bickmore, T. and T. Giorgino. 2004. Some novel aspects of health communication from a dialogue systems perspective. In AAAI Fall Symposium on Dialogue Systems for Health Communication, pages 275\u2013291, Washington, DC."},{"key":"R6","unstructured":"Boye, J. 2007. Dialogue management for automatic troubleshooting and other problem-solving applications. In Proceedings SIGDial'07, pages 247\u2013255, Antwerp."},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1007\/BF01257890"},{"key":"R8","unstructured":"Cheng, H., H. Bratt, R. Mishra, E. Shriberg, S. Upson, J. Chen, F. Weng, S. Peters, L. Cavedon, and J. Niekrasz. 2004. A Wizard of Oz framework for collecting spoken human-computer dialogs. In Proceedings of the International Conference on Spoken Language Processing, pages 2,269\u20132,272, Jeju."},{"key":"R10","unstructured":"Cuayahuitl, H. 2009. Hierarchical Reinforcement Learning for Spoken Dialogue Systems. Ph.D. thesis, University of Edinburgh."},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Cuayahuitl, H., S. Renals, O. Lemon, and H. Shimodaira. 2005. Human-computer dialogue simulation using hidden Markov models. In Proceedings of ASRU 2005, Cancun.","DOI":"10.1109\/ASRU.2005.1566485"},{"key":"R12","unstructured":"Dale, R. 1988. Generating Referring Expressions in a Domain of Objects and Processes. Ph.D. thesis, University of Edinburgh."},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00064"},{"key":"R14","unstructured":"Dethlefs, N. and H. Cuayahuitl. 2010. Hierarchical reinforcement learning for adaptive text generation. In Proceedings of the 6th International Natural Language Generation Conference, pages 37\u201345, Dublin."},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Dzikovska, M. O., C. Callaway, E. Farrow, M. Marques-Pita, C. Matheson, and J. D. Moore. 2007. Adaptive tutorial dialogue systems using deep NLP techniques. In Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics 2007, pages 5\u20136, Morristown, NJ.","DOI":"10.3115\/1614164.1614167"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Eckert, W., E. Levin, and R. Pieraccini. 1997. User modeling for spoken dialogue system evaluation. In Proceedings of ASRU 1997, pages 80\u201387, Santa Barbara, CA.","DOI":"10.1109\/ASRU.1997.658991"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2009.12.002"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/0885-2308(91)90019-M"},{"key":"R19","doi-asserted-by":"crossref","unstructured":"Ga\u0161i\u0107, M., C. Breslin, M. Henderson, D. Kim, M. Szummer, B. Thomson, P. Tsiakoulis, and S. Young. 2013. On-line policy optimisation of Bayesian spoken dialogue systems via human interaction. In Proceedings of ICASSP 2013, Canada.","DOI":"10.1109\/ICASSP.2013.6639297"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Gatt, A. and A. Belz. 2008. Attribute selection for referring expression generation: New algorithms and evaluation methods. In Proceedings of INLG-2008, pages 50\u201358, Salt Fork, OH.","DOI":"10.3115\/1708322.1708333"},{"key":"R21","unstructured":"Gatt, A. and K. van Deemter. 2009. Generating plural NPs in discourse: Evidence from the GNOME corpus. In Proceedings of the Workshop on Production of Referring Expressions: Bridging Computational and Psycholinguistic Approaches (PRE-CogSci-09), Amsterdam."},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Georgila, K., J. Henderson, and O. Lemon. 2005. Learning user simulations for information state update dialogue systems. In Proceedings of Eurospeech\/Interspeech, pages 893\u2013896, Lisbon.","DOI":"10.21437\/Interspeech.2005-401"},{"key":"R23","unstructured":"Grice, H. P. 1975. Logic and conversation. Syntax and Semantics: Vol 3, Speech Acts, pages 43\u201358."},{"key":"R24","doi-asserted-by":"crossref","unstructured":"Hajdinjak, M. and F. Miheli. 2003. The Wizard of Oz system for weather information retrieval. In Proceedings of the 6th International Conference TSD, pages 400\u2013405, Czech Republic.","DOI":"10.1007\/978-3-540-39398-6_57"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2008.07-028-R2-05-82"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1037\/1076-898X.5.2.205"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1037\/0096-3445.116.1.26"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Janarthanam, S. 2011. Learning User Modeling Strategies for Adaptive Referring Expression Generation in Spoken Dialogue Systems. Ph.D. thesis, University of Edinburgh.","DOI":"10.1007\/978-3-642-15573-4_4"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.3115\/1610195.1610209"},{"key":"R30","unstructured":"Janarthanam, S., O. Lemon, P. Bartie, T. Dalmas, A. Dickinson, X. Liu, J. Mackaness, and W. Goetze. 2013. Evaluating a city exploration dialogue system with integrated question-answering and pedestrian navigation. In Proceedings of ACL 2013, pages 1,660\u20131,668."},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1613\/jair.301"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1007\/BF00141081"},{"key":"R33","unstructured":"Keizer, S., M. Ga\u0161i\u0107, F. Jurcicek, F. Mairesse, B. Thomson, K. Yu, and S. Young. 2010. Parameter estimation for agenda-based user simulation. In Proceedings of SIGDial 2010, pages 116\u2013123, Tokyo."},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.1991.tb00403.x"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1162\/089120103321337430"},{"key":"R39","unstructured":"Kullback, S. 1987. Letter to the Editor: The Kullback-Leibler distance. The American Statistician, 41(4):340\u2013341."},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729694"},{"key":"R41","unstructured":"Lemon, O. 2008. Adaptive natural language generation in dialogue using reinforcement learning. In Proceedings of SEMdial'08, pages 149\u2013156, London."},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2006.326774"},{"key":"R43","unstructured":"Levin, E., R. Pieraccini, and W. Eckert. 1997. Learning dialogue strategies within the Markov decision process framework. In Proceedings of ASRU 1997, pages 72\u201379, Santa Barbara, CA."},{"key":"R44","unstructured":"Lopez-Cozar, R., Z. Callejas, M. Gea, and G. Montoro. 2005. Multimodal, multilingual and adaptive dialogue system for ubiquitous interaction in educational space. In Proceedings of Applied Spoken Language Interaction in Distributed Environments (ASIDE), Aalborg."},{"key":"R45","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-010-9076-2"},{"key":"R46","doi-asserted-by":"crossref","unstructured":"McKeown, K., J. Robin, and M. Tanenblatt. 1993. Tailoring lexical choice to the user's vocabulary in multimedia explanation generation. In Proceedings ACL 1993, pages 226\u2013234, Columbus, OH.","DOI":"10.3115\/981574.981605"},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Nguyen, L. and P. Do. 2009. Combination of Bayesian network and overlay model in user modeling. InProceedings of the 9th International Conference on Computational Science, volume 5545\/2009 of LNCS, pages 5\u201314, Baton Rouge, LA.","DOI":"10.1007\/978-3-642-01973-9_2"},{"key":"R48","unstructured":"Paris, C. L. 1984. Determining the level of expertise. In First Annual Workshop on Theoretical Issues in Conceptual Information Processing, Atlanta, GA."},{"key":"R49","unstructured":"Rambow, O. 1990. Domain communication knowledge. In Proceedings of the Fifth International Workshop on Natural Language Generation 1990, pages 87\u201394, Dawson, PA."},{"key":"R50","doi-asserted-by":"crossref","unstructured":"Reiter, E. and R. Dale. 1992. A fast algorithm for the generation of referring expressions. In Proceedings COLING-1992, pages 232\u2013238, Nantes.","DOI":"10.3115\/992066.992105"},{"key":"R51","unstructured":"Reiter, E. and R. Dale. 1995. Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Science, 18:233\u2013263."},{"key":"R53","doi-asserted-by":"crossref","unstructured":"Rieser, V. and O. Lemon. 2009. Natural language generation as planning under uncertainty for spoken dialogue systems. In Proceedings EACL 2009, pages 638\u2013691, Athens.","DOI":"10.3115\/1609067.1609143"},{"key":"R54","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00038"},{"key":"R55","doi-asserted-by":"crossref","unstructured":"Rogers, S., C. Fiechter, and C. Thompson. 2000. Adaptive user interfaces for automotive environments. In IEEE Intelligent Vehicles Symposium, pages 662\u2013667, Dearborn, MI.","DOI":"10.1109\/IVS.2000.898424"},{"key":"R56","doi-asserted-by":"crossref","unstructured":"Schatzmann, J., B. Thomson, K. Weilhammer, H. Ye, and S. J. Young. 2007. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In Proceedings of HLT\/NAACL 2007, pages 149\u2013152, Rochester, NY.","DOI":"10.3115\/1614108.1614146"},{"key":"R57","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888906000944"},{"key":"R58","unstructured":"Shapiro, D. and P. Langley. 2002. Separating skills from preference: Using learning to program by reward. In Proceedings ICML-02, pages 570\u2013577, Sydney."},{"key":"R59","doi-asserted-by":"publisher","DOI":"10.1049\/cp:20070418"},{"key":"R62","unstructured":"Whittaker, S., M. Walker, and J. Moore. 2002. Fish or fowl: A Wizard of Oz evaluation of dialogue strategies in the restaurant domain. In Language Resources and Evaluation Conference, pages 1,602\u20131,609, Las Palmas."},{"key":"R63","unstructured":"Williams, J. and S. J. Young. 2003. Using Wizard-of-Oz simulations to bootstrap reinforcement learning based dialogue management systems. In Proceedings SIGdial'03, pages 135\u2013139, Sapporo."}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00203","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,3]],"date-time":"2025-05-03T11:14:45Z","timestamp":1746270885000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/40\/4\/883-920\/1487"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,12]]},"references-count":55,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2014,12]]}},"alternative-id":["10.1162\/COLI_a_00203"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00203","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,12]]}}}