{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T06:22:40Z","timestamp":1763619760607,"version":"3.41.0"},"reference-count":56,"publisher":"Cambridge University Press (CUP)","issue":"3","license":[{"start":{"date-parts":[[2009,7,1]],"date-time":"2009-07-01T00:00:00Z","timestamp":1246406400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2009,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n\t  <jats:p>Richly annotated dialogue corpora are essential for new research directions in statistical learning approaches to dialogue management, context-sensitive interpretation, and context-sensitive speech recognition. In particular, large dialogue corpora annotated with contextual information and speech acts are urgently required. We explore how existing dialogue corpora (usually consisting of utterance transcriptions) can be automatically processed to yield new corpora where dialogue context and speech acts are accurately represented. We present a conceptual and computational framework for generating such corpora. As an example, we present and evaluate an automatic annotation system which builds \u2018Information State Update\u2019 (ISU) representations of dialogue context for the <jats:sc>Communicator<\/jats:sc> (2000 and 2001) corpora of human\u2013machine dialogues (2,331 dialogues). The purposes of this annotation are to generate corpora for reinforcement learning of dialogue policies, for building user simulations, for evaluating different dialogue strategies against a baseline, and for training models for context-dependent interpretation and speech recognition. The automatic annotation system parses system and user utterances into speech acts and builds up sequences of dialogue context representations using an ISU dialogue manager. We present the architecture of the automatic annotation system and a detailed example to illustrate how the system components interact to produce the annotations. We also evaluate the annotations, with respect to the task completion metrics of the original corpus and in comparison to hand-annotated data and annotations produced by a baseline automatic system. The automatic annotations perform well and largely outperform the baseline automatic annotations in all measures. The resulting annotated corpus has been used to train high-quality user simulations and to learn successful dialogue strategies. The final corpus will be made publicly available.<\/jats:p>","DOI":"10.1017\/s1351324909005105","type":"journal-article","created":{"date-parts":[[2009,6,4]],"date-time":"2009-06-04T10:02:19Z","timestamp":1244109739000},"page":"315-353","source":"Crossref","is-referenced-by-count":9,"title":["Automatic annotation of context and speech acts for dialogue corpora"],"prefix":"10.1017","volume":"15","author":[{"given":"KALLIRROI","family":"GEORGILA","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"OLIVER","family":"LEMON","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"JAMES","family":"HENDERSON","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"JOHANNA D.","family":"MOORE","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2009,7,1]]},"reference":[{"key":"S1351324909005105_manual_ref-3","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010091302035"},{"key":"S1351324909005105_manual_ref-30","doi-asserted-by":"crossref","unstructured":"Rieser, V. , Kruijff-Korbayov\u00e1, I. , and Lemon, O. 2005a. A corpus collection and annotation framework for learning multimodal clarification strategies. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, Lisbon, Portugal, pp. 97\u2013106.","DOI":"10.18653\/v1\/2005.sigdial-1.11"},{"key":"S1351324909005105_manual_ref-1","doi-asserted-by":"crossref","unstructured":"Andreani, G. , Di Fabbrizio, G. , Gilbert, M. , Gillick, D. , Hakkani-T\u00fcr, D. , and Lemon, O. 2006. Let's DiSCoH: collecting an annotated open corpus with dialogue acts and reward signals for natural language helpdesks. In Proceedings of the IEEE\/ACL Workshop on Spoken Language Technology, Aruba, 2006, pp. 218\u201321.","DOI":"10.1109\/SLT.2006.326794"},{"key":"S1351324909005105_manual_ref-19","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324900002539"},{"key":"S1351324909005105_manual_ref-22","doi-asserted-by":"publisher","DOI":"10.1145\/1017494.1017496"},{"key":"S1351324909005105_manual_ref-7","doi-asserted-by":"crossref","unstructured":"Gabsdil, M. , and Lemon, O. 2004. Combining acoustic and pragmatic features to predict recognition performance in spoken dialogue systems. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL), Barcelona, Spain, pp. 344\u201351.","DOI":"10.3115\/1218955.1218999"},{"key":"S1351324909005105_manual_ref-21","doi-asserted-by":"crossref","unstructured":"Lemon, O. , Georgila, K. , Henderson, J. , and Stuttle, M. 2006b. An ISU dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the TALK in-car system. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Trento, Italy, pp. 119\u201322.","DOI":"10.3115\/1608974.1608986"},{"key":"S1351324909005105_manual_ref-33","unstructured":"Rieser, V. , and Lemon, O. 2008. Learning effective multimodal dialogue strategies from Wizard-of-Oz data: Bootstrapping and evaluation. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL\u2013HLT), Columbus, OH, pp. 638\u201346."},{"key":"S1351324909005105_manual_ref-15","unstructured":"Henderson, J. , Lemon, O. , and Georgila, K. 2005. Hybrid reinforcement\/supervised learning for dialogue policies from COMMUNICATOR data. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialogue Systems, International Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, UK, pp. 68\u201375."},{"key":"S1351324909005105_manual_ref-34","doi-asserted-by":"crossref","unstructured":"Samuel, K. , Carberry, S. , and Vijay-Shanker, K. 1998. Dialogue act tagging with transformation-based learning. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics (ACL\u2013COLING), Montreal, Quebec, Canada, pp. 1150\u20136.","DOI":"10.3115\/980691.980757"},{"key":"S1351324909005105_manual_ref-35","doi-asserted-by":"crossref","unstructured":"Schatzmann, J. , Georgila, K. , and Young, S. 2005a Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, Lisbon, Portugal, pp. 45\u201354.","DOI":"10.18653\/v1\/2005.sigdial-1.6"},{"key":"S1351324909005105_manual_ref-4","doi-asserted-by":"publisher","DOI":"10.1037\/10096-006"},{"volume-title":"Logic, Language, and Computation","year":"1996","author":"Ginzburg","key":"S1351324909005105_manual_ref-13"},{"key":"S1351324909005105_manual_ref-42","doi-asserted-by":"publisher","DOI":"10.1162\/089120100561737"},{"key":"S1351324909005105_manual_ref-43","unstructured":"Traum, D. 1994. A computational theory of grounding in natural language conversation. PhD Thesis, Department of Computer Science, University of Rochester."},{"key":"S1351324909005105_manual_ref-2","unstructured":"Bos, J. , Klein, E. , Lemon, O. , and Oka, T. 2003. DIPPER: description and formalisation of an Information-State Update dialogue system architecture. In Proceedings of the 4th SIGdial Workshop on Discourse and Dialogue, Sapporo, Japan, pp. 115\u201324."},{"key":"S1351324909005105_manual_ref-37","doi-asserted-by":"crossref","unstructured":"Schatzmann, J. , Thomson, B. , and Young, S. 2007. Statistical user simulation with a hidden agenda. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue, Antwerp, Belgium, pp. 273\u201382.","DOI":"10.18653\/v1\/2007.sigdial-1.48"},{"key":"S1351324909005105_manual_ref-50","doi-asserted-by":"crossref","unstructured":"Walker, M. , and Passonneau, R. 2001. DATE: a dialogue act tagging scheme for evaluation of spoken dialogue systems. In Proceedings of the Human Language Technologies Conference, San Diego, CA, pp. 1\u20138.","DOI":"10.3115\/1072133.1072148"},{"key":"S1351324909005105_manual_ref-51","doi-asserted-by":"crossref","unstructured":"Walker, M. , Rudnicky, A. , Aberdeen, J. , Bratt, E. , Garofolo, J. , Hastie, H. , Le, A. , Pellom, B. , Potamianos, A. , Passonneau, R. , Prasad, R. , Roukos, S. , Sanders, G. , Seneff, S. , Stallard, D. , and Whittaker, S. 2002. DARPA communicator evaluation: progress from 2000 to 2001. In Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP), Denver, CO, pp. 273\u20136.","DOI":"10.21437\/ICSLP.2002-133"},{"key":"S1351324909005105_manual_ref-14","first-page":"175","article-title":"Attention, intentions, and the structure of discourse","volume":"12","author":"Grosz","year":"1986","journal-title":"Computational Linguistics"},{"key":"S1351324909005105_manual_ref-29","doi-asserted-by":"crossref","unstructured":"Ries, K. 1999. HMM and neural network based speech act detection. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Phoenix, AZ, pp. 497\u2013500.","DOI":"10.1109\/ICASSP.1999.758171"},{"key":"S1351324909005105_manual_ref-39","unstructured":"Scheffler, K. , and Young, S. 2001. Corpus-based dialogue simulation for automatic strategy learning and evaluation. In Proceedings of the Workshop on Adaptation in Dialogue Systems, North American Chapter of the Association for Computational Linguistics (NAACL), Pittsburgh, PA, pp. 64\u201370."},{"key":"S1351324909005105_manual_ref-44","doi-asserted-by":"publisher","DOI":"10.1093\/jos\/17.1.7"},{"key":"S1351324909005105_manual_ref-46","doi-asserted-by":"crossref","unstructured":"Walker, M. , Aberdeen, J. , Boland, J. , Bratt, E. , Garofolo, J. , Hirschman, L. , Le, A. , Lee, S. , Narayanan, S. , Papineni, K. , Pellom, B. , Polifroni, J. , Potamianos, A. , Prabhu, P. , Rudnicky, A. , Sanders, G. , Seneff, S. , Stallard, D. , and Whittaker, S. 2001a. DARPA Communicator dialog travel planning systems: The June 2000 data collection. In Proceedings of the 7th European Conference on Speech Communication and Technology (EUROSPEECH), Aalborg, Denmark, pp. 1371\u20134.","DOI":"10.21437\/Eurospeech.2001-355"},{"key":"S1351324909005105_manual_ref-48","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324900002503"},{"key":"S1351324909005105_manual_ref-24","doi-asserted-by":"publisher","DOI":"10.1109\/89.817450"},{"key":"S1351324909005105_manual_ref-27","doi-asserted-by":"crossref","unstructured":"Reithinger, N. , and Klesen, M. 1997. Dialogua act classification using language models. In Proceedings of the 5th European Conference on Speech Communication and Technology (EUROSPEECH), Rhodes, Greece, pp. 2235\u20138.","DOI":"10.21437\/Eurospeech.1997-589"},{"key":"S1351324909005105_manual_ref-40","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139173438"},{"key":"S1351324909005105_manual_ref-49","doi-asserted-by":"crossref","unstructured":"Walker, M. A. , Passonneau, R. J. , and Boland, J. E. 2001b. Quantitative and qualitative evaluation of DARPA Communicator spoken dialogue systems. In Proceedings of the 39th Meeting of the Association for Computational Linguistics (ACL), Toulouse, France, pp. 515\u201322.","DOI":"10.3115\/1073012.1073078"},{"key":"S1351324909005105_manual_ref-11","unstructured":"Georgila, K. , Wolters, M. , Karaiskos, V. , Kronenthal, M. , Logie, R. , Mayo, N. , Moore, J. D. , and Watson, M. 2008a. A fully annotated corpus for studying the effect of cognitive ageing on users' interactions with spoken dialogue systems. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Marrakech, Morocco, pp. 938\u201344."},{"key":"S1351324909005105_manual_ref-28","doi-asserted-by":"crossref","unstructured":"Reithinger, N. , and Maier, E. 1995. Utilizing statistical dialogue act processing in VERBMOBIL. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL), Cambridge, MA, pp. 116\u201321.","DOI":"10.3115\/981658.981674"},{"key":"S1351324909005105_manual_ref-32","doi-asserted-by":"crossref","unstructured":"Rieser, V. , and Lemon, O. 2006. Using machine learning to explore human multimodal clarification strategies. In Proceedings of the 44th Annual Meeting of the Association for Computational Linguistics (ACL), Sydney, Australia, pp. 659\u201366.","DOI":"10.3115\/1273073.1273158"},{"key":"S1351324909005105_manual_ref-8","doi-asserted-by":"crossref","unstructured":"Georgila, K. , Henderson, J. , and Lemon, O. 2005a. Learning user simulations for Information State Update dialogue systems. In Proceedings of the 9th European Conference on Speech Communication and Technology (INTERSPEECH\u2013EUROSPEECH), Lisbon, Portugal, pp. 893\u20136.","DOI":"10.21437\/Interspeech.2005-401"},{"key":"S1351324909005105_manual_ref-18","unstructured":"Kipp, M. 1998 The neural path to dialogue acts. In Proceedings of the 13th European Conference on Artificial Intelligence (ECAI), Brighton, UK, pp. 175\u20139."},{"key":"S1351324909005105_manual_ref-45","doi-asserted-by":"crossref","unstructured":"Traum, D. R. , and Allen, J. 1994. Discourse obligations in dialogue processing. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, Las Cruces, NM, pp. 1\u20138.","DOI":"10.3115\/981732.981733"},{"key":"S1351324909005105_manual_ref-53","doi-asserted-by":"crossref","unstructured":"Williams, J. D. , and Young, S. 2005. Scaling up POMDPs for dialog management: the \u201csummary POMDP\u201d method. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, San Juan, Puerto Rico, pp. 177\u201382.","DOI":"10.1109\/ASRU.2005.1566498"},{"key":"S1351324909005105_manual_ref-56","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-47987-2_59"},{"key":"S1351324909005105_manual_ref-23","unstructured":"Lesch, S. , Kleinbauer, T. , and Alexandersson, J. 2005. A new metric for the evaluation of dialog act classification. In Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL: DIALOR), Nancy, France, pp. 143\u20136."},{"key":"S1351324909005105_manual_ref-10","unstructured":"Georgila, K. , Lemon, O. , and Henderson, J. 2005b. Automatic annotation of COMMUNICATOR dialogue data for learning dialogue strategies and user simulations. In Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL: DIALOR), Nancy, France, pp. 61\u20138."},{"key":"S1351324909005105_manual_ref-31","doi-asserted-by":"crossref","unstructured":"Rieser, V. , Kruijff-Korbayov\u00e1, I. , and Lemon, O. 2005b. A framework for learning multimodal clarification strategies. In Proceedings of the 7th International Conference on Multimodal Interfaces (ICMI), Trento, Italy.","DOI":"10.18653\/v1\/2005.sigdial-1.11"},{"key":"S1351324909005105_manual_ref-36","doi-asserted-by":"crossref","unstructured":"Schatzmann, J. , Stuttle, M. N. , Weilhammer, K. , and Young, S. 2005b. Effects of the user model on simulation-based learning of dialogue strategies. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, San Juan, Puerto Rico, pp. 220\u20135.","DOI":"10.1109\/ASRU.2005.1566539"},{"key":"S1351324909005105_manual_ref-38","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888906000944"},{"key":"S1351324909005105_manual_ref-52","doi-asserted-by":"publisher","DOI":"10.1007\/11551874_58"},{"key":"S1351324909005105_manual_ref-41","first-page":"956","article-title":"Reinforcement learning for spoken dialogue systems","volume":"12","author":"Singh","year":"1999","journal-title":"Advances in Neural Information Processing Systems"},{"key":"S1351324909005105_manual_ref-9","unstructured":"Georgila, K. , Henderson, J. , and Lemon, O. 2006. User simulation for spoken dialogue systems: learning and evaluation. In Proceedings of the 9th International Conference on Spoken Language Processing (INTERSPEECH\u2013ICSLP), Pittsburgh, PA, pp. 1065\u201368."},{"key":"S1351324909005105_manual_ref-16","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2008.07-028-R2-05-82"},{"volume-title":"The HTK Book","year":"2005","author":"Young","key":"S1351324909005105_manual_ref-55"},{"key":"S1351324909005105_manual_ref-26","unstructured":"Poesio, M. , Cooper, R. , Larsson, S. , Matheson, C. , and Traum, D. 1999. Annotating conversations for information state update. In Proceedings of the 3rd Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL: AMSTELOGUE), Amsterdam, Netherlands."},{"key":"S1351324909005105_manual_ref-25","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324906004165"},{"key":"S1351324909005105_manual_ref-47","doi-asserted-by":"crossref","unstructured":"Walker, M. A. , Fromer, J. C. , and Narayanan, S. 1998. Learning optimal dialogue strategies: a case study of a spoken dialogue agent for email. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics (ACL\u2013COLING), Montreal, Quebec, Canada, pp. 1345\u201351.","DOI":"10.3115\/980691.980788"},{"key":"S1351324909005105_manual_ref-54","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2000.0593"},{"key":"S1351324909005105_manual_ref-5","doi-asserted-by":"crossref","unstructured":"Core, M. G. , Moore, J. D. , and Zinn, C. 2003. The role of initiative in tutorial dialogue. In Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Budapest, Hungary, pp. 67\u201374.","DOI":"10.3115\/1067807.1067818"},{"key":"S1351324909005105_manual_ref-12","doi-asserted-by":"crossref","unstructured":"Georgila, K. , Wolters, M. , and Moore, J. D. 2008b. Simulating the behaviour of older versus younger users when interacting with spoken dialogue systems. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL\u2013HLT), Columbus, OH, pp. 49\u201352.","DOI":"10.3115\/1557690.1557704"},{"key":"S1351324909005105_manual_ref-20","doi-asserted-by":"crossref","unstructured":"Lemon, O. , Georgila, K. , and Henderson, J. 2006a. Evaluating effectiveness and portability of reinforcement learned dialogue strategies with real users: the TALK TownInfo evaluation. In Proceedings of the IEEE\/ACL Workshop on Spoken Language Technology, Aruba, pp. 178\u201381.","DOI":"10.1109\/SLT.2006.326774"},{"key":"S1351324909005105_manual_ref-6","doi-asserted-by":"crossref","unstructured":"Frampton, M. , and Lemon, O. 2006. Learning more effective dialogue strategies using limited dialogue move features. In Proceedings of the 44th Annual Meeting of the Association for Computational Linguistics (ACL), Sydney, Australia, pp. 185\u201392.","DOI":"10.3115\/1220175.1220199"},{"key":"S1351324909005105_manual_ref-17","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324905004067"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324909005105","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,20]],"date-time":"2025-06-20T18:25:03Z","timestamp":1750443903000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324909005105\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,7]]},"references-count":56,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2009,10]]}},"alternative-id":["S1351324909005105"],"URL":"https:\/\/doi.org\/10.1017\/s1351324909005105","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"type":"print","value":"1351-3249"},{"type":"electronic","value":"1469-8110"}],"subject":[],"published":{"date-parts":[[2009,7]]}}}