{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:44:16Z","timestamp":1750308256096,"version":"3.41.0"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","license":[{"start":{"date-parts":[[2004,11,1]],"date-time":"2004-11-01T00:00:00Z","timestamp":1099267200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2004,11]]},"abstract":"<jats:p>This article describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems. No well-established measure has yet been proposed for evaluating discourse understanding, which has made it necessary to evaluate it only on the basis of the system's total performance. Such evaluations, however, are greatly influenced by task domains and dialogue strategies. To find a measure that enables good estimation of system performance only from discourse understanding results, we enumerated possible discourse-understanding-related metrics and calculated their correlation with the system's total performance through dialogue experiments.<\/jats:p>","DOI":"10.1145\/1035112.1035113","type":"journal-article","created":{"date-parts":[[2005,8,1]],"date-time":"2005-08-01T15:52:28Z","timestamp":1122911548000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Evaluating discourse understanding in spoken dialogue systems"],"prefix":"10.1145","volume":"1","author":[{"given":"Ryuichiro","family":"Higashinaka","sequence":"first","affiliation":[{"name":"NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan"}]},{"given":"Noboru","family":"Miyazaki","sequence":"additional","affiliation":[{"name":"NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan"}]},{"given":"Mikio","family":"Nakano","sequence":"additional","affiliation":[{"name":"NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan"}]},{"given":"Kiyoaki","family":"Aikawa","sequence":"additional","affiliation":[{"name":"NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan"}]}],"member":"320","published-online":{"date-parts":[[2004,11]]},"reference":[{"volume-title":"Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL). 191--199","author":"Abella A.","key":"e_1_2_1_1_1","unstructured":"Abella , A. and Gorin , A. L . 1999. Construct algebra: Analytical dialogue management . In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL). 191--199 . 10.3115\/1034678.1034714 Abella, A. and Gorin, A. L. 1999. Construct algebra: Analytical dialogue management. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL). 191--199. 10.3115\/1034678.1034714"},{"volume-title":"Proceedings of the International Conference on Intelligent User Interfaces (IUI). 1--8. 10","author":"Allen J.","key":"e_1_2_1_2_1","unstructured":"Allen , J. , Ferguson , G. , and Stent , A . 2001. An architecture for more realistic conversational systems . In Proceedings of the International Conference on Intelligent User Interfaces (IUI). 1--8. 10 .1145\/359784.359822 Allen, J., Ferguson, G., and Stent, A. 2001. An architecture for more realistic conversational systems. In Proceedings of the International Conference on Intelligent User Interfaces (IUI). 1--8. 10.1145\/359784.359822"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/0004-3702(80)90042-9","article-title":"Analyzing intention in utterances","volume":"15","author":"Allen J. F.","year":"1980","unstructured":"Allen , J. F. and Perrault , C. R. 1980 . Analyzing intention in utterances . Artif. Intel. 15 , 143 -- 178 . Allen, J. F. and Perrault, C. R. 1980. Analyzing intention in utterances. Artif. Intel. 15, 143--178.","journal-title":"Artif. Intel."},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/0004-3702(77)90018-2","article-title":"GUS, a frame driven dialog system","volume":"8","author":"Bobrow D. G.","year":"1977","unstructured":"Bobrow , D. G. , Kaplan , R. M. , Kay , M. , Norman , D. A. , Thompson , H. , and Winograd , T. 1977 . GUS, a frame driven dialog system . Artif. Intel. 8 , 155 -- 173 . Bobrow, D. G., Kaplan, R. M., Kay, M., Norman, D. A., Thompson, H., and Winograd, T. 1977. GUS, a frame driven dialog system. Artif. Intel. 8, 155--173.","journal-title":"Artif. Intel."},{"volume-title":"Plan Recognition in Natural Language Dialogue","author":"Carberry S.","key":"e_1_2_1_5_1","unstructured":"Carberry , S. 1990. Plan Recognition in Natural Language Dialogue . MIT Press , Cambridge, Mass . Carberry, S. 1990. Plan Recognition in Natural Language Dialogue. MIT Press, Cambridge, Mass."},{"key":"e_1_2_1_6_1","volume-title":"-J","author":"Chang C.-C.","year":"2001","unstructured":"Chang , C.-C. and Lin , C . -J . 2001 . LIBSVM: a library for support vector machines. Software available at http:\/\/www.csie.ntu.edu.tw\/&sim;cjlin\/libsvm. Chang, C.-C. and Lin, C.-J. 2001. LIBSVM: a library for support vector machines. Software available at http:\/\/www.csie.ntu.edu.tw\/&sim;cjlin\/libsvm."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 6th Applied Natural Language Processing Conference (NLP). 97--104","author":"Chu-Carroll J.","year":"2000","unstructured":"Chu-Carroll , J. 2000 . MIMIC: An adaptive mixed initiative spoken dialogue system for information queries . In Proceedings of the 6th Applied Natural Language Processing Conference (NLP). 97--104 . 10.3115\/974147.974161 Chu-Carroll, J. 2000. MIMIC: An adaptive mixed initiative spoken dialogue system for information queries. In Proceedings of the 6th Applied Natural Language Processing Conference (NLP). 97--104. 10.3115\/974147.974161"},{"key":"e_1_2_1_8_1","first-page":"361","article-title":"Vector-based natural language call routing","volume":"25","author":"Chu-Carroll J.","year":"1999","unstructured":"Chu-Carroll , J. and Carpenter , B. 1999 . Vector-based natural language call routing . Comp. Ling. 25 , 3, 361 -- 388 . Chu-Carroll, J. and Carpenter, B. 1999. Vector-based natural language call routing. Comp. Ling. 25, 3, 361--388.","journal-title":"Comp. Ling."},{"volume-title":"Proceedings of the 8th European Conference on Speech Communication and Technology (Eurospeech). 657--660","author":"Dohsaka K.","key":"e_1_2_1_9_1","unstructured":"Dohsaka , K. , Yasuda , N. , and Aikawa , K . 2003. Efficient spoken dialogue control depending on the speech recognition rate and system's database . In Proceedings of the 8th European Conference on Speech Communication and Technology (Eurospeech). 657--660 . Dohsaka, K., Yasuda, N., and Aikawa, K. 2003. Efficient spoken dialogue control depending on the speech recognition rate and system's database. In Proceedings of the 8th European Conference on Speech Communication and Technology (Eurospeech). 657--660."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL). 48--57","author":"Doran C.","year":"1807","unstructured":"Doran , C. , Aberdeen , J. , Damianos , L. , and Hirschman , L . 2001. Comparing several aspects of human-computer and human-human dialogues . In Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL). 48--57 . 10.3115\/11 1807 8.1118085 Doran, C., Aberdeen, J., Damianos, L., and Hirschman, L. 2001. Comparing several aspects of human-computer and human-human dialogues. In Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL). 48--57. 10.3115\/1118078.1118085"},{"volume-title":"Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP). 1--4.","author":"Glass J.","key":"e_1_2_1_11_1","unstructured":"Glass , J. , Polifroni , J. , Seneff , S. , and Zue , V . 2000. Data collection and performance evaluation of spoken dialogue systems: The MIT experience . In Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP). 1--4. Glass, J., Polifroni, J., Seneff, S., and Zue, V. 2000. Data collection and performance evaluation of spoken dialogue systems: The MIT experience. In Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP). 1--4."},{"volume-title":"Proceedings of the International Conference on Spoken Language Processing (ICSLP). 701--704","author":"Goddeau D.","key":"e_1_2_1_12_1","unstructured":"Goddeau , D. , Meng , H. , Polifroni , J. , Seneff , S. , and Busayapongchai , S . 1996. A form-based dialogue manager for spoken language applications . In Proceedings of the International Conference on Spoken Language Processing (ICSLP). 701--704 . Goddeau, D., Meng, H., Polifroni, J., Seneff, S., and Busayapongchai, S. 1996. A form-based dialogue manager for spoken language applications. In Proceedings of the International Conference on Spoken Language Processing (ICSLP). 701--704."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-6393(97)00040-X"},{"volume-title":"Proceedings of the International Conference on Spoken Language Processing (ICSLP). 829--832","author":"Higashinaka R.","key":"e_1_2_1_14_1","unstructured":"Higashinaka , R. , Miyazaki , N. , Nakano , M. , and Aikawa , K . 2002. A method for evaluating incremental utterance understanding in spoken dialogue systems . In Proceedings of the International Conference on Spoken Language Processing (ICSLP). 829--832 . Higashinaka, R., Miyazaki, N., Nakano, M., and Aikawa, K. 2002. A method for evaluating incremental utterance understanding in spoken dialogue systems. In Proceedings of the International Conference on Spoken Language Processing (ICSLP). 829--832."},{"volume-title":"Proceedings of the 8th European Conference on Speech Communication and Technology (EUROSPEECH). 1941--1944","author":"Higashinaka R.","key":"e_1_2_1_15_1","unstructured":"Higashinaka , R. , Miyazaki , N. , Nakano , M. , and Aikawa , K . 2003. Evaluating discourse understanding in spoken dialogue systems . In Proceedings of the 8th European Conference on Speech Communication and Technology (EUROSPEECH). 1941--1944 . Higashinaka, R., Miyazaki, N., Nakano, M., and Aikawa, K. 2003. Evaluating discourse understanding in spoken dialogue systems. In Proceedings of the 8th European Conference on Speech Communication and Technology (EUROSPEECH). 1941--1944."},{"volume-title":"Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL). 240--247","author":"Higashinaka R.","key":"e_1_2_1_16_1","unstructured":"Higashinaka , R. , Nakano , M. , and Aikawa , K . 2003. Corpus-based discourse understanding in spoken dialogue systems . In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL). 240--247 . 10.3115\/1075096.1075127 Higashinaka, R., Nakano, M., and Aikawa, K. 2003. Corpus-based discourse understanding in spoken dialogue systems. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL). 240--247. 10.3115\/1075096.1075127"},{"volume-title":"Proceedings of the 19th International Conference on Computational Linguistics (COLING). 342--348","author":"Hirao T.","key":"e_1_2_1_17_1","unstructured":"Hirao , T. , Isozaki , H. , Maeda , E. , and Matsumoto , Y . 2002. Extracting important sentences with support vector machines . In Proceedings of the 19th International Conference on Computational Linguistics (COLING). 342--348 . 10.3115\/1072228.1072281 Hirao, T., Isozaki, H., Maeda, E., and Matsumoto, Y. 2002. Extracting important sentences with support vector machines. In Proceedings of the 19th International Conference on Computational Linguistics (COLING). 342--348. 10.3115\/1072228.1072281"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 18th International Conference on Computational Linguistics (COLING).","volume":"1","author":"Komatani K.","unstructured":"Komatani , K. and Kawahara , T . 2000. Flexible mixed-initiative dialogue management using concept-level confidence measures of speech recognizer output . In Proceedings of the 18th International Conference on Computational Linguistics (COLING). Vol. 1 . 467--473. 10.3115\/990820.990888 Komatani, K. and Kawahara, T. 2000. Flexible mixed-initiative dialogue management using concept-level confidence measures of speech recognizer output. In Proceedings of the 18th International Conference on Computational Linguistics (COLING). Vol. 1. 467--473. 10.3115\/990820.990888"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-6393(99)00067-9"},{"volume-title":"Proceedings of the European Conference on Speech Communication and Technology (EUROSPEECH). 1691--1694","author":"Lee A.","key":"e_1_2_1_20_1","unstructured":"Lee , A. , Kawahara , T. , and Shikano , K . 2001. Julius---an open source real-time large vocabulary recognition engine . In Proceedings of the European Conference on Speech Communication and Technology (EUROSPEECH). 1691--1694 . Lee, A., Kawahara, T., and Shikano, K. 2001. Julius---an open source real-time large vocabulary recognition engine. In Proceedings of the European Conference on Speech Communication and Technology (EUROSPEECH). 1691--1694."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505285"},{"volume-title":"Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL). 150--159","author":"Nakano M.","key":"e_1_2_1_22_1","unstructured":"Nakano , M. , Miyazaki , N. , Yasuda , N. , Sugiyama , A. , Hirasawa , J. , Dohsaka , K. , and Aikawa , K . 2000. WIT: A toolkit for building robust and real-time spoken dialogue systems . In Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL). 150--159 . 10.3115\/1117736.1117753 Nakano, M., Miyazaki, N., Yasuda, N., Sugiyama, A., Hirasawa, J., Dohsaka, K., and Aikawa, K. 2000. WIT: A toolkit for building robust and real-time spoken dialogue systems. In Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL). 150--159. 10.3115\/1117736.1117753"},{"key":"e_1_2_1_23_1","first-page":"15","article-title":"COLLAGEN: Applying collaborative discourse theory","volume":"22","author":"Rich C.","year":"2001","unstructured":"Rich , C. , Sidner , C. , and Lesh , N. 2001 . COLLAGEN: Applying collaborative discourse theory . AI Magazine 22 , 4, 15 -- 25 . Rich, C., Sidner, C., and Lesh, N. 2001. COLLAGEN: Applying collaborative discourse theory. AI Magazine 22, 4, 15--25.","journal-title":"AI Magazine"},{"volume-title":"Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP). 130--134","author":"Rudnicky A. I.","key":"e_1_2_1_24_1","unstructured":"Rudnicky , A. I. , Bennett , C. , Black , A. , Chotomongcol , A. , Lenzo , K. , Oh , A. , and Singh , R . 2000. Task and Domain Specific Modelling in the Carnegie Mellon Communicator System . In Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP). 130--134 . Rudnicky, A. I., Bennett, C., Black, A., Chotomongcol, A., Lenzo, K., Oh, A., and Singh, R. 2000. Task and Domain Specific Modelling in the Carnegie Mellon Communicator System. In Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP). 130--134."},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/S0885-2308(02)00011-6","article-title":"Response planning and generation in the mercury flight reservation system","volume":"16","author":"Seneff S.","year":"2002","unstructured":"Seneff , S. 2002 . Response planning and generation in the mercury flight reservation system . Comput. Speech Lang. 16 , 3 -- 4 , 283--312. Seneff, S. 2002. Response planning and generation in the mercury flight reservation system. Comput. Speech Lang. 16, 3--4, 283--312.","journal-title":"Comput. Speech Lang."},{"key":"e_1_2_1_26_1","unstructured":"Smola A. J. and Sch\u00f6lkopf B. 1998. A tutorial on support vector regression. NeuroCOLT2 Technical Report (NC2-TR-1998-030).  Smola A. J. and Sch\u00f6lkopf B. 1998. A tutorial on support vector regression. NeuroCOLT2 Technical Report (NC2-TR-1998-030)."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the ACM Special Interest Group on Computer-Human Interaction (SIGCHI). 211--217","author":"Sparks R.","year":"1916","unstructured":"Sparks , R. , Meiskey , L. , and Brunner , H . 1994. An object-oriented approach to dialogue management in spoken language systems . In Proceedings of the ACM Special Interest Group on Computer-Human Interaction (SIGCHI). 211--217 . 10.1145\/ 1916 66.191749 Sparks, R., Meiskey, L., and Brunner, H. 1994. An object-oriented approach to dialogue management in spoken language systems. In Proceedings of the ACM Special Interest Group on Computer-Human Interaction (SIGCHI). 211--217. 10.1145\/191666.191749"},{"volume-title":"Proceedings of the European Conference on Speech Communication and Technology (EUROSPEECH). 1419--1422","author":"Sturm J.","key":"e_1_2_1_28_1","unstructured":"Sturm , J. , den Os , E. , and Boves , L . 1999. Dialogue management in the Dutch ARISE train timetable information system . In Proceedings of the European Conference on Speech Communication and Technology (EUROSPEECH). 1419--1422 . Sturm, J., den Os, E., and Boves, L. 1999. Dialogue management in the Dutch ARISE train timetable information system. In Proceedings of the European Conference on Speech Communication and Technology (EUROSPEECH). 1419--1422."},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/89.890065","article-title":"A Japanese TTS system based on multi-form units and a speech modification algorithm with harmonics reconstruction","volume":"9","author":"Takano S.","year":"2001","unstructured":"Takano , S. , Tanaka , K. , Mizuno , H. , Abe , M. , and Nakajima , S. 2001 . A Japanese TTS system based on multi-form units and a speech modification algorithm with harmonics reconstruction . IEEE Trans. Speech Audio Process. 9 , 1, 3 -- 10 . Takano, S., Tanaka, K., Mizuno, H., Abe, M., and Nakajima, S. 2001. A Japanese TTS system based on multi-form units and a speech modification algorithm with harmonics reconstruction. IEEE Trans. Speech Audio Process. 9, 1, 3--10.","journal-title":"IEEE Trans. Speech Audio Process."},{"volume-title":"The Nature of Statistical Learnig Theory","author":"Vapnik V.","key":"e_1_2_1_30_1","unstructured":"Vapnik , V. 1995. The Nature of Statistical Learnig Theory . Springer . Vapnik, V. 1995. The Nature of Statistical Learnig Theory. Springer."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324900002503"},{"volume-title":"Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics. 271--280","author":"Walker M. A.","key":"e_1_2_1_32_1","unstructured":"Walker , M. A. , Litman , D. J. , Kamm , C. A. , and Abella , A . 1997. PARADISE: A Framework for Evaluating Spoken Dialogue Agents . In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics. 271--280 . 10.3115\/979617.979652 Walker, M. A., Litman, D. J., Kamm, C. A., and Abella, A. 1997. PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics. 271--280. 10.3115\/979617.979652"},{"volume-title":"Proceedings of the European Conference on Machine Learning Poster Papers. 128--137","author":"Wang Y.","key":"e_1_2_1_33_1","unstructured":"Wang , Y. and Witten , I. H . 1997. Induction of model trees for predicting continuous classes . In Proceedings of the European Conference on Machine Learning Poster Papers. 128--137 . Wang, Y. and Witten, I. H. 1997. Induction of model trees for predicting continuous classes. In Proceedings of the European Conference on Machine Learning Poster Papers. 128--137."},{"key":"e_1_2_1_34_1","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann.","author":"Witten I. H.","year":"1999","unstructured":"Witten , I. H. and Frank , E . 1999 . Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann. Witten, I. H. and Frank, E. 1999. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann."},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","first-page":"1166","DOI":"10.1109\/5.880078","article-title":"Conversational interfaces: Advances and challenges","volume":"88","author":"Zue V. W.","year":"2000","unstructured":"Zue , V. W. and Glass , J. R. 2000 . Conversational interfaces: Advances and challenges . In Proceedings of the IEEE 88 , 8, 1166 -- 1180 . Zue, V. W. and Glass, J. R. 2000. Conversational interfaces: Advances and challenges. In Proceedings of the IEEE 88, 8, 1166--1180.","journal-title":"Proceedings of the IEEE"}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1035112.1035113","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1035112.1035113","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:23:40Z","timestamp":1750267420000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1035112.1035113"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2004,11]]},"references-count":35,"alternative-id":["10.1145\/1035112.1035113"],"URL":"https:\/\/doi.org\/10.1145\/1035112.1035113","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"type":"print","value":"1550-4875"},{"type":"electronic","value":"1550-4883"}],"subject":[],"published":{"date-parts":[[2004,11]]},"assertion":[{"value":"2004-11-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}