{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T01:02:00Z","timestamp":1774400520809,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":31,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T00:00:00Z","timestamp":1697846400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,10,21]]},"DOI":"10.1145\/3583780.3615286","type":"proceedings-article","created":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T07:45:26Z","timestamp":1697874326000},"page":"4185-4189","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["On the Reliability of User Feedback for Evaluating the Quality of Conversational Agents"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-5526-1834","authenticated-orcid":false,"given":"Jordan","family":"Massiah","sequence":"first","affiliation":[{"name":"Amazon &amp; University College London, London, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4734-4532","authenticated-orcid":false,"given":"Emine","family":"Yilmaz","sequence":"additional","affiliation":[{"name":"Amazon &amp; University College London, London, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0776-0550","authenticated-orcid":false,"given":"Yunlong","family":"Jiao","sequence":"additional","affiliation":[{"name":"Amazon, London, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5158-6630","authenticated-orcid":false,"given":"Gabriella","family":"Kazai","sequence":"additional","affiliation":[{"name":"Amazon, London, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,10,21]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512058"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_3_2_1_3_1","volume-title":"Kate Ridgeway, Joshua Levy, Swanand Joshi, and Alborz Geramifard.","author":"Bodigutla Praveen Kumar","year":"2019","unstructured":"Praveen Kumar Bodigutla , Spyros Matsoukas , Longshaokan Marshall Wang , Kate Ridgeway, Joshua Levy, Swanand Joshi, and Alborz Geramifard. 2019 . Domain- independent turn-level dialogue quality evaluation via user satisfaction estimation. Praveen Kumar Bodigutla, Spyros Matsoukas, Longshaokan Marshall Wang, Kate Ridgeway, Joshua Levy, Swanand Joshi, and Alborz Geramifard. 2019. Domain- independent turn-level dialogue quality evaluation via user satisfaction estimation."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.347"},{"key":"e_1_3_2_1_5_1","first-page":"33","volume-title":"Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00","author":"Buckley Chris","unstructured":"Chris Buckley and Ellen M. Voorhees . 2000. Evaluating evaluation measure stability . In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00 , page 33 -- 40 , New York, NY, USA. Association for Computing Machinery. Chris Buckley and Ellen M. Voorhees. 2000. Evaluating evaluation measure stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00, page 33--40, New York, NY, USA. Association for Computing Machinery."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340631.3394856"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080804"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358047"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-020-09866-x"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.326"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1059981.1059982"},{"key":"e_1_3_2_1_12_1","volume-title":"Robertaiq: An efficient framework for automatic interaction quality estimation of dialogue systems.","author":"Gupta Saurabh","year":"2021","unstructured":"Saurabh Gupta , Xing Fan , Derek Liu , Benjamin Yao , Yuan Ling , Kun Zhou , Tuan-Hung Pham , and Edward Guo . 2021 . Robertaiq: An efficient framework for automatic interaction quality estimation of dialogue systems. Saurabh Gupta, Xing Fan, Derek Liu, Benjamin Yao, Yuan Ling, Kun Zhou, Tuan-Hung Pham, and Edward Guo. 2021. Robertaiq: An efficient framework for automatic interaction quality estimation of dialogue systems."},{"issue":"1","key":"e_1_3_2_1_13_1","first-page":"4","article-title":"A brief guide to decisions at each step of the propensity score matching process","volume":"21","author":"Harris Heather","year":"2016","unstructured":"Heather Harris and S Jeanne Horst . 2016 . A brief guide to decisions at each step of the propensity score matching process . Practical Assessment, Research, and Evaluation , 21 ( 1 ): 4 . Heather Harris and S Jeanne Horst. 2016. A brief guide to decisions at each step of the propensity score matching process. Practical Assessment, Research, and Evaluation, 21(1):4.","journal-title":"Practical Assessment, Research, and Evaluation"},{"key":"e_1_3_2_1_14_1","first-page":"1183","volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM '18","author":"Hashemi Seyyed Hadi","unstructured":"Seyyed Hadi Hashemi , Kyle Williams , Ahmed El Kholy , Imed Zitouni , and Paul A. Crook . 2018. Measuring user satisfaction on smart speaker intelligent assistants using intent sensitive query embeddings . In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM '18 , page 1183 -- 1192 , New York, NY, USA. Association for Computing Machinery. Seyyed Hadi Hashemi, Kyle Williams, Ahmed El Kholy, Imed Zitouni, and Paul A. Crook. 2018. Measuring user satisfaction on smart speaker intelligent assistants using intent sensitive query embeddings. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM '18, page 1183--1192, New York, NY, USA. Association for Computing Machinery."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Thorsten Hennig-Thurau and Alexander Klee. 1997. The impact of customer satisfaction and relationship quality on customer retention: A critical reassessment and model development. Psychology & marketing 14(8):737--764.  Thorsten Hennig-Thurau and Alexander Klee. 1997. The impact of customer satisfaction and relationship quality on customer retention: A critical reassessment and model development. Psychology & marketing 14(8):737--764.","DOI":"10.1002\/(SICI)1520-6793(199712)14:8<737::AID-MAR2>3.0.CO;2-F"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741669"},{"key":"e_1_3_2_1_17_1","first-page":"1257","volume-title":"The 41st international acm sigir conference on research & development in information retrieval","author":"Kiesel Johannes","unstructured":"Johannes Kiesel , Arefeh Bahrami , Benno Stein , Avishek Anand , and Matthias Hagen . 2018. Toward voice query clarification . In The 41st international acm sigir conference on research & development in information retrieval , pages 1257 -- 1260 . Johannes Kiesel, Arefeh Bahrami, Benno Stein, Avishek Anand, and Matthias Hagen. 2018. Toward voice query clarification. In The 41st international acm sigir conference on research & development in information retrieval, pages 1257--1260."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914754"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911521"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC48229.2022.9871333"},{"key":"e_1_3_2_1_21_1","first-page":"1533","volume-title":"Proceedings of the 2018 world wide web conference","author":"Liu Mengyang","year":"2018","unstructured":"Mengyang Liu , Yiqun Liu , Jiaxin Mao , Cheng Luo , Min Zhang , and Shaoping Ma . 2018 . \" satisfaction with failure\" or\" unsatisfied success\" investigating the relationship between search success and user satisfaction . In Proceedings of the 2018 world wide web conference , pages 1533 -- 1542 . Mengyang Liu, Yiqun Liu, Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. \" satisfaction with failure\" or\" unsatisfied success\" investigating the relationship between search success and user satisfaction. In Proceedings of the 2018 world wide web conference, pages 1533--1542."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767721"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911507"},{"key":"e_1_3_2_1_24_1","unstructured":"Dookun Park Hao Yuan Dongmin Kim Yinglei Zhang Spyros Matsoukas Young-Bum Kim Ruhi Sarikaya Edward Guo Yuan Ling Kevin Quinn Tuan-Hung Pham Benjamin Yao and Sungjin Lee. 2020. Large-scale hybrid approach for predicting user satisfaction with conversational agents.  Dookun Park Hao Yuan Dongmin Kim Yinglei Zhang Spyros Matsoukas Young-Bum Kim Ruhi Sarikaya Edward Guo Yuan Ling Kevin Quinn Tuan-Hung Pham Benjamin Yao and Sungjin Lee. 2020. Large-scale hybrid approach for predicting user satisfaction with conversational agents."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753687"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753687"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9273-z"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970398.2970443"},{"key":"e_1_3_2_1_29_1","first-page":"3369","volume-title":"Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)","author":"Schmitt Alexander","year":"2012","unstructured":"Alexander Schmitt , Stefan Ultes , and Wolfgang Minker . 2012 . A parameterized and annotated spoken dialog corpus of the Carnegie Mellon University let's go bus information system . In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) , pages 3369 -- 3373 , Istanbul, Turkey. European Language Resources Association (ELRA). Alexander Schmitt, Stefan Ultes, and Wolfgang Minker. 2012. A parameterized and annotated spoken dialog corpus of the Carnegie Mellon University let's go bus information system. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3369--3373, Istanbul, Turkey. European Language Resources Association (ELRA)."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22948"},{"key":"e_1_3_2_1_31_1","first-page":"316","volume-title":"Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '02","author":"Ellen","unstructured":"Ellen M. Voorhees and Chris Buckley. 2002. The effect of topic set size on retrieval experiment error . In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '02 , page 316 -- 323 , New York, NY, USA. Association for Computing Machinery. Ellen M. Voorhees and Chris Buckley. 2002. The effect of topic set size on retrieval experiment error. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '02, page 316--323, New York, NY, USA. Association for Computing Machinery."}],"event":{"name":"CIKM '23: The 32nd ACM International Conference on Information and Knowledge Management","location":"Birmingham United Kingdom","acronym":"CIKM '23","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 32nd ACM International Conference on Information and Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583780.3615286","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3583780.3615286","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:59Z","timestamp":1750178219000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583780.3615286"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,21]]},"references-count":31,"alternative-id":["10.1145\/3583780.3615286","10.1145\/3583780"],"URL":"https:\/\/doi.org\/10.1145\/3583780.3615286","relation":{},"subject":[],"published":{"date-parts":[[2023,10,21]]},"assertion":[{"value":"2023-10-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}