{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T21:31:57Z","timestamp":1777152717125,"version":"3.51.4"},"reference-count":138,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T00:00:00Z","timestamp":1709769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Hong Kong RGC GRF","award":["RGC\/HKBU12201620"],"award-info":[{"award-number":["RGC\/HKBU12201620"]}]},{"name":"Hong Kong Baptist University IG-FNRA","award":["RC-FNRA-IG\/21-22\/SCI\/01"],"award-info":[{"award-number":["RC-FNRA-IG\/21-22\/SCI\/01"]}]},{"name":"Hong Kong Baptist University Start-up","award":["RC-STARTUP\/21-22\/23"],"award-info":[{"award-number":["RC-STARTUP\/21-22\/23"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>\n            An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user\u2019s perspective remains elusive. The GUI-based system evaluation criteria may be inadequate for their conversational counterparts. This article presents our proposed unifying framework,\n            <jats:bold>\n              <jats:italic>CRS-Que<\/jats:italic>\n            <\/jats:bold>\n            , to evaluate the user experience of CRSs. This new evaluation framework is developed based on\n            <jats:italic>ResQue<\/jats:italic>\n            , a popular user-centric evaluation framework for recommender systems. Additionally, it includes user experience metrics of conversation (e.g., understanding, response quality, humanness) under two dimensions of\n            <jats:italic>ResQue<\/jats:italic>\n            (i.e., Perceived Qualities and User Beliefs). Following the psychometric modeling method, we validate our framework by evaluating two conversational recommender systems in different scenarios:\n            <jats:italic>music exploration<\/jats:italic>\n            and\n            <jats:italic>mobile phone purchase<\/jats:italic>\n            . The results of the two studies support the validity and reliability of the constructs in our framework and reveal how conversation constructs and recommendation constructs interact and influence the overall user experience of the CRS. We believe this framework could help researchers conduct standardized user-centric research for conversational recommender systems and provide practitioners with insights into designing and evaluating a CRS from users\u2019 perspectives.\n          <\/jats:p>","DOI":"10.1145\/3631534","type":"journal-article","created":{"date-parts":[[2023,11,2]],"date-time":"2023-11-02T22:20:16Z","timestamp":1698963616000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["<i>CRS-Que<\/i>\n            : A User-centric Evaluation Framework for Conversational Recommender Systems"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3926-7277","authenticated-orcid":false,"given":"Yucheng","family":"Jin","sequence":"first","affiliation":[{"name":"Hong Kong Baptist University, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5842-838X","authenticated-orcid":false,"given":"Li","family":"Chen","sequence":"additional","affiliation":[{"name":"Hong Kong Baptist University, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8506-3825","authenticated-orcid":false,"given":"Wanling","family":"Cai","sequence":"additional","affiliation":[{"name":"Hong Kong Baptist University, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8327-4466","authenticated-orcid":false,"given":"Xianglin","family":"Zhao","sequence":"additional","affiliation":[{"name":"Hong Kong Baptist University, Hong Kong, China"}]}],"member":"320","published-online":{"date-parts":[[2024,3,7]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2021.3078384"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2838739.2838753"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.107.2.238"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025209"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-021-01582-9"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/w17-5522"},{"key":"e_1_3_3_8_2","volume-title":"Confirmatory Factor Analysis for Applied Research: 3. Introduction to CFA","author":"Brown Timothy A.","year":"2015","unstructured":"Timothy A. Brown. 2015. Confirmatory Factor Analysis for Applied Research: 3. Introduction to CFA. Guilford Publications."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340631.3394856"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","unstructured":"Wanling Cai Yucheng Jin and Li Chen. 2021. Critiquing for music exploration in conversational recommender systems. In Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201921). Association for Computing Machinery New York NY 480\u2013490. DOI:10.1145\/3397481.3450657","DOI":"10.1145\/3397481.3450657"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517471"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/thms.2021.3131674"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025919"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2020.1841438"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/1597538.1597564"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/1454008.1454022"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-008-9057-x"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9108-6"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","unstructured":"Li Chen and Feng Wang. 2017. Explaining recommendations based on feature sentiments in product reviews. In Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201917). Association for Computing Machinery New York NY 17\u201328. DOI:10.1145\/3025171.3025173","DOI":"10.1145\/3025171.3025173"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","unstructured":"Li Chen Yonghua Yang Ningxia Wang Keping Yang and Quan Yuan. 2019. How serendipity improves user satisfaction with recommendations? A large-scale user evaluation. In Proceedings of the World Wide Web Conference (WWW\u201919). Association for Computing Machinery New York NY 240\u2013250. DOI:10.1145\/3308558.3313469","DOI":"10.1145\/3308558.3313469"},{"key":"e_1_3_3_21_2","volume-title":"Academy of Management Proceedings","volume":"2017","author":"Cheung Gordon W.","year":"2017","unstructured":"Gordon W. Cheung and Chang Wang. 2017. Current approaches for assessing convergent and discriminant validity with SEM: Issues and solutions. In Academy of Management Proceedings, Vol. 2017. Academy of Management Briarcliff Manor, NY."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939746"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-008-9051-3"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-23765-2_11"},{"key":"e_1_3_3_25_2","volume-title":"Proceedings of 1st International Conference on Language Resources and Evaluation","author":"Dale Robert","year":"1998","unstructured":"Robert Dale and Chris Mellish. 1998. Towards evaluation in natural language generation. In Proceedings of 1st International Conference on Language Resources and Evaluation."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.2307\/249008"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1037\/xap0000092"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3342775.3342809"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/tlt.2017.2732349"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-5915.2007.00174.x"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.06.002"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","unstructured":"Mouzhi Ge Carla Delgado-Battenfeld and Dietmar Jannach. 2010. Beyond accuracy: Evaluating recommender systems by coverage and serendipity. In Proceedings of the Conference on Recommender Systems (RecSys\u201910). Association for Computing Machinery New York NY 257\u2013260. DOI:10.1145\/1864708.1864761","DOI":"10.1145\/1864708.1864761"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2013.12.007"},{"key":"e_1_3_3_34_2","unstructured":"Asma Ghandeharioun Judy Hanwen Shen Natasha Jaques Craig Ferguson Noah Jones Agata Lapedriza and Rosalind Picard. 2019. Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems. Curran Associates Inc. Red Hook NY USA."},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2019.01.020"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/w18-5704"},{"key":"e_1_3_3_37_2","volume-title":"Multivariate Data Analysis","author":"Jr Joseph F. Hair","year":"2009","unstructured":"Joseph F. Hair Jr, William C. Black, Barry J. Babin, and Rolph E. Anderson. 2009. Multivariate Data Analysis (7th ed.). Prentice Hall, Upper Saddle River, NJ."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2014.12.005"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/1349822.1349838"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.17185\/duepublico\/75905"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3469595.3469596"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.2197\/ipsjjip.22.669"},{"key":"e_1_3_3_45_2","volume-title":"SPSS Explained","author":"Hinton Perry Roy","year":"2004","unstructured":"Perry Roy Hinton, Charlotte Brownlow, and Isabella McMurray. 2004. SPSS Explained. Psychology Press."},{"issue":"1","key":"e_1_3_3_46_2","first-page":"53","article-title":"Structural equation modelling: Guidelines for determining model fit","volume":"6","author":"Hooper Daire","year":"2008","unstructured":"Daire Hooper, Joseph Coughlan, and Michael R. Mullen. 2008. Structural equation modelling: Guidelines for determining model fit. Electron. J. Busin. Res. Meth. 6, 1 (2008), 53\u201360.","journal-title":"Electron. J. Busin. Res. Meth."},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.4018\/joeuc.2000100102"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-022-10229-x"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453154"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-53676-7_2"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401234"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357923"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472307.3484164"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","unstructured":"Yucheng Jin Nyi Nyi Htun Nava Tintarev and Katrien Verbert. 2019. ContextPlay: Evaluating user control for context-aware music recommendation. In Proceedings of the User Modeling Adaptation and Personalization Conference (UMAP\u201919). Association for Computing Machinery New York NY 294\u2013302. DOI:10.1145\/3320435.3320445","DOI":"10.1145\/3320435.3320445"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2909132.2909269"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-019-09247-2"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240358"},{"key":"e_1_3_3_58_2","volume-title":"Proceedings of the Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents-Development and Evaluation Workshop Programme (RE-WOCHAT\u201916)","author":"Kaleem Mohammed","year":"2016","unstructured":"Mohammed Kaleem, Omar Alobadi, James O\u2019Shea, and Keeley Crockett. 2016. Framework for the formulation of metrics for conversational agent evaluation. In Proceedings of the Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents-Development and Evaluation Workshop Programme (RE-WOCHAT\u201916)."},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/2926720"},{"key":"e_1_3_3_60_2","article-title":"Recommendation as a communication game: Self-supervised bot-play for goal-oriented dialogue","author":"Kang Dongyeop","year":"2019","unstructured":"Dongyeop Kang, Anusha Balakrishnan, Pararth Shah, Paul Crook, Y-Lan Boureau, and Jason Weston. 2019. Recommendation as a communication game: Self-supervised bot-play for goal-oriented dialogue. arXiv preprint arXiv:1909.03922 (2019).","journal-title":"arXiv preprint arXiv:1909.03922"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109873"},{"issue":"16","key":"e_1_3_3_62_2","first-page":"349","article-title":"User adaptive chatbot for mitigating depression","volume":"118","author":"Kataria Pratik","year":"2018","unstructured":"Pratik Kataria, Kiran Rode, Akshay Jain, Prachi Dwivedi, and Sukhada Bhingarkar. 2018. User adaptive chatbot for mitigating depression. Int. J. Pure Appl. Math. 118, 16 (2018), 349\u2013361.","journal-title":"Int. J. Pure Appl. Math."},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8535.1993.tb00076.x"},{"key":"e_1_3_3_64_2","first-page":"2390","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Kizilcec Ren\u00e9 F.","year":"2016","unstructured":"Ren\u00e9 F. Kizilcec. 2016. How much information? Effects of transparency on trust in an algorithmic interface. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 2390\u20132395."},{"issue":"4","key":"e_1_3_3_65_2","article-title":"Principles & practice of structural equation modelling","volume":"40","author":"Kline Rex B.","year":"1999","unstructured":"Rex B. Kline and Darcy A. Santor. 1999. Principles & practice of structural equation modelling. Canad. Psychol. 40, 4 (1999), 381.","journal-title":"Canad. Psychol."},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365966"},{"key":"e_1_3_3_67_2","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1145\/2043932.2043960","volume-title":"Proceedings of the 5th ACM Conference on Recommender Systems","author":"Knijnenburg Bart P.","year":"2011","unstructured":"Bart P. Knijnenburg, Niels J. M. Reijmer, and Martijn C. Willemsen. 2011. Each to his own: How different users call for different interaction methods in recommender systems. In Proceedings of the 5th ACM Conference on Recommender Systems. 141\u2013148."},{"key":"e_1_3_3_68_2","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1145\/1639714.1639793","volume-title":"Proceedings of the 3rd ACM Conference on Recommender Systems","author":"Knijnenburg Bart P.","year":"2009","unstructured":"Bart P. Knijnenburg and Martijn C. Willemsen. 2009. Understanding the effect of adaptive preference elicitation methods on user satisfaction of a recommender system. In Proceedings of the 3rd ACM Conference on Recommender Systems. 381\u2013384."},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9118-4"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9112-x"},{"key":"e_1_3_3_71_2","article-title":"Commercial chatbot: Performance evaluation, usability metrics and quality standards of embodied conversational agents","volume":"2","author":"Kuligowska Karolina","year":"2015","unstructured":"Karolina Kuligowska. 2015. Commercial chatbot: Performance evaluation, usability metrics and quality standards of embodied conversational agents. Profess. Cent. Busin. Res. 2 (2015).","journal-title":"Profess. Cent. Busin. Res."},{"key":"e_1_3_3_72_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Kunkel Johannes","year":"2019","unstructured":"Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and J\u00fcrgen Ziegler. 2019. Let me explain: Impact of personal and impersonal explanations on trust in recommender systems. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1\u201312."},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2017.02.005"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2022.2108060"},{"key":"e_1_3_3_75_2","article-title":"Deep conversational recommender in travel","volume":"1907","author":"Liao Lizi","year":"2019","unstructured":"Lizi Liao, Ryuichi Takanobu, Yunshan Ma, Xun Yang, Minlie Huang, and Tat-Seng Chua. 2019. Deep conversational recommender in travel. ArXiv abs\/1907.00710 (2019).","journal-title":"ArXiv"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/1297231.1297235"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.7892\/boris.105398"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.2753\/JEC1086-4415110202"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/1125451.1125659"},{"key":"e_1_3_3_80_2","article-title":"Perceived usefulness: A silver bullet to assure user data availability for online recommendation systems","volume":"139","author":"Mican Daniel","year":"2020","unstructured":"Daniel Mican, Dan-Andrei Sitar-T\u0103ut, and Ovidiu-Ioan Moisescu. 2020. Perceived usefulness: A silver bullet to assure user data availability for online recommendation systems. Decis. Supp. Syst. 139 (2020), 113420.","journal-title":"Decis. Supp. Syst."},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-019-09250-7"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-03840-3_39"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10796-017-9782-y"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.elerap.2016.09.003"},{"key":"e_1_3_3_85_2","first-page":"472","volume-title":"Proceedings of the International Conference on Human-Computer Interaction","author":"Novick David","year":"2014","unstructured":"David Novick and Iv\u00e1n Gris. 2014. Building rapport between human and ECA: A pilot study. In Proceedings of the International Conference on Human-Computer Interaction. Springer, 472\u2013480."},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-017-9195-0"},{"key":"e_1_3_3_87_2","volume-title":"Psychometric Theory 3E","author":"Nunnally Jum C.","year":"1994","unstructured":"Jum C. Nunnally. 1994. Psychometric Theory 3E. Tata McGraw-Hill Education."},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/1040830.1040870"},{"key":"e_1_3_3_89_2","first-page":"311","volume-title":"Proceedings of the Conference of the Association for Computational Linguistics (ACL\u201902)","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the Conference of the Association for Computational Linguistics (ACL\u201902). 311\u2013318."},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3406499.3415079"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3349537.3351899"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1086\/209405"},{"key":"e_1_3_3_93_2","first-page":"824","volume-title":"Proceedings of the 12th ACM International Conference on Web Search and Data Mining","author":"Priyogi Bilih","year":"2019","unstructured":"Bilih Priyogi. 2019. Preference elicitation strategy for conversational recommender system. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining. 824\u2013825."},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.bushor.2019.08.005"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/1111449.1111475"},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043962"},{"key":"e_1_3_3_97_2","doi-asserted-by":"publisher","DOI":"10.2753\/MIS0742-1222250405"},{"key":"e_1_3_3_98_2","article-title":"Evaluating quality of chatbots and intelligent conversational agents","author":"Radziwill Nicole M.","year":"2017","unstructured":"Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579 (2017).","journal-title":"arXiv preprint arXiv:1704.04579"},{"key":"e_1_3_3_99_2","article-title":"Application of humanization to survey chatbots: Change in chatbot perception, interaction experience, and survey data quality","volume":"126","author":"Rhim Jungwook","year":"2022","unstructured":"Jungwook Rhim, Minji Kwak, Yeaeun Gong, and Gahgene Gweon. 2022. Application of humanization to survey chatbots: Change in chatbot perception, interaction experience, and survey data quality. Comput. Hum. Behav. 126 (2022), 107034.","journal-title":"Comput. Hum. Behav."},{"key":"e_1_3_3_100_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-7637-6"},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12193-009-0028-2"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.1007\/1-4020-2730-3_2"},{"key":"e_1_3_3_103_2","first-page":"1399","volume-title":"Proceedings of the Conference on Computer Supported Cooperative Work","author":"Said Alan","year":"2013","unstructured":"Alan Said, Ben Fields, Brijnesh J. Jain, and Sahin Albayrak. 2013. User-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm. In Proceedings of the Conference on Computer Supported Cooperative Work. 1399\u20131408."},{"key":"e_1_3_3_104_2","doi-asserted-by":"publisher","DOI":"10.1080\/07421222.2020.1790204"},{"key":"e_1_3_3_105_2","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919)","author":"See Abigail","year":"2019","unstructured":"Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? How controllable attributes affect human judgments. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919)."},{"key":"e_1_3_3_106_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Seering Joseph","year":"2020","unstructured":"Joseph Seering, Michal Luria, Connie Ye, Geoff Kaufman, and Jessica Hammer. 2020. It takes a village: Integrating an adaptive chatbot into an online gaming community. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1\u201313."},{"key":"e_1_3_3_107_2","article-title":"How do users interact with algorithm recommender systems? The interaction of users, algorithms, and performance","volume":"109","author":"Shin Donghee","year":"2020","unstructured":"Donghee Shin. 2020. How do users interact with algorithm recommender systems? The interaction of users, algorithms, and performance. Comput. Hum. Behav. 109 (2020), 106344.","journal-title":"Comput. Hum. Behav."},{"issue":"12","key":"e_1_3_3_108_2","doi-asserted-by":"crossref","first-page":"2680","DOI":"10.1177\/1461444821993801","article-title":"The perception of humanness in conversational journalism: An algorithmic information-processing perspective","volume":"24","author":"Shin Donghee","year":"2022","unstructured":"Donghee Shin. 2022. The perception of humanness in conversational journalism: An algorithmic information-processing perspective. New Media Societ. 24, 12 (2022), 2680\u20132704.","journal-title":"New Media Societ."},{"key":"e_1_3_3_109_2","doi-asserted-by":"publisher","DOI":"10.1509\/jmkg.69.1.32.55512"},{"issue":"1","key":"e_1_3_3_110_2","article-title":"On the sins of short-form development.","volume":"12","author":"Smith Gregory T.","year":"2000","unstructured":"Gregory T. Smith, Denis M. McCarthy, and Kristen G. Anderson. 2000. On the sins of short-form development. Psychol. Assess. 12, 1 (2000), 102.","journal-title":"Psychol. Assess."},{"key":"e_1_3_3_111_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2022.2121038"},{"key":"e_1_3_3_112_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210002"},{"key":"e_1_3_3_113_2","first-page":"151","volume-title":"Proceedings of the 2nd Artificial Intelligence and Cloud Computing Conference","author":"Svenningsson Nina","year":"2019","unstructured":"Nina Svenningsson and Montathar Faraon. 2019. Artificial intelligence in conversational agents: A study of factors related to perceived humanness in chatbots. In Proceedings of the 2nd Artificial Intelligence and Cloud Computing Conference. 151\u2013161."},{"key":"e_1_3_3_114_2","volume-title":"Proceedings of the 26th International Conference on Intelligent User Interfaces","author":"Svikhuushina Ekaterina","year":"2021","unstructured":"Ekaterina Svikhuushina and Pearl Pu. 2021. Key qualities of conversational chatbots\u2014The PEACE model. In Proceedings of the 26th International Conference on Intelligent User Interfaces."},{"key":"e_1_3_3_115_2","first-page":"312","volume-title":"Designing Interactive Systems","author":"Swearingen Kirsten","year":"2002","unstructured":"Kirsten Swearingen and Rashmi Sinha. 2002. Interaction design for recommender systems. In Designing Interactive Systems, Vol. 6. Citeseer, 312\u2013334."},{"key":"e_1_3_3_116_2","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1145\/2507157.2507223","volume-title":"Proceedings of the 7th ACM Conference on Recommender Systems","author":"Taramigkou Maria","year":"2013","unstructured":"Maria Taramigkou, Efthimios Bothos, Konstantinos Christidis, Dimitris Apostolou, and Gregoris Mentzas. 2013. Escape the bubble: Guided exploration of music preferences for serendipity and novelty. In Proceedings of the 7th ACM Conference on Recommender Systems. 335\u2013338."},{"key":"e_1_3_3_117_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15327965pli0104_1"},{"key":"e_1_3_3_118_2","first-page":"203","volume-title":"Proceedings of the ACM Conference on Recommender Systems","author":"Tintarev Nava","year":"2007","unstructured":"Nava Tintarev. 2007. Explanations of recommendations. In Proceedings of the ACM Conference on Recommender Systems. 203\u2013206."},{"key":"e_1_3_3_119_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3_15"},{"key":"e_1_3_3_120_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9117-5"},{"key":"e_1_3_3_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3446906"},{"key":"e_1_3_3_122_2","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172959"},{"key":"e_1_3_3_123_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09281-5"},{"key":"e_1_3_3_124_2","volume-title":"Proceedings of the 9th International Conference on Spoken Language Processing","author":"Turunen Markku","year":"2006","unstructured":"Markku Turunen, Jaakko Hakulinen, and Anssi Kainulainen. 2006. Evaluation of a spoken dialogue system with usability tests and long-term pilot studies: Similarities and differences. In Proceedings of the 9th International Conference on Spoken Language Processing."},{"key":"e_1_3_3_125_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324900002503"},{"key":"e_1_3_3_126_2","doi-asserted-by":"crossref","first-page":"271","DOI":"10.3115\/976909.979652","volume-title":"Proceedings of the 35th Annual Meeting of the Association of Computational Linguistics","author":"Walker Marilyn","year":"1997","unstructured":"Marilyn Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proceedings of the 35th Annual Meeting of the Association of Computational Linguistics. 271\u2013280."},{"key":"e_1_3_3_127_2","first-page":"113","volume-title":"Proceedings of the ACM SIGIR on International Conference on Theory of Information Retrieval","author":"Wang Disen","year":"2020","unstructured":"Disen Wang and Hui Fang. 2020. Length adaptive regularization for retrieval-based chatbot models. In Proceedings of the ACM SIGIR on International Conference on Theory of Information Retrieval. 113\u2013120."},{"key":"e_1_3_3_128_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734454.1734578"},{"key":"e_1_3_3_129_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10257-014-0269-9"},{"key":"e_1_3_3_130_2","volume-title":"Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialogue Systems","author":"W\u00e4rnest\u00e5l Pontus","year":"2005","unstructured":"Pontus W\u00e4rnest\u00e5l. 2005. User evaluation of a conversational recommender system. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialogue Systems. Citeseer."},{"issue":"1","key":"e_1_3_3_131_2","first-page":"1","article-title":"Evaluating embodied conversational agents in multimodal interfaces","volume":"1","author":"Weiss Benjamin","year":"2015","unstructured":"Benjamin Weiss, Ina Wechsung, Christine K\u00fchnel, and Sebastian M\u00f6ller. 2015. Evaluating embodied conversational agents in multimodal interfaces. Comput. Cognit. Sci. 1, 1 (2015), 1\u201321.","journal-title":"Comput. Cognit. Sci."},{"key":"e_1_3_3_132_2","doi-asserted-by":"publisher","DOI":"10.23668\/psycharchives.12784"},{"key":"e_1_3_3_133_2","doi-asserted-by":"publisher","DOI":"10.1080\/10510974.2018.1557233"},{"issue":"1","key":"e_1_3_3_134_2","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1080\/00220973.2010.531299","article-title":"Using the modification index and standardized expected parameter change for model modification","volume":"80","author":"Whittaker Tiffany A.","year":"2012","unstructured":"Tiffany A. Whittaker. 2012. Using the modification index and standardized expected parameter change for model modification. J. Experim. Educ. 80, 1 (2012), 26\u201344.","journal-title":"J. Experim. Educ."},{"key":"e_1_3_3_135_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2021.08.016"},{"key":"e_1_3_3_136_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-018-9205-x"},{"key":"e_1_3_3_137_2","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1007\/11768012_25","volume-title":"Proceedings of the 4th International Conference on Adaptive Hypermedia and Adaptive Web-based Systems (AH\u201906)","author":"Zhang Jiyong","year":"2006","unstructured":"Jiyong Zhang and Pearl Pu. 2006. A comparative study of compound critique generation in conversational recommender systems. In Proceedings of the 4th International Conference on Adaptive Hypermedia and Adaptive Web-based Systems (AH\u201906). Springer, 234\u2013243."},{"key":"e_1_3_3_138_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1205"},{"key":"e_1_3_3_139_2","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1145\/3267851.3267880","volume-title":"Proceedings of the 18th International Conference on Intelligent Virtual Agents","author":"Zhao Ran","year":"2018","unstructured":"Ran Zhao, Oscar J. Romero, and Alex Rudnicky. 2018. SOGO: A social intelligent negotiation dialogue system. In Proceedings of the 18th International Conference on Intelligent Virtual Agents. 239\u2013246."}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631534","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3631534","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:53Z","timestamp":1750291433000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631534"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,7]]},"references-count":138,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3631534"],"URL":"https:\/\/doi.org\/10.1145\/3631534","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,7]]},"assertion":[{"value":"2022-12-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}