{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T20:06:54Z","timestamp":1792526814311,"version":"4.5.0"},"reference-count":121,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T00:00:00Z","timestamp":1657584000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T00:00:00Z","timestamp":1657584000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008905","name":"University of Klagenfurt","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008905","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Artif Intell Rev"],"published-print":{"date-parts":[[2023,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Conversational recommender systems aim to interactively support online users in their information search and decision-making processes in an intuitive way. With the latest advances in voice-controlled devices, natural language processing, and AI in general, such systems received increased attention in recent years. Technically, conversational recommenders are usually complex multi-component applications and often consist of multiple machine learning models and a natural language user interface. Evaluating such a complex system in a holistic way can therefore be challenging, as it requires (i) the assessment of the quality of the different learning components, and (ii) the quality perception of the system as a whole by users. Thus, a mixed methods approach is often required, which may combine objective (computational) and subjective (perception-oriented) evaluation techniques. In this paper, we review common evaluation approaches for conversational recommender systems, identify possible limitations, and outline future directions towards more holistic evaluation practices.<\/jats:p>","DOI":"10.1007\/s10462-022-10229-x","type":"journal-article","created":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T07:05:04Z","timestamp":1657609504000},"page":"2365-2400","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":62,"title":["Evaluating conversational recommender systems"],"prefix":"10.1007","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4698-8507","authenticated-orcid":false,"given":"Dietmar","family":"Jannach","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,7,12]]},"reference":[{"key":"10229_CR1","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s11257-019-09256-1","volume":"30","author":"H Abdollahpouri","year":"2020","unstructured":"Abdollahpouri H, Adomavicius G, Burke R, Guy I, Jannach D, Kamishima T, Krasnodebski J, Pizzato L (2020) Multistakeholder recommendation: survey and research directions. User Model User Adapt Interact 30:127\u2013158","journal-title":"User Model User Adapt Interact"},{"issue":"1","key":"10229_CR2","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1287\/isre.2017.0703","volume":"29","author":"G Adomavicius","year":"2018","unstructured":"Adomavicius G, Bockstedt JC, Curley SP, Zhang J (2018) Effects of online recommendations on consumers\u2019 willingness to pay. Inf Syst Res 29(1):84\u2013102","journal-title":"Inf Syst Res"},{"key":"10229_CR3","doi-asserted-by":"crossref","unstructured":"\u00c1lvarez\u00a0M\u00e1rquez JO, Ziegler J (2016) Hootle+: a group recommender system supporting preference negotiation. In: Collaboration and technology. pp 151\u2013166","DOI":"10.1007\/978-3-319-44799-5_12"},{"key":"10229_CR4","unstructured":"Angara P, Jim\u00e9nez M, Agarwal K, Jain H, Jain R, Stege U, Ganti S, M\u00fcller HA, Ng JW (2017) Foodie Fooderson: a conversational agent for the smart kitchen. In: CASCON\u201917. pp 247\u2013253"},{"key":"10229_CR5","doi-asserted-by":"crossref","unstructured":"Argal A, Gupta S, Modi A, Pandey P, Shim S, Choo C (2018) Intelligent travel chatbot for predictive recommendation in Echo platform. In: CCWC\u201918. pp 176\u2013183","DOI":"10.1109\/CCWC.2018.8301732"},{"key":"10229_CR6","doi-asserted-by":"crossref","unstructured":"Ashktorab Z, Jain M, Liao QV, Weisz JD (2019) Resilient chatbots: repair strategy preferences for conversational breakdowns. In: CHI\u201919. p 254","DOI":"10.1145\/3290605.3300484"},{"key":"10229_CR7","doi-asserted-by":"crossref","unstructured":"Averjanova O, Ricci F, Nguyen Q (2008a) Map-based interaction with a conversational mobile recommender system. In: UBICOMM\u201908. pp 212\u2013218","DOI":"10.1109\/UBICOMM.2008.16"},{"key":"10229_CR8","doi-asserted-by":"crossref","unstructured":"Averjanova O, Ricci F, Nguyen QN (2008b) Map-based interaction with a conversational mobile recommender system. In: UBICOMM\u201908. pp 212\u2013218","DOI":"10.1109\/UBICOMM.2008.16"},{"key":"10229_CR9","doi-asserted-by":"crossref","unstructured":"Baizal ZA, Murti YR, Adiwijaya (2017) Evaluating functional requirements-based compound critiquing on conversational recommender system. In: 5th International conference on information and communication technology (ICoIC7). pp 1\u20136","DOI":"10.1109\/ICoICT.2017.8074656"},{"key":"10229_CR10","doi-asserted-by":"crossref","unstructured":"Balaraman V, Sheikhalishahi S, Magnini B (2021) Recent neural methods on dialogue state tracking for task-oriented dialogue systems: a survey. In: Proceedings of the 22nd annual meeting of the special interest group on discourse and dialogue. pp 239\u2013251","DOI":"10.18653\/v1\/2021.sigdial-1.25"},{"key":"10229_CR11","unstructured":"Burke R (1999) The Wasabi personal shopper: a case-based recommender system. In: AAAI\u201999. pp 844\u2013849"},{"key":"10229_CR12","unstructured":"Burke RD, Hammond KJ, Young BC (1996) Knowledge-based navigation of complex information spaces. In: AAAI\u201996. pp 462\u2013468"},{"key":"10229_CR13","doi-asserted-by":"crossref","unstructured":"Cai W, Chen L (2020) Predicting user intents and satisfaction with dialogue-based conversational recommendations. In: UMAP\u201920. pp 33\u201342","DOI":"10.1145\/3340631.3394856"},{"key":"10229_CR14","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.patrec.2017.06.011","volume":"99","author":"BD Carolis","year":"2017","unstructured":"Carolis BD, de Gemmis M, Lops P, Palestra G (2017) Recognizing users feedback from non-verbal communicative acts in conversational recommender systems. Pattern Recognit Lett 99:87\u201395","journal-title":"Pattern Recognit Lett"},{"key":"10229_CR15","doi-asserted-by":"crossref","unstructured":"Cerezo J, Kubelka J, Robbes R, Bergel A (2019) Building an expert recommender chatbot. In: 2019 IEEE\/ACM 1st international workshop on Bots in software engineering (BotSE). pp 59\u201363","DOI":"10.1109\/BotSE.2019.00022"},{"key":"10229_CR16","doi-asserted-by":"crossref","unstructured":"Chandrashekara AA, Talluri RKM, Sivarathri SS, Mitra R, Calyam P, Kee K, Nair S (2018) Fuzzy-based conversational recommender for data-intensive science gateway applications. In: BigData\u201918. pp 4870\u20134875","DOI":"10.1109\/BigData.2018.8622046"},{"issue":"1\u20132","key":"10229_CR18","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1007\/s11257-011-9108-6","volume":"22","author":"L Chen","year":"2012","unstructured":"Chen L, Pu P (2012) Critiquing-based recommenders: survey and emerging trends. User Model User Adapt Interact 22(1\u20132):125\u2013150","journal-title":"User Model User Adapt Interact"},{"issue":"2","key":"10229_CR17","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1145\/3166054.3166058","volume":"19","author":"H Chen","year":"2017","unstructured":"Chen H, Liu X, Yin D, Tang J (2017) A survey on dialogue systems: recent advances and new frontiers. SIGKDD Explor Newsl 19(2):25\u201335","journal-title":"SIGKDD Explor Newsl"},{"key":"10229_CR19","doi-asserted-by":"crossref","unstructured":"Chen Q, Lin J, Zhang Y, Ding M, Cen Y, Yang H, Tang J (2019) Towards knowledge-based recommender dialog system. In: EMNLP-IJCNLP\u201919. pp 1803\u20131813","DOI":"10.18653\/v1\/D19-1189"},{"key":"10229_CR20","doi-asserted-by":"crossref","unstructured":"Christakopoulou K, Radlinski F, Hofmann K (2016) Towards conversational recommender systems. In: KDD\u201916. pp 815\u2013824","DOI":"10.1145\/2939672.2939746"},{"key":"10229_CR21","doi-asserted-by":"crossref","unstructured":"Christakopoulou K, Beutel A, Li R, Jain S, Chi EH (2018) Q &R: a two-stage approach toward interactive recommendation. In: KDD\u201918. pp 139\u2013148","DOI":"10.1145\/3219819.3219894"},{"key":"10229_CR22","doi-asserted-by":"crossref","unstructured":"Clarizia F, Colace F, Lombardi M, Pascale F (2018) A context aware recommender system for digital storytelling. In: AINA\u201918. pp 542\u2013549","DOI":"10.1109\/AINA.2018.00085"},{"key":"10229_CR23","doi-asserted-by":"crossref","unstructured":"Colace F, De\u00a0Santo M, Pascale F, Lemma S, Lombardi M (2017) BotWheels: a petri net based chatbot for recommending tires. In: DATA\u201917. pp 350\u2013358","DOI":"10.5220\/0006491903500358"},{"key":"10229_CR24","doi-asserted-by":"crossref","unstructured":"Contreras D, Salam\u00f3 M, Rodr\u00edguez I, Puig A (2014) An approach to improve user experience with conversational recommenders through a 3D virtual environment. In: Proceedings of the XV international conference on human computer interaction, interacci\u00f3n\u201914","DOI":"10.1145\/2662253.2662319"},{"issue":"4","key":"10229_CR25","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1109\/MCE.2017.2728819","volume":"7","author":"D Contreras","year":"2018","unstructured":"Contreras D, Salamo M, Rodriguez I, Puig A (2018) Shopping decisions made in a virtual world: defining a state-based model of collaborative and conversational user-recommender interactions. IEEE Consum Electron Mag 7(4):26\u201335","journal-title":"IEEE Consum Electron Mag"},{"issue":"3","key":"10229_CR26","doi-asserted-by":"publisher","first-page":"319","DOI":"10.2307\/249008","volume":"13","author":"FD Davis","year":"1989","unstructured":"Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 13(3):319\u2013340","journal-title":"MIS Q"},{"key":"10229_CR27","doi-asserted-by":"crossref","unstructured":"Delgado J, Davidson R (2002) Knowledge bases and user profiling in travel and hospitality recommender systems. In: ENTER\u201902","DOI":"10.1007\/978-3-7091-6132-6_1"},{"key":"10229_CR28","unstructured":"Dietz LW, Myftija S, W\u00f6rndl W (2019) Designing a conversational travel recommender system based on data-driven destination characterization. In: ACM RecSys workshop on recommenders in tourism. pp 17\u201321"},{"issue":"01","key":"10229_CR29","doi-asserted-by":"publisher","first-page":"009","DOI":"10.1055\/s-0039-1688757","volume":"58","author":"A Fadhil","year":"2019","unstructured":"Fadhil A, Wang Y, Reiterer H (2019) Assistive conversational agent for health coaching: a validation study. Methods Inf Med 58(01):009\u2013023","journal-title":"Methods Inf Med"},{"key":"10229_CR30","doi-asserted-by":"crossref","unstructured":"Ferraro A, Jannach D, Serra X (2020) Exploring longitudinal effects of session-based recommendations. In: Proceedings of the 2020 ACM conference on recommender systems (RecSys\u201920)","DOI":"10.1145\/3383313.3412213"},{"key":"10229_CR31","unstructured":"Finch SE, Choi JD (2020) Towards unified dialogue system evaluation: a comprehensive analysis of current evaluation protocols. In: Proceedings of the 21th annual meeting of the special interest group on discourse and dialogue (SIGDIAL\u201920). pp 236\u2013245"},{"key":"10229_CR32","doi-asserted-by":"crossref","unstructured":"Gao J, Galley M, Li L (2018) Neural approaches to conversational AI. In: The 41st international ACM SIGIR conference on research and development in information retrieval (SIGIR\u201918), SIGIR\u201918. pp 1371\u20131374","DOI":"10.1145\/3209978.3210183"},{"key":"10229_CR33","doi-asserted-by":"crossref","unstructured":"Ghazvininejad M, Brockett C, Chang M, Dolan B, Gao J, Yih W, Galley M (2018) A knowledge-grounded neural conversation model. In: AAAI\u201918. pp 5110\u20135117","DOI":"10.1609\/aaai.v32i1.11977"},{"issue":"4","key":"10229_CR34","first-page":"13:1","volume":"6","author":"CA Gomez-Uribe","year":"2015","unstructured":"Gomez-Uribe CA, Hunt N (2015) The Netflix recommender system: algorithms, business value, and innovation. Trans Manag Inf Syst 6(4):13:1-13:19","journal-title":"Trans Manag Inf Syst"},{"key":"10229_CR35","doi-asserted-by":"crossref","unstructured":"Grasch P, Felfernig A, Reinfrank F (2013) ReComment: towards critiquing-based recommendation with speech interaction. In: RecSys\u201913. pp 157\u2013164","DOI":"10.1145\/2507157.2507161"},{"key":"10229_CR36","doi-asserted-by":"crossref","unstructured":"Greco C, Suglia A, Basile P, Semeraro G (2017) Converse-Et-Impera: exploiting deep learning and hierarchical reinforcement learning for conversational recommender systems. In: AI*IA 2017. pp 372\u2013386","DOI":"10.1007\/978-3-319-70169-1_28"},{"key":"10229_CR37","unstructured":"Hammond KJ, Burke R, Schmitt K (1994) A case-based approach to knowledge navigation. In: AAAI\u201994 KDD workshop. pp 383\u2013393"},{"key":"10229_CR38","doi-asserted-by":"crossref","unstructured":"Hayati SA, Kang D, Zhu Q, Shi W, Yu Z (2020) INSPIRED: toward sociable recommendation dialog systems. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). pp 8142\u20138152","DOI":"10.18653\/v1\/2020.emnlp-main.654"},{"issue":"1","key":"10229_CR39","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1145\/963770.963772","volume":"22","author":"JL Herlocker","year":"2004","unstructured":"Herlocker JL, Konstan JA, Terveen LG, Riedl JT (2004) Evaluating collaborative filtering recommender systems. Trans Inf Syst 22(1):5\u201353","journal-title":"Trans Inf Syst"},{"key":"10229_CR40","doi-asserted-by":"publisher","DOI":"10.1561\/9781680831627","volume-title":"Online evaluation for information retrieval","author":"K Hofmann","year":"2016","unstructured":"Hofmann K, Li L, Radlinski F (2016) Online evaluation for information retrieval. Now Publishers Inc, Boston"},{"key":"10229_CR41","doi-asserted-by":"crossref","unstructured":"Hong ZW, Huang RT, Chin KY, Yen CC, Lin JM (2010) An interactive agent system for supporting knowledge-based recommendation: a case study on an e-Novel recommender system. In: ICUIMC\u201910. pp 53:1\u201353:8","DOI":"10.1145\/2108616.2108681"},{"key":"10229_CR42","doi-asserted-by":"publisher","first-page":"113250","DOI":"10.1016\/j.dss.2020.113250","volume":"131","author":"A Iovine","year":"2020","unstructured":"Iovine A, Narducci F, Semeraro G (2020) Conversational recommender systems and natural language: a study through the ConveRSE framework. Decis Support Syst 131:113250","journal-title":"Decis Support Syst"},{"key":"10229_CR43","unstructured":"Jannach D (2004) ADVISOR SUITE\u2014a knowledge-based sales advisory system. In: ECAI\u201904. pp 720\u2013724"},{"key":"10229_CR44","doi-asserted-by":"crossref","unstructured":"Jannach D, Adomavicius G (2016) Recommendations with a purpose. In: Proceedings of the 10th ACM conference on recommender systems, RecSys\u201916. pp 7\u201310","DOI":"10.1145\/2959100.2959186"},{"issue":"4","key":"10229_CR45","first-page":"79","volume":"41","author":"D Jannach","year":"2020","unstructured":"Jannach D, Bauer C (2020) Escaping the McNamara Fallacy: towards more impactful recommender systems research. AI Mag 41(4):79\u201395","journal-title":"AI Mag"},{"issue":"4","key":"10229_CR46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3370082","volume":"10","author":"D Jannach","year":"2019","unstructured":"Jannach D, Jugovac M (2019) Measuring the business value of recommender systems. ACM TMIS 10(4):1\u201323","journal-title":"ACM TMIS"},{"key":"10229_CR47","unstructured":"Jannach D, Manzoor A (2020) End-to-end learning for conversational recommendation: a long way to go? In: IntRS workshop at ACM RecSys 2020"},{"issue":"5","key":"10229_CR48","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1007\/s11257-015-9165-3","volume":"25","author":"D Jannach","year":"2015","unstructured":"Jannach D, Lerche L, Kamehkhosh I, Jugovac M (2015) What recommenders recommend: an analysis of recommendation biases and possible countermeasures. User Model User Adapt Interact 25(5):427\u2013491","journal-title":"User Model User Adapt Interact"},{"issue":"5","key":"10229_CR49","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3453154","volume":"54","author":"D Jannach","year":"2021","unstructured":"Jannach D, Manzoor A, Cai W, Chen L (2021) A survey on conversational recommender systems. ACM Comput Surv 54(5):1\u201336","journal-title":"ACM Comput Surv"},{"key":"10229_CR50","doi-asserted-by":"crossref","unstructured":"Jin Y, Cai W, Chen L, Htun NN, Verbert K (2019) MusicBot: evaluating critiquing-based music recommenders with conversational interaction. In: CIKM\u201919. pp 951\u2013960","DOI":"10.1145\/3357384.3357923"},{"key":"10229_CR51","doi-asserted-by":"crossref","unstructured":"Kamei K, Shinozawa K, Ikeda T, Utsumi A, Miyashita T, Hagita N (2010) Recommendation from robots in a real-world retail shop. In: International conference on multimodal interfaces and the workshop on machine learning for multimodal interaction, ICMI-MLMI\u201910","DOI":"10.1145\/1891903.1891929"},{"issue":"1\u20132","key":"10229_CR52","first-page":"1","volume":"3","author":"D Kelly","year":"2009","unstructured":"Kelly D (2009) Methods for evaluating interactive information retrieval systems with users. Found Trends Inf Retr 3(1\u20132):1\u2013224","journal-title":"Found Trends Inf Retr"},{"issue":"3","key":"10229_CR53","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1111\/j.1467-8535.1993.tb00076.x","volume":"24","author":"J Kirakowski","year":"1993","unstructured":"Kirakowski J, Corbett M (1993) Sumi: the software usability measurement inventory. Br J Educ Technol 24(3):210\u2013212","journal-title":"Br J Educ Technol"},{"issue":"4","key":"10229_CR54","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1007\/s11257-011-9118-4","volume":"22","author":"B Knijnenburg","year":"2012","unstructured":"Knijnenburg B, Willemsen M, Gantner Z, Soncu H, Newell C (2012) Explaining the user experience of recommender systems. User Model User Adapt Interact 22(4):441\u2013504","journal-title":"User Model User Adapt Interact"},{"key":"10229_CR55","doi-asserted-by":"publisher","DOI":"10.1017\/9781108653985","volume-title":"Trustworthy online controlled experiments: a practical guide to A\/B testing","author":"R Kohavi","year":"2020","unstructured":"Kohavi R, Tang D, Xu Y (2020) Trustworthy online controlled experiments: a practical guide to A\/B testing. Cambridge University Press, Cambridge"},{"key":"10229_CR56","doi-asserted-by":"publisher","unstructured":"Kostric I, Balog K, Radlinski F (2021) Soliciting user preferences in conversational recommender systems via usage-related questions. Proceedings of fifteenth ACM conference on recommender systems (RecSys \u201921), pp. 724\u2013729. https:\/\/doi.org\/10.1145\/3460231.3478861","DOI":"10.1145\/3460231.3478861"},{"key":"10229_CR57","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1016\/j.ijhcs.2017.02.005","volume":"103","author":"S Lee","year":"2017","unstructured":"Lee S, Choi J (2017) Enhancing user experience with conversational agent for movie recommendation: effects of self-disclosure and reciprocity. Int J Hum Comput Stud 103:95\u2013105","journal-title":"Int J Hum Comput Stud"},{"key":"10229_CR58","unstructured":"Li R, Kahou SE, Schulz H, Michalski V, Charlin L, Pal C (2018) Towards deep conversational recommendations. In: NIPS\u201918. pp 9725\u20139735"},{"key":"10229_CR59","unstructured":"Liao L, Takanobu R, Ma Y, Yang X, Huang M, Chua TS (2019) Deep conversational recommender in travel. http:\/\/arxiv.org\/abs\/1907.00710"},{"key":"10229_CR60","doi-asserted-by":"publisher","first-page":"1031","DOI":"10.1002\/mar.21491","volume":"38","author":"EC Ling","year":"2021","unstructured":"Ling EC, Tussyadiah I, Tuomi A, Stienmetz J, Ioannou A (2021) Factors influencing users\u2019 adoption and use of conversational agents: a systematic review. Psychol Mark 38:1031\u20131051","journal-title":"Psychol Mark"},{"key":"10229_CR61","doi-asserted-by":"crossref","unstructured":"Liu CW, Lowe R, Serban I, Noseworthy M, Charlin L, Pineau J (2016) How NOT to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. In: EMNLP\u201916. pp 2122\u20132132","DOI":"10.18653\/v1\/D16-1230"},{"key":"10229_CR62","doi-asserted-by":"crossref","unstructured":"Liu Z, Wang H, Niu ZY, Wu H, Che W, Liu T (2020) Towards conversational recommendation over multi-type dialogs. In: Proceedings of the 58th annual meeting of the association for computational linguistics. pp 1036\u20131049","DOI":"10.18653\/v1\/2020.acl-main.98"},{"issue":"10","key":"10229_CR63","doi-asserted-by":"publisher","first-page":"1876","DOI":"10.1109\/TKDE.2011.116","volume":"24","author":"MS Llorente","year":"2012","unstructured":"Llorente MS, Guerrero SE (2012) Increasing retrieval quality in conversational recommenders. IEEE Trans. Knowl. Data Eng. 24(10):1876\u20131888","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"10229_CR64","doi-asserted-by":"crossref","unstructured":"Loepp B, Hussein T, Ziegler J (2014) Choice-based preference elicitation for collaborative filtering recommender systems. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI\u201914. pp 3085\u20133094","DOI":"10.1145\/2556288.2557069"},{"issue":"4","key":"10229_CR65","first-page":"385","volume":"2","author":"S Loh","year":"2010","unstructured":"Loh S, Lichtnow D, Kampff AJC, de Oliveira JPM (2010) Recommendation of complementary material during chat discussions. Knowl Manag E-Learn 2(4):385\u2013399","journal-title":"Knowl Manag E-Learn"},{"key":"10229_CR66","doi-asserted-by":"crossref","unstructured":"Lombardi M, Pascale F, Santaniello D (2019) An application for cultural heritage using a chatbot. In: 2019 2nd international conference on computer applications information security (ICCAIS). pp 1\u20135","DOI":"10.1109\/CAIS.2019.8769525"},{"key":"10229_CR67","doi-asserted-by":"crossref","unstructured":"Louvan S, Magnini B (2020) Recent neural methods on slot filling and intent classification for task-oriented dialogue systems: a survey. In: Proceedings of the 28th international conference on computational linguistics, COLING 2020. pp 480\u2013496","DOI":"10.18653\/v1\/2020.coling-main.42"},{"key":"10229_CR68","doi-asserted-by":"crossref","unstructured":"Mahmood T, Ricci F (2009) Improving recommender systems with adaptive conversational strategies. In: HT\u201909. pp 73\u201382","DOI":"10.1145\/1557914.1557930"},{"key":"10229_CR69","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1007\/s10257-013-0222-3","volume":"12","author":"T Mahmood","year":"2014","unstructured":"Mahmood T, Mujtaba G, Venturini A (2014) Dynamic personalization in conversational recommender systems. Inf Syst e-Bus Manag 12:213\u2013238","journal-title":"Inf Syst e-Bus Manag"},{"key":"10229_CR70","doi-asserted-by":"crossref","unstructured":"Manzoor A, Jannach D (2021) Generation-based vs. retrieval-based conversational recommendation: a user-centric comparison. In: 15th ACM conference on recommender systems (RecSys\u201921)","DOI":"10.1145\/3460231.3475942"},{"key":"10229_CR71","doi-asserted-by":"crossref","unstructured":"McCarthy K, Reilly J, McGinty L, Smyth B (2004) On the dynamic generation of compound critiques in conversational recommender systems. In: AH\u201904. pp 176\u2013184","DOI":"10.1007\/978-3-540-27780-4_21"},{"issue":"3","key":"10229_CR72","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1287\/isre.13.3.334.81","volume":"13","author":"DH McKnight","year":"2002","unstructured":"McKnight DH, Choudhury V, Kacmar CJ (2002) Developing and validating trust measures for e-commerce: an integrative typology. Inf Syst Res 13(3):334\u2013359","journal-title":"Inf Syst Res"},{"key":"10229_CR73","doi-asserted-by":"crossref","unstructured":"Moon S, Shah P, Kumar A, Subba R (2019) OpenDialKG: explainable conversational reasoning with attention-based walks over knowledge graphs. In: ACL\u201919. pp 845\u2013854","DOI":"10.18653\/v1\/P19-1081"},{"key":"10229_CR74","doi-asserted-by":"crossref","unstructured":"Narducci F, de\u00a0Gemmis M, Lops P, Semeraro G (2018) Improving the user experience with a conversational recommender system. In: AI*IA\u201918. pp 528\u2013538","DOI":"10.1007\/978-3-030-03840-3_39"},{"key":"10229_CR75","doi-asserted-by":"crossref","unstructured":"Nie L, Wang W, Hong R, Wang M, Tian Q (2019) Multimodal dialog system: generating responses via adaptive decoders. In: MM\u201919. pp 1098\u20131106","DOI":"10.1145\/3343031.3350923"},{"issue":"1","key":"10229_CR76","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1080\/01449290903004012","volume":"29","author":"AA Ozok","year":"2010","unstructured":"Ozok AA, Fan Q, Norcio AF (2010) Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: Results from a college student population. Behav Inf Technol 29(1):57\u201383","journal-title":"Behav Inf Technol"},{"key":"10229_CR77","doi-asserted-by":"crossref","unstructured":"Pecune F, Murali S, Tsai V, Matsuyama Y, Cassell J (2019a) A model of social explanations for a conversational movie recommendation system. In: Proceedings of the 7th international conference on human-agent interaction, HAI\u201919. pp 135\u2013143","DOI":"10.1145\/3349537.3351899"},{"key":"10229_CR78","doi-asserted-by":"crossref","unstructured":"Pecune F, Murali S, Tsai V, Matsuyama Y, Cassell J (2019b) A model of social explanations for a conversational movie recommendation system. In: Proceedings of the 7th international conference on human-agent interaction, HAI\u201919. pp 135\u2013143","DOI":"10.1145\/3349537.3351899"},{"key":"10229_CR79","doi-asserted-by":"crossref","unstructured":"Pu P, Chen L (2010) A user-centric evaluation framework of recommender systems. In: Proceedings of the ACM RecSys 2010 workshop on user-centric evaluation of recommender systems and their interfaces (UCERSTI). pp 14\u201321","DOI":"10.1145\/2043932.2043962"},{"key":"10229_CR80","doi-asserted-by":"crossref","unstructured":"Pu P, Zhou M, Castagnos S (2009) Critiquing recommenders for public taste products. In: RecSys\u201909. pp 249\u2013252","DOI":"10.1145\/1639714.1639760"},{"key":"10229_CR81","doi-asserted-by":"crossref","unstructured":"Pu P, Chen L, Hu R (2011) A user-centric evaluation framework for recommender systems. In: RecSys\u201911. pp 157\u2013164","DOI":"10.1145\/2043932.2043962"},{"issue":"4\u20135","key":"10229_CR82","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/s11257-011-9115-7","volume":"22","author":"P Pu","year":"2012","unstructured":"Pu P, Chen L, Hu R (2012) Evaluating recommender systems from the user\u2019s perspective: survey of the state of the art. User Model User Adapt Interact 22(4\u20135):317\u2013355","journal-title":"User Model User Adapt Interact"},{"key":"10229_CR83","doi-asserted-by":"crossref","unstructured":"Qiu M, Li FL, Wang S, Gao X, Chen Y, Zhao W, Chen H, Huang J, Chu W (2017) Alime chat: a sequence to sequence and rerank based chatbot engine. In: ACL\u201917. pp 498\u2013503","DOI":"10.18653\/v1\/P17-2079"},{"key":"10229_CR84","doi-asserted-by":"crossref","unstructured":"Radlinski F, Boutilier C, Ramachandran D, Vendrov I (2022) Subjective attributes in conversational recommendation systems: challenges and opportunities. In: Proceedings of the 36th AAAI conference on artificial intelligence","DOI":"10.1609\/aaai.v36i11.21492"},{"key":"10229_CR85","unstructured":"Radziwill NM, Benton MC (2017) Evaluating quality of chatbots and intelligent conversational agents. http:\/\/arxiv.org\/abs\/1704.04579"},{"issue":"3\u20134","key":"10229_CR86","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1007\/s10462-005-9004-8","volume":"24","author":"R Rafter","year":"2005","unstructured":"Rafter R, Smyth B (2005) Conversational collaborative recommendation\u2014an experimental analysis. Artif Intell Rev 24(3\u20134):301\u2013318","journal-title":"Artif Intell Rev"},{"issue":"4","key":"10229_CR87","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3394592","volume":"38","author":"X Ren","year":"2020","unstructured":"Ren X, Yin H, Chen T, Wang H, Hung NQV, Huang Z, Zhang X (2020) CRSAL: conversational recommender systems with adversarial learning. ACM Trans Inf Syst 38(4):1\u201340","journal-title":"ACM Trans Inf Syst"},{"issue":"3","key":"10229_CR88","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MIS.2007.43","volume":"22","author":"F Ricci","year":"2007","unstructured":"Ricci F, Nguyen QN (2007) Acquiring and revising preferences in a critique-based mobile recommender system. Intell Syst 22(3):22\u201329","journal-title":"Intell Syst"},{"key":"10229_CR89","doi-asserted-by":"crossref","unstructured":"Ricci F, Nguyen QN, Averjanova O (2010) Exploiting a map-based interface in conversational recommender systems for mobile travelers. In: Tourism informatics, IGI. pp 73\u201379","DOI":"10.4018\/978-1-60566-818-5.ch005"},{"issue":"4","key":"10229_CR90","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1561\/1500000009","volume":"4","author":"M Sanderson","year":"2010","unstructured":"Sanderson M (2010) Test collection based evaluation of information retrieval systems. Found Trends\u00ae Inf Retr 4(4):247\u2013375","journal-title":"Found Trends\u00ae Inf Retr"},{"key":"10229_CR91","first-page":"265","volume-title":"Recommender systems handbook","author":"G Shani","year":"2015","unstructured":"Shani G, Gunawardana A (2015) Evaluating recommendation systems. Recommender systems handbook. Springer US, New York, pp 265\u2013308"},{"issue":"3\u20134","key":"10229_CR92","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1023\/A:1020757023711","volume":"18","author":"H Shimazu","year":"2002","unstructured":"Shimazu H (2002) ExpertClerk: a conversational case-based reasoning tool for developing salesclerk agents in E-commerce webshops. Artif Intell Rev 18(3\u20134):223\u2013244","journal-title":"Artif Intell Rev"},{"key":"10229_CR93","doi-asserted-by":"crossref","unstructured":"Siangchin N, Samanchuen T (2019) Chatbot implementation for ICD-10 recommendation system. In: ICESI\u201919. pp 1\u20136","DOI":"10.1109\/ICESI.2019.8863009"},{"key":"10229_CR94","unstructured":"Smyth B, McGinty L (2003) An analysis of feedback strategies in conversational recommender systems. In: Proceedings of the 14th national conference on artificial intelligence and cognitive science (AICS\u201903). pp 211\u2013216"},{"key":"10229_CR95","doi-asserted-by":"crossref","unstructured":"Smyth B, McGinty L, Reilly J, McCarthy K (2004) Compound critiques for conversational recommender systems. In: WI\u201904. pp 145\u2013151","DOI":"10.1109\/WI.2004.10098"},{"key":"10229_CR96","doi-asserted-by":"crossref","unstructured":"Sun M, Li F, Lee J, Zhou K, Lebanon G, Zha H (2013) Learning multiple-question decision trees for cold-start recommendation. In: WSDM\u201913. pp 445\u2013454","DOI":"10.1145\/2433396.2433451"},{"issue":"1","key":"10229_CR97","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1613\/jair.1318","volume":"21","author":"CA Thompson","year":"2004","unstructured":"Thompson CA, G\u00f6ker MH, Langley P (2004) A personalized system for conversational recommendations. J Artif Intell Res 21(1):393\u2013428","journal-title":"J Artif Intell Res"},{"key":"10229_CR98","doi-asserted-by":"crossref","unstructured":"Trabelsi W, Wilson N, Bridge D (2013) Comparative preferences induction methods for conversational recommenders. In: Proceedings of the third international conference on algorithmic decision theory, ADT 2013. pp 363\u2013374","DOI":"10.1007\/978-3-642-41575-3_28"},{"key":"10229_CR99","doi-asserted-by":"crossref","unstructured":"Tsumita D, Takagi T (2019) Dialogue based recommender system that flexibly mixes utterances and recommendations. In: WI\u201919. pp 51\u201358","DOI":"10.1145\/3350546.3352500"},{"key":"10229_CR100","doi-asserted-by":"crossref","unstructured":"Viappiani P, Pu P, Faltings B (2007) Conversational recommenders with adaptive suggestions. In: RecSys\u201907. pp 89\u201396","DOI":"10.1145\/1297231.1297246"},{"issue":"5","key":"10229_CR101","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1207\/s15516709cog2805_8","volume":"28","author":"M Walker","year":"2004","unstructured":"Walker M, Whittaker S, Stent A, Maloor P, Moore J, Johnston M, Vasireddy G (2004) Generation and evaluation of user tailored responses in multimodal dialogue. Cogn Sci 28(5):811\u2013840","journal-title":"Cogn Sci"},{"issue":"3","key":"10229_CR102","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1287\/isre.1120.0445","volume":"24","author":"W Wang","year":"2013","unstructured":"Wang W, Benbasat I (2013) Research note\u2014a contingency approach to investigating the effects of user-system interaction modes of online decision aids. Inf Syst Res 24(3):861\u2013876","journal-title":"Inf Syst Res"},{"key":"10229_CR103","doi-asserted-by":"crossref","unstructured":"W\u00e4rnest\u00e5l P (2005) User evaluation of a conversational recommender system. In: IJCAI\u201905 workshop on knowledge and reasoning in practical dialogue systems","DOI":"10.1007\/11527886_78"},{"key":"10229_CR104","doi-asserted-by":"crossref","unstructured":"Widyantoro DH, Baizal Z (2014) A framework of conversational recommender system based on user functional requirements. In: ICoICT\u201914. pp 160\u2013165","DOI":"10.1109\/ICoICT.2014.6914058"},{"issue":"4","key":"10229_CR105","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1007\/s11257-016-9178-6","volume":"26","author":"MC Willemsen","year":"2016","unstructured":"Willemsen MC, Graus MP, Knijnenburg BP (2016) Understanding the role of latent feature diversification on choice difficulty and satisfaction. User Model User Adapt Interact 26(4):347\u2013389","journal-title":"User Model User Adapt Interact"},{"key":"10229_CR106","doi-asserted-by":"crossref","unstructured":"Wu G, Luo K, Sanner S, Soh H (2019) Deep language-based critiquing for recommender systems. In: Proceedings of the 13th ACM conference on recommender systems, RecSys\u201919. pp 137\u2013145","DOI":"10.1145\/3298689.3347009"},{"issue":"3","key":"10229_CR107","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1080\/07421222.2017.1373011","volume":"34","author":"DJ Xu","year":"2017","unstructured":"Xu DJ, Benbasat I, Cenfetelli RT (2017) A two-stage model of generating product advice: proposing and testing the complementarity principle. J Manag Inf Syst 34(3):826\u2013862","journal-title":"J Manag Inf Syst"},{"key":"10229_CR108","doi-asserted-by":"crossref","unstructured":"Yan Z, Duan N, Chen P, Zhou M, Zhou J, Li Z (2017) Building task-oriented dialogue systems for online shopping. In: AAAI\u201917. pp 4618\u20134626","DOI":"10.1609\/aaai.v31i1.11182"},{"key":"10229_CR109","doi-asserted-by":"crossref","unstructured":"Yang L, Sobolev M, Tsangouri C, Estrin D (2018) Understanding user interactions with podcast recommendations delivered via voice. In: RecSys\u201918. pp 190\u2013194","DOI":"10.1145\/3240323.3240389"},{"key":"10229_CR110","doi-asserted-by":"crossref","unstructured":"Yu T, Shen Y, Zhang R, Zeng X, Jin H (2019a) Vision-language recommendation via attribute augmented multimodal reinforcement learning. In: MM\u201919. pp 39\u201347","DOI":"10.1145\/3343031.3350935"},{"key":"10229_CR111","doi-asserted-by":"crossref","unstructured":"Yu T, Shen Y, Zhang R, Zeng X, Jin H (2019b) Vision-language recommendation via attribute augmented multimodal reinforcement learning. In: Proceedings of the 27th ACM international conference on multimedia, MM\u201919. pp 39\u201347","DOI":"10.1145\/3343031.3350935"},{"key":"10229_CR112","doi-asserted-by":"crossref","unstructured":"Zanker M, Bricman M, Gordea S, Jannach D, Jessenitschnig M (2006) Persuasive online-selling in quality and taste domains. In: 7th international conference on electronic commerce and web technologies (EC-Web 2006), Krakow. pp 51\u201360","DOI":"10.1007\/11823865_6"},{"key":"10229_CR113","doi-asserted-by":"crossref","unstructured":"Zeng J, Nakano YI, Morita T, Kobayashi I, Yamaguchi T (2018) Eliciting user food preferences in terms of taste and texture in spoken dialogue systems. In: MHFI\u201918. pp 1\u20135","DOI":"10.1145\/3279954.3279959"},{"key":"10229_CR115","doi-asserted-by":"crossref","unstructured":"Zhang S, Balog K (2020) Evaluating conversational recommender systems via user simulation. In: Gupta R, Liu Y, Tang J, Prakash BA (eds) Proceedings 26th ACM SIGKDD conference on knowledge discovery and data mining, KDD\u201920. pp 1512\u20131520","DOI":"10.1145\/3394486.3403202"},{"key":"10229_CR114","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1287\/isre.2019.0876","volume":"31","author":"J Zhang","year":"2019","unstructured":"Zhang J, Adomavicius G, Gupta A, Ketter W (2019) Consumption and performance: understanding longitudinal dynamics of recommender systems via an agent-based simulation framework. Inf Syst Res 31:76\u2013101","journal-title":"Inf Syst Res"},{"key":"10229_CR116","doi-asserted-by":"crossref","unstructured":"Zhang T, Liu Y, Zhong P, Zhang C, Wang H, Miao C (2021) Kecrs: towards knowledge-enriched conversational recommendation system. http:\/\/arxiv.org\/abs\/2105.08261","DOI":"10.18653\/v1\/2022.nlp4convai-1.17"},{"issue":"4","key":"10229_CR117","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3337967","volume":"10","author":"G Zhao","year":"2019","unstructured":"Zhao G, Fu H, Song R, Sakai T, Chen Z, Xie X, Qian X (2019) Personalized reason generation for explainable song recommendation. ACM Trans Intell Syst Technol 10(4):1\u201321","journal-title":"ACM Trans Intell Syst Technol"},{"key":"10229_CR118","doi-asserted-by":"crossref","unstructured":"Zhou K, Zhao WX, Bian S, Zhou Y, Wen J, Yu J (2020a) Improving conversational recommender systems via knowledge graph based semantic fusion. In: Proceedings ACM SIGKDD conference on knowledge discovery and data mining. pp 1006\u20131014","DOI":"10.1145\/3394486.3403143"},{"key":"10229_CR119","doi-asserted-by":"crossref","unstructured":"Zhou K, Zhao WX, Bian S, Zhou Y, Wen JR, Yu J (2020b) Improving conversational recommender systems via knowledge graph based semantic fusion. In: KDD\u201920. pp 1006\u20131014","DOI":"10.1145\/3394486.3403143"},{"key":"10229_CR120","doi-asserted-by":"crossref","unstructured":"Zhou K, Zhou Y, Zhao WX, Wang X, Wen JR (2020c) Towards topic-guided conversational recommender system. In: Proceedings of the 28th international conference on computational linguistics. pp 4128\u20134139","DOI":"10.18653\/v1\/2020.coling-main.365"},{"key":"10229_CR121","doi-asserted-by":"crossref","unstructured":"Zhou K, Wang X, Zhou Y, Shang C, Cheng Y, Zhao WX, Li Y, Wen JR (2021) CRSLab: an open-source toolkit for building conversational recommender system. https:\/\/arxiv.org\/abs\/2101.00939","DOI":"10.18653\/v1\/2021.acl-demo.22"}],"container-title":["Artificial Intelligence Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-022-10229-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10462-022-10229-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10462-022-10229-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,28]],"date-time":"2024-09-28T23:05:25Z","timestamp":1727564725000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10462-022-10229-x"}},"subtitle":["A landscape of research"],"short-title":[],"issued":{"date-parts":[[2022,7,12]]},"references-count":121,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,3]]}},"alternative-id":["10229"],"URL":"https:\/\/doi.org\/10.1007\/s10462-022-10229-x","relation":{},"ISSN":["0269-2821","1573-7462"],"issn-type":[{"value":"0269-2821","type":"print"},{"value":"1573-7462","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,12]]},"assertion":[{"value":"12 July 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}