{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:10:40Z","timestamp":1784178640418,"version":"3.55.0"},"reference-count":17,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,6,1]],"date-time":"2024-06-01T00:00:00Z","timestamp":1717200000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p>Conversational Agents are increasingly integrated into our daily routines, assisting us with various tasks, from simple commands such as scheduling events to more complex conversational search interactions. Such conversational search systems are traditionally evaluated with word-overlap metrics such as F1 score and accuracy. The full-day workshop on Search-Oriented Conversational Artificial Intelligence (SCAI) at CHIIR 2024 explored the evaluation of conversational search systems from the user's perspective. This interactive workshop included multiple panel discussions and working groups focused on developing and discussing innovative, user-centered evaluation methods for these systems. This paper, co-authored by both organizers and participants of the workshop, presents a summary of the insights gathered from the panel discussions and working groups.<\/jats:p>\n          <jats:p>\n            <jats:bold>Date<\/jats:bold>\n            : 14 March 2024.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Website<\/jats:bold>\n            : https:\/\/scai.info\/scai-2024\/.\n          <\/jats:p>","DOI":"10.1145\/3687273.3687282","type":"journal-article","created":{"date-parts":[[2024,8,7]],"date-time":"2024-08-07T22:24:53Z","timestamp":1723069493000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024"],"prefix":"10.1145","volume":"58","author":[{"given":"Alexander","family":"Frummet","sequence":"first","affiliation":[{"name":"Universit\u00e4t Regensburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Papenmeier","sequence":"additional","affiliation":[{"name":"University of Twente, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maik","family":"Fr\u00f6be","sequence":"additional","affiliation":[{"name":"Friedrich-Schiller-Universit\u00e4t Jena, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johannes","family":"Kiesel","sequence":"additional","affiliation":[{"name":"Bauhaus-Universit\u00e4t Weimar, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vaibhav","family":"Adlakha","sequence":"additional","affiliation":[{"name":"Mila-Quebec AI Institute, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Norbert","family":"Braunschweiler","sequence":"additional","affiliation":[{"name":"Toshiba Europe Limited, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mateusz","family":"Dubiel","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Satanu","family":"Ghosh","sequence":"additional","affiliation":[{"name":"University of New Hampshire, US"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marcel","family":"Gohsen","sequence":"additional","affiliation":[{"name":"Bauhaus-Universit\u00e4t Weimar, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christin","family":"Kreutz","sequence":"additional","affiliation":[{"name":"TH Mittelhessen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Milad","family":"Momeni","sequence":"additional","affiliation":[{"name":"University of Regina, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Markus","family":"Nilles","sequence":"additional","affiliation":[{"name":"Trier University, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sachin Pathiyan","family":"Cherumanal","sequence":"additional","affiliation":[{"name":"RMIT University, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abbas","family":"Pirmoradi","sequence":"additional","affiliation":[{"name":"University of Regina, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Thomas","sequence":"additional","affiliation":[{"name":"Microsoft, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johanne R.","family":"Trippas","sequence":"additional","affiliation":[{"name":"RMIT University, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ines","family":"Zelch","sequence":"additional","affiliation":[{"name":"Leipzig University, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oleg","family":"Zendel","sequence":"additional","affiliation":[{"name":"RMIT University, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,8,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01074-z"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.21437\/ICSLP.2002-559"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3624918.3629549"},{"key":"e_1_2_1_4_1","first-page":"65","volume-title":"Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization@ACL 2005","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare R. Voss, editors, Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65--72. Association for Computational Linguistics, 2005. URL https:\/\/aclanthology.org\/W05-0909\/."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121111"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 2nd International Workshop on Search-Oriented Conversational AI, SCAI@EMNLP 2018","author":"Chuklin Aleksandr","year":"2018","unstructured":"Aleksandr Chuklin, Jeff Dalton, Julia Kiseleva, Alexey Borisov, and Mikhail Burtsev, editors. Proceedings of the 2nd International Workshop on Search-Oriented Conversational AI, SCAI@EMNLP 2018, Brussels, Belgium, October 31, 2018, 2018. Association for Computational Linguistics. ISBN 978-1-948087-75-9. URL https:\/\/aclanthology.org\/volumes\/W18-57\/."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 5th International Workshop on Search-Oriented Conversational AI (SCAI)","author":"Dalton Jeff","year":"2020","unstructured":"Jeff Dalton, Aleksandr Chuklin, Julia Kiseleva, and Mikhail Burtsev, editors. Proceedings of the 5th International Workshop on Search-Oriented Conversational AI (SCAI), Online, November 2020. Association for Computational Linguistics. URL https:\/\/aclanthology.org\/2020.scai-1.0."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3627508.3638310"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591981"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657793"},{"key":"e_1_2_1_11_1","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:\/\/aclanthology.org\/W04-1013."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462894"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531700"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657914"},{"key":"e_1_2_1_16_1","first-page":"4913","volume-title":"Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC","author":"Vakulenko Svitlana","year":"2022","unstructured":"Svitlana Vakulenko, Johannes Kiesel, and Maik Fr\u00f6be. Scai-qrecc shared task on conversational question answering. In Nicoletta Calzolari, Fr\u00e9d\u00e9ric B\u00e9chet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, H\u00e9l\u00e8ne Mazo, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 4913--4922. European Language Resources Association, 2022. URL https:\/\/aclanthology.org\/2022.lrec-1.525."},{"key":"e_1_2_1_17_1","volume-title":"8th International Conference on Learning Representations, ICLR 2020","author":"Zhang Tianyi","year":"2020","unstructured":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:\/\/openreview.net\/forum?id=SkeHuCVFDr."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687273.3687282","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3687273.3687282","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:10:01Z","timestamp":1750295401000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687273.3687282"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6]]},"references-count":17,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1145\/3687273.3687282"],"URL":"https:\/\/doi.org\/10.1145\/3687273.3687282","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2024,6]]},"assertion":[{"value":"2024-08-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}