{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:10:45Z","timestamp":1784178645156,"version":"3.55.0"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:00:00Z","timestamp":1715904000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>Clarifying underlying user information needs by asking clarifying questions is an important feature of modern conversational search systems. However, evaluation of such systems through answering prompted clarifying questions requires significant human effort, which can be time-consuming and expensive. In our recent work, we proposed an approach to tackle these issues with a user simulator,<jats:italic>USi<\/jats:italic>. Given a description of an information need,<jats:italic>USi<\/jats:italic>is capable of automatically answering clarifying questions about the topic throughout the search session. However, while the answers generated by<jats:italic>USi<\/jats:italic>are both in line with the underlying information need and in natural language, a deeper understanding of such utterances is lacking. Thus, in this work, we explore utterance formulation of large language model (LLM)\u2013based user simulators. To this end, we first analyze the differences between<jats:italic>USi<\/jats:italic>, based on GPT-2, and the next generation of generative LLMs, such as GPT-3. Then, to gain a deeper understanding of LLM-based utterance generation, we compare the generated answers to the recently proposed set of patterns of human-based query reformulations. Finally, we discuss potential applications as well as limitations of LLM-based user simulators and outline promising directions for future work on the topic.<\/jats:p>","DOI":"10.1145\/3650041","type":"journal-article","created":{"date-parts":[[2024,3,5]],"date-time":"2024-03-05T12:10:21Z","timestamp":1709640621000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Analysing Utterances in LLM-Based User Simulation for Conversational Search"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-0036-0490","authenticated-orcid":false,"given":"Ivan","family":"Sekuli\u0107","sequence":"first","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9447-4172","authenticated-orcid":false,"given":"Mohammad","family":"Alinannejadi","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8672-0700","authenticated-orcid":false,"given":"Fabio","family":"Crestani","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,5,17]]},"reference":[{"key":"e_1_3_2_2_2","volume-title":"WSDM","author":"Abbasiantaeb Zahra","year":"2024","unstructured":"Zahra Abbasiantaeb, Yifei Yuan, Evangelos Kanoulas, and Mohammad Aliannejadi. 2024. Let the LLMs talk: Simulating human-to-human conversational QA via zero-shot LLM-to-LLM interactions. In WSDM. ACM."},{"key":"e_1_3_2_3_2","unstructured":"Mohammad Aliannejadi Zahra Abbasiantaeb Shubham Chatterjee Jeffery Dalton and Leif Azzopardi. 2024. TREC iKAT 2023: The interactive knowledge assistance track overview. arXiv preprint arXiv:2401.01330 (2024)."},{"key":"e_1_3_2_4_2","unstructured":"Mohammad Aliannejadi Julia Kiseleva Aleksandr Chuklin Jeff Dalton and Mikhail Burtsev. 2020. ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ). (2020)."},{"key":"e_1_3_2_5_2","first-page":"475","volume-title":"SIGIR","author":"Aliannejadi Mohammad","year":"2019","unstructured":"Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In SIGIR. 475\u2013484."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3451964.3451967"},{"key":"e_1_3_2_7_2","first-page":"15","volume-title":"SIGIR","author":"Azzopardi Leif","year":"2011","unstructured":"Leif Azzopardi. 2011. The economics in interactive information retrieval. In SIGIR. ACM, 15\u201324."},{"key":"e_1_3_2_8_2","unstructured":"Krisztian Balog. 2021. Conversational AI from an information retrieval perspective: Remaining challenges and a case for user simulation. (2021)."},{"key":"e_1_3_2_9_2","volume-title":"ACL","author":"Belz Anja","year":"2006","unstructured":"Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In ACL."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"Guorui Bian Michael McAleer and Wing-Keung Wong. 2011. A trinomial test for paired data when there are many ties. Mathematics and Computers in Simulation 81 6 (2011) 1153\u20131160.","DOI":"10.1016\/j.matcom.2010.11.002"},{"key":"e_1_3_2_11_2","first-page":"2","volume-title":"SIGDIAL","author":"Black Alan W.","year":"2011","unstructured":"Alan W. Black, Susanne Burger, Alistair Conkie, Helen Hastie, Simon Keizer, Oliver Lemon, Nicolas Merigaud, Gabriel Parent, Gabriel Schubiner, Blaise Thomson, et\u00a0al. 2011. Spoken dialog challenge 2010: Comparison of live and control test results. In SIGDIAL. 2\u20137."},{"key":"e_1_3_2_12_2","volume-title":"CHIIR","author":"Braslavski Pavel","year":"2017","unstructured":"Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly?: Analyzing clarification questions in CQA. In CHIIR."},{"key":"e_1_3_2_13_2","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D. Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et\u00a0al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020) 1877\u20131901."},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","unstructured":"Pawe\u0142 Budzianowski and Ivan Vulic. 2019. Hello it\u2019s GPT-2 \u2014 how can I help you? Towards the use of pretrained language models for task-oriented dialogue systems. EMNLP-IJCNLP 2019 (2019) 15.","DOI":"10.18653\/v1\/D19-5602"},{"key":"e_1_3_2_15_2","volume-title":"ACL","author":"Callison-Burch Chris","year":"2006","unstructured":"Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In ACL."},{"key":"e_1_3_2_16_2","first-page":"611","volume-title":"CIKM","author":"Carterette Ben","year":"2011","unstructured":"Ben Carterette, Evangelos Kanoulas, and Emine Yilmaz. 2011. Simulating simple user behavior for system effectiveness evaluation. In CIKM. 611\u2013620."},{"key":"e_1_3_2_17_2","first-page":"164","volume-title":"CLEF","author":"Chuklin Aleksandr","year":"2019","unstructured":"Aleksandr Chuklin, Aliaksei Severyn, Johanne R. Trippas, Enrique Alfonseca, Hanna Silen, and Damiano Spina. 2019. Using audio transformations to improve comprehension in voice question answering. In CLEF. Springer, 164\u2013170."},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Michael D. Cooper. 1973. A simulation model of an information retrieval system. Information Storage and Retrieval 9 1 (1973) 13\u201332.","DOI":"10.1016\/0020-0271(73)90004-1"},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Fabio Crestani and Heather Du. 2006. Written versus spoken queries: A qualitative and quantitative comparative analysis. Journal of the American Society for Information Science and Technology 57 7 (2006) 881\u2013890.","DOI":"10.1002\/asi.20350"},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","unstructured":"Jeffrey Dalton Chenyan Xiong and Jamie Callan. 2020. TREC CAsT 2019: The conversational assistance track overview. arXiv preprint arXiv:2003.13624 (2020).","DOI":"10.6028\/NIST.SP.1266.cast-overview"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Jan Deriu Alvaro Rodrigo Arantxa Otegi Guillermo Echegoyen Sophie Rosset Eneko Agirre and Mark Cieliebak. 2021. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review 54 1 (2021) 755\u2013810.","DOI":"10.1007\/s10462-020-09866-x"},{"key":"e_1_3_2_22_2","first-page":"323","volume-title":"SIGIR","author":"Diaz Fernando","year":"2009","unstructured":"Fernando Diaz and Jaime Arguello. 2009. Adaptation of offline vertical selection predictions in the presence of user feedback. In SIGIR. 323\u2013330."},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"Nouha Dziri Andrea Madotto Osmar Zaiane and Avishek Joey Bose. 2021. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. arXiv preprint arXiv:2104.08455 (2021).","DOI":"10.18653\/v1\/2021.emnlp-main.168"},{"key":"e_1_3_2_24_2","unstructured":"Angela Fan Mike Lewis and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 (2018)."},{"key":"e_1_3_2_25_2","unstructured":"Tianyu Gao Adam Fisch and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 (2020)."},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"Michael D. Gordon. 1990. Evaluating the effectiveness of information retrieval systems using simulated queries. Journal of the American Society for Information Science 41 5 (1990) 313\u2013323.","DOI":"10.1002\/(SICI)1097-4571(199007)41:5<313::AID-ASI1>3.0.CO;2-G"},{"key":"e_1_3_2_27_2","volume-title":"The Computer Simulation of Information Retrieval Systems","author":"Griffiths Jos\u00e9-Marie","year":"1976","unstructured":"Jos\u00e9-Marie Griffiths. 1976. The Computer Simulation of Information Retrieval Systems. Ph. D. Dissertation. University of London (University College)."},{"key":"e_1_3_2_28_2","first-page":"1131","volume-title":"SIGIR","author":"Hashemi Helia","year":"2020","unstructured":"Helia Hashemi, Hamed Zamani, and W. Bruce Croft. 2020. Guided Transformer: Leveraging multiple external sources for representation learning in conversational search. In SIGIR. 1131\u20131140."},{"key":"e_1_3_2_29_2","unstructured":"Jordan Hoffmann Sebastian Borgeaud Arthur Mensch Elena Buchatskaya Trevor Cai Eliza Rutherford Diego de Las Casas Lisa Anne Hendricks Johannes Welbl Aidan Clark et\u00a0al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022)."},{"key":"e_1_3_2_30_2","unstructured":"Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019)."},{"key":"e_1_3_2_31_2","first-page":"159","volume-title":"CHI","author":"Horvitz Eric","year":"1999","unstructured":"Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In CHI. 159\u2013166."},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Gaya K. Jayasinghe William Webber Mark Sanderson Lasitha S. Dharmasena and J. Shane Culpepper. 2015. Statistical comparisons of non-deterministic IR systems using two dimensional variance. Information Processing & Management 51 5 (2015) 677\u2013694.","DOI":"10.1016\/j.ipm.2015.06.005"},{"key":"e_1_3_2_33_2","first-page":"1257","volume-title":"SIGIR","author":"Kiesel Johannes","year":"2018","unstructured":"Johannes Kiesel, Arefeh Bahrami, Benno Stein, Avishek Anand, and Matthias Hagen. 2018. Toward voice query clarification. In SIGIR. 1257\u20131260."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531814"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3409256.3409817"},{"key":"e_1_3_2_36_2","first-page":"540","volume-title":"EMNLP-IJCNLP","author":"Kryscinski Wojciech","year":"2019","unstructured":"Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In EMNLP-IJCNLP. 540\u2013551."},{"key":"e_1_3_2_37_2","unstructured":"Margaret Li Jason Weston and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087 (2019)."},{"key":"e_1_3_2_38_2","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 74\u201381."},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","unstructured":"Aldo Lipani Ben Carterette and Emine Yilmaz. 2021. How am I doing?: Evaluating conversational search systems offline. ACM TOIS (2021).","DOI":"10.1145\/3451160"},{"key":"e_1_3_2_40_2","unstructured":"Tom Lotze Stefan Klut Mohammad Aliannejadi and Evangelos Kanoulas. 2021. Ranking clarifying questions based on predicted user engagement. CoRR abs\/2103.06192 (2021)."},{"key":"e_1_3_2_41_2","first-page":"6","volume-title":"Proceedings of the 4th Conference on Conversational User Interfaces (CUI\u201922)","author":"Meyer Selina","year":"2022","unstructured":"Selina Meyer, David Elsweiler, Bernd Ludwig, Marcos Fernandez-Pichel, and David E. Losada. 2022. Do we still need human assessors? Prompt-based GPT-3 user simulation in conversational AI. In Proceedings of the 4th Conference on Conversational User Interfaces (CUI\u201922). 6 pages."},{"key":"e_1_3_2_42_2","volume-title":"Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing","author":"Moosavi Nafise Sadat","year":"2020","unstructured":"Nafise Sadat Moosavi, Angela Fan, Vered Shwartz, Goran Glava\u0161, Shafiq Joty, Alex Wang, and Thomas Wolf (Eds.). 2020. Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing. Association for Computational Linguistics."},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","unstructured":"Javed Mostafa Snehasis Mukhopadhyay and Mathew Palakal. 2003. Simulation studies of different dimensions of users\u2019 interests and their impact on user modeling and information filtering. Information Retrieval 6 2 (2003) 199\u2013223.","DOI":"10.1023\/A:1023932221048"},{"key":"e_1_3_2_44_2","first-page":"2241","volume-title":"EMNLP","author":"Novikova Jekaterina","year":"2017","unstructured":"Jekaterina Novikova, Ond\u0159ej Du\u0161ek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In EMNLP. 2241\u20132252."},{"key":"e_1_3_2_45_2","volume-title":"SIGIR","author":"Owoicho Paul","year":"2023","unstructured":"Paul Owoicho, Ivan Sekuli\u0107, Mohammad Aliannejadi, Jeff Dalton, and Fabio Crestani. 2023. Exploiting simulated user feedback for conversational search: Ranking, rewriting, and beyond. In SIGIR."},{"key":"e_1_3_2_46_2","first-page":"311","volume-title":"ACL","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In ACL. 311\u2013318."},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","unstructured":"Baolin Peng Chenguang Zhu Chunyuan Li Xiujun Li Jinchao Li Michael Zeng and Jianfeng Gao. 2020. Few-shot natural language generation for task-oriented dialog. arXiv preprint arXiv:2002.12328 (2020).","DOI":"10.18653\/v1\/2020.findings-emnlp.17"},{"key":"e_1_3_2_48_2","unstructured":"Gustavo Penha and Claudia Hauff. 2020. Challenges in the evaluation of conversational search systems. KDD Workshop on Conversational Systems Towards Mainstream Adoption (2020)."},{"key":"e_1_3_2_49_2","first-page":"25","volume-title":"CHIIR","author":"Qu Chen","year":"2019","unstructured":"Chen Qu, Liu Yang, W. Bruce Croft, Yongfeng Zhang, Johanne R. Trippas, and Minghui Qiu. 2019. User intent prediction in information-seeking conversations. In CHIIR. 25\u201333."},{"key":"e_1_3_2_50_2","unstructured":"Alec Radford Jeff Wu Rewon Child David Luan Dario Amodei and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. (2019)."},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1145\/3020165.3020183","volume-title":"CHIIR","author":"Radlinski Filip","year":"2017","unstructured":"Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In CHIIR. 117\u2013126."},{"key":"e_1_3_2_52_2","first-page":"2736","volume-title":"ACL (1)","author":"Rao Sudha","year":"2018","unstructured":"Sudha Rao and Hal Daum\u00e9. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In ACL (1). 2736\u20132745."},{"key":"e_1_3_2_53_2","unstructured":"Sudha Rao and Hal Daum\u00e9 III. 2019. Answer-based adversarial training for generating clarification questions. arXiv:1904.02281 (2019)."},{"key":"e_1_3_2_54_2","first-page":"1715","volume-title":"WWW","author":"Ren Gary","year":"2018","unstructured":"Gary Ren, Xiaochuan Ni, Manish Malik, and Qifa Ke. 2018. Conversational query understanding using sequence to sequence modeling. In WWW. 1715\u20131724."},{"key":"e_1_3_2_55_2","unstructured":"Pengjie Ren Zhumin Chen Zhaochun Ren Evangelos Kanoulas Christof Monz and Maarten de Rijke. 2020. Conversations with search engines. ACM Transactions on Information Systems 1 1 (2020)."},{"key":"e_1_3_2_56_2","first-page":"1160","volume-title":"The Web Conference","author":"Rosset Corbin","year":"2020","unstructured":"Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. 2020. Leading conversational search by suggesting useful questions. In The Web Conference. 1160\u20131170."},{"key":"e_1_3_2_57_2","first-page":"587","volume-title":"ECIR","author":"Salle Alexandre","year":"2021","unstructured":"Alexandre Salle, Shervin Malmasi, Oleg Rokhlenko, and Eugene Agichtein. 2021. Studying the effectiveness of conversational search refinement through user simulation. In ECIR. 587\u2013602."},{"key":"e_1_3_2_58_2","first-page":"1702","volume-title":"NAACL","author":"See Abigail","year":"2019","unstructured":"Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? How controllable attributes affect human judgments. In NAACL. 1702\u20131723."},{"key":"e_1_3_2_59_2","volume-title":"Proceedings of the 29th Text REtrieval Conference, TREC","author":"Sekuli\u0107 Ivan","year":"2020","unstructured":"Ivan Sekuli\u0107, Mohammad Aliannejadi, and Fabio Crestani. 2020. Extending the use of previous relevant utterances for response ranking in conversational search. In Proceedings of the 29th Text REtrieval Conference, TREC."},{"key":"e_1_3_2_60_2","volume-title":"ICTIR","author":"Sekuli\u0107 Ivan","year":"2021","unstructured":"Ivan Sekuli\u0107, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards facet-driven generation of clarifying questions for conversational search. In ICTIR."},{"key":"e_1_3_2_61_2","first-page":"619","volume-title":"ECIR (1)","author":"Sekuli\u0107 Ivan","year":"2021","unstructured":"Ivan Sekuli\u0107, Mohammad Aliannejadi, and Fabio Crestani. 2021. User engagement prediction for clarification in search. In ECIR (1). 619\u2013633."},{"key":"e_1_3_2_62_2","volume-title":"WSDM\u201922: International Conference on Web Search and Data Mining","author":"Sekuli\u0107 Ivan","year":"2022","unstructured":"Ivan Sekuli\u0107, Mohammad Aliannejadi, and Fabio Crestani. 2022. Evaluating mixed-initiative conversational search systems via user simulation. In WSDM\u201922: International Conference on Web Search and Data Mining (Phoenix, AZ)."},{"key":"e_1_3_2_63_2","volume-title":"ECIR","author":"Sekuli\u0107 Ivan","year":"2022","unstructured":"Ivan Sekuli\u0107, Mohammad Aliannejadi, and Fabio Crestani. 2022. Exploiting document-based features for clarification in conversational search. In ECIR."},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","unstructured":"Ivan Sekuli\u0107 Amir Soleimani Mohammad Aliannejadi and Fabio Crestani. 2020. Longformer for MS MARCO document re-ranking task. arXiv preprint arXiv:2009.09392 (2020).","DOI":"10.6028\/NIST.SP.1266.deep-USI"},{"key":"e_1_3_2_65_2","first-page":"2210","volume-title":"EMNLP","author":"Shao Yuanlong","year":"2017","unstructured":"Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In EMNLP. 2210\u20132219."},{"key":"e_1_3_2_66_2","first-page":"341","volume-title":"CICLing","author":"Stent Amanda","year":"2005","unstructured":"Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In CICLing. 341\u2013351."},{"key":"e_1_3_2_67_2","doi-asserted-by":"crossref","unstructured":"Weiwei Sun Shuo Zhang Krisztian Balog Zhaochun Ren Pengjie Ren Zhumin Chen and Maarten de Rijke. 2021. Simulating user satisfaction for the evaluation of task-oriented dialogue systems. arXiv preprint arXiv:2105.03748 (2021).","DOI":"10.1145\/3404835.3463241"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210002"},{"key":"e_1_3_2_69_2","unstructured":"Ilya Sutskever Oriol Vinyals and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. NeurIPS 27 (2014) 3104\u20133112."},{"key":"e_1_3_2_70_2","first-page":"236","volume-title":"SIGIR","author":"Tague Jean","year":"1980","unstructured":"Jean Tague, Michael Nelson, and Harry Wu. 1980. Problems in the simulation of bibliographic retrieval systems. In SIGIR. 236\u2013255."},{"key":"e_1_3_2_71_2","first-page":"231","volume-title":"ACL (2)","author":"Tian Zhiliang","year":"2017","unstructured":"Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. 2017. How to make context more useful? An empirical study on context-aware neural conversational models. In ACL (2). 231\u2013236."},{"key":"e_1_3_2_72_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et\u00a0al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_3_2_73_2","volume-title":"ECIR","author":"Vakulenko Svitlana","year":"2021","unstructured":"Svitlana Vakulenko, Nikos Voskarides, Zhucheng Tu, and Shayne Longpre. 2021. A comparison of question rewriting methods for conversational passage retrieval. In ECIR."},{"key":"e_1_3_2_74_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)."},{"key":"e_1_3_2_75_2","first-page":"2193","volume-title":"ACL (1)","author":"Wang Yansen","year":"2018","unstructured":"Yansen Wang, Chenyi Liu, Minlie Huang, and Liqiang Nie. 2018. Learning to ask questions in open-domain conversational systems with typed decoders. In ACL (1). 2193\u20132203."},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911542"},{"key":"e_1_3_2_77_2","volume-title":"TREC","author":"Yang Grace Hui","year":"2016","unstructured":"Grace Hui Yang and Ian Soboroff. 2016. TREC 2016 dynamic domain track overview. In TREC."},{"key":"e_1_3_2_78_2","first-page":"418","volume-title":"The Web Conference","author":"Zamani Hamed","year":"2020","unstructured":"Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020. Generating clarifying questions for information retrieval. In The Web Conference. 418\u2013428."},{"key":"e_1_3_2_79_2","first-page":"1512","volume-title":"KDD","author":"Zhang Shuo","year":"2020","unstructured":"Shuo Zhang and Krisztian Balog. 2020. Evaluating conversational recommender systems via user simulation. In KDD. 1512\u20131520."},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531936"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650041","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3650041","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:43Z","timestamp":1750291423000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650041"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,17]]},"references-count":79,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3650041"],"URL":"https:\/\/doi.org\/10.1145\/3650041","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,17]]},"assertion":[{"value":"2023-05-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}