{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T14:56:42Z","timestamp":1781189802420,"version":"3.54.1"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T00:00:00Z","timestamp":1781136000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100003246","name":"Dutch Research Council","doi-asserted-by":"crossref","award":["024.004.022, NWA.1389.20.183, and KICH3.LTP.20.006"],"award-info":[{"award-number":["024.004.022, NWA.1389.20.183, and KICH3.LTP.20.006"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"crossref"}]},{"name":"European Union\u2019s Horizon Europe program","award":["101070212 (FINDHR) and No. 101201510 (UNITE)"],"award-info":[{"award-number":["101070212 (FINDHR) and No. 101201510 (UNITE)"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>\n                    Clarifying questions enable\n                    <jats:bold>Conversational Search (CS)<\/jats:bold>\n                    systems to resolve underspecified queries by eliciting missing information from users. However, how prompting strategies shape the quality of clarifying questions and how such questions should be evaluated at scale remains understudied. We present\n                    <jats:bold>Automatic GENeration and evaluaTion of Clarifying Questions (AGENT-CQ)<\/jats:bold>\n                    , a framework for systematically generating and evaluating clarifying questions and simulated user responses using\n                    <jats:bold>Large Language Models (LLMs)<\/jats:bold>\n                    . To support scalable and multi-perspective evaluation, we introduce\n                    <jats:italic toggle=\"yes\">CrowdLLM<\/jats:italic>\n                    , an LLM-based evaluation paradigm that simulates diverse annotator judgments through distinct evaluator personas. Our experiments span both open-domain CS and a regulatory question-answering setting, allowing us to examine the extent to which clarification strategies generalize across domains with different interaction constraints. Across settings, temperature-variation prompting leads to higher quality clarifying questions than baseline prompting and human-authored questions on several dimensions of the task. In addition, LLM-generated clarifying questions lead to improved downstream retrieval performance than human-authored questions in open-domain search. Together, AGENT-CQ and CrowdLLM provide a practical framework for studying and improving clarification strategies in conversational IR systems.\n                  <\/jats:p>","DOI":"10.1145\/3809182","type":"journal-article","created":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T14:52:08Z","timestamp":1776264728000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with Large Language Models"],"prefix":"10.1145","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5301-4244","authenticated-orcid":false,"given":"Clemencia","family":"Siro","sequence":"first","affiliation":[{"name":"University of Amsterdam, Amsterdam, The Netherlands and Centrum Wiskunde en Informatica, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7275-5398","authenticated-orcid":false,"given":"Yifei","family":"Yuan","sequence":"additional","affiliation":[{"name":"ETH Z\u00fcrich, Z\u00fcrich, Switzerland and University of Copenhagen, Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9447-4172","authenticated-orcid":false,"given":"Mohammad","family":"Aliannejadi","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1086-0202","authenticated-orcid":false,"given":"Maarten","family":"de Rijke","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,11]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1145\/3616855.3635856","volume-title":"Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM \u201924)and","author":"Abbasiantaeb Zahra","year":"2024","unstructured":"Zahra Abbasiantaeb, Yifei Yuan, Evangelos Kanoulas, and Mohammad Aliannejadi. 2024. Let the LLMs talk: Simulating human-to-human conversational QA via zero-shot LLM-to-LLM interactions. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM \u201924). Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andr\u00e9s Mu\u00f1oz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii (Eds.), ACM, New York, NY, 8\u201317. DOI: 10.1145\/3616855.3635856"},{"key":"e_1_3_1_3_2","unstructured":"Herv\u00e9 Abdi and Lynne J. Williams. 2010. Tukey\u2019s Honestly Significant Difference (HSD) Test. Technical Report. University of Texas at Dallas."},{"key":"e_1_3_1_4_2","unstructured":"Mohammad Aliannejadi Julia Kiseleva Aleksandr Chuklin Jeff Dalton and Mikhail Burtsev. 2020. ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ). arXiv:2009.11352. Retrieved from https:\/\/arxiv.org\/abs\/2009.11352"},{"key":"e_1_3_1_5_2","doi-asserted-by":"crossref","first-page":"4473","DOI":"10.18653\/v1\/2021.emnlp-main.367","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201921)","author":"Aliannejadi Mohammad","year":"2021","unstructured":"Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue corpora with clarifying questions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201921). Association for Computational Linguistics, 4473\u20134484."},{"key":"e_1_3_1_6_2","first-page":"475","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201919)and","author":"Aliannejadi Mohammad","year":"2019","unstructured":"Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201919). Benjamin Piwowarski, Max Chevalier, \u00c9ric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.), ACM, New York, NY, 475\u2013484. DOI: 10.1145\/3331184.3331265"},{"key":"e_1_3_1_7_2","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1145\/3020165.3022149","volume-title":"Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (CHIIR \u201917)","author":"Braslavski Pavel","year":"2017","unstructured":"Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? Analyzing clarification questions in CQA. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (CHIIR \u201917). Ragnar Nordlie, Nils Pharo, Luanne Freund, Birger Larsen, and Dan Russel (Eds.), ACM, New York, NY, 345\u2013348. DOI: 10.1145\/3020165.3022149"},{"key":"e_1_3_1_8_2","first-page":"191","volume-title":"Proceedings of the 31st ACM International Conference on Information and Knowledge Management","author":"Chen Jiangui","year":"2022","unstructured":"Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yiqun Liu, Yixin Fan, and Xueqi Cheng. 2022. CorpusBrain: Pre-train a generative retrieval model for knowledge-intensive language tasks. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management. Mohammad Al Hasan and Li Xiong (Eds.), ACM, New York, NY, 191\u2013200. DOI: 10.1145\/3511808.3557271"},{"key":"e_1_3_1_9_2","doi-asserted-by":"crossref","unstructured":"Yi Chen Rui Wang Haiyun Jiang Shuming Shi and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. arXiv:2304.00723. Retrieved from https:\/\/arxiv.org\/abs\/2304.00723","DOI":"10.18653\/v1\/2023.findings-ijcnlp.32"},{"key":"e_1_3_1_10_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-031-02294-4","volume-title":"Click Models for Web Search","author":"Chuklin Aleksandr","year":"2015","unstructured":"Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015. Click Models for Web Search. Synthesis Lectures on Information Concepts, Retrieval, and Services, Vol. 7, Morgan & Claypool Publishers."},{"key":"e_1_3_1_11_2","volume-title":"Proceedings of the 18th Text REtrieval Conference (TREC \u201909)","author":"Clarke Charles L. A.","year":"2009","unstructured":"Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the TREC 2009 web track. In Proceedings of the 18th Text REtrieval Conference (TREC \u201909). Ellen M. Voorhees and Lori P. Buckland (Eds.), National Institute of Standards and Technology (NIST). Retrieved from http:\/\/trec.nist.gov\/pubs\/trec18\/papers\/WEB09.OVERVIEW.pdf"},{"issue":"4","key":"e_1_3_1_12_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit","volume":"70","author":"Cohen Jacob","year":"1968","unstructured":"Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70, 4 (1968), 213\u2013220.","journal-title":"Psychological Bulletin"},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","first-page":"10602","DOI":"10.18653\/v1\/2023.findings-emnlp.711","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201923)","author":"Deng Yang","year":"2023","unstructured":"Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. 2023. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. In Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201923). Houda Bouamor, Juan Pino, and Kalika Bali (Eds.), Association for Computational Linguistics, 10602\u201310621. DOI: 10.18653\/v1\/2023.findings-emnlp.711"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/S10462-020-09866-X"},{"key":"e_1_3_1_15_2","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171\u20134186."},{"key":"e_1_3_1_16_2","unstructured":"Kaustubh D. Dhole. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. arXiv:200807559. Retrieved from https:\/\/arxiv.org\/abs\/2008.07559"},{"key":"e_1_3_1_17_2","doi-asserted-by":"crossref","first-page":"7250","DOI":"10.18653\/v1\/2022.acl-long.501","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL \u201922)","author":"Dou Yao","year":"2022","unstructured":"Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2022. Is GPT-3 text indistinguishable from human text? Scarecrow: A framework for scrutinizing machine text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL \u201922). Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.), Association for Computational Linguistics, 7250\u20137274. DOI: 10.18653\/V1\/2022.ACL-LONG.501"},{"issue":"30","key":"e_1_3_1_18_2","doi-asserted-by":"crossref","first-page":"e2305016120","DOI":"10.1073\/pnas.2305016120","article-title":"ChatGPT outperforms crowd workers for text-annotation tasks","volume":"120","author":"Gilardi Fabrizio","year":"2023","unstructured":"Fabrizio Gilardi, Meysam Alizadeh, and Ma\u00ebl Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120, 30 (2023), e2305016120.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.xinn.2025.101253"},{"key":"e_1_3_1_20_2","first-page":"1131","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201920)","author":"Hashemi Helia","year":"2020","unstructured":"Helia Hashemi, Hamed Zamani, and W. Bruce Croft. 2020. Guided transformer: Leveraging multiple external sources for representation learning in conversational search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201920). ACM, New York, NY, 1131\u20131140. DOI: 10.1145\/3397271.3401061"},{"key":"e_1_3_1_21_2","unstructured":"Vitor Jeronymo Luiz Henrique Bonifacio Hugo Queiroz Abonizio Marzieh Fadaee Roberto de Alencar Lotufo Jakub Zavrel and Rodrigo Frassetto Nogueira. 2023. InPars-v2: Large language models as efficient dataset generators for information retrieval. arXiv:2301.01820. Retrieved from https:\/\/arxiv.org\/abs\/2301.01820"},{"issue":"6","key":"e_1_3_1_22_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3534965","article-title":"How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges","volume":"55","author":"Keyvan Kimiya","year":"2022","unstructured":"Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys 55, 6 (2022), 1\u201340.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_1_23_2","first-page":"1257","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR \u201918)","author":"Kiesel Johannes","year":"2018","unstructured":"Johannes Kiesel, Arefeh Bahrami, Benno Stein, Avishek Anand, and Matthias Hagen. 2018. Toward voice query clarification. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR \u201918). Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.), ACM, New York, NY, 1257\u20131260."},{"key":"e_1_3_1_24_2","first-page":"121","volume-title":"Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval (CHIIR \u201916)","author":"Kiseleva Julia","year":"2016","unstructured":"Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C. Crook, Imed Zitouni, and Tasos Anastasakos. 2016. Understanding user satisfaction with intelligent assistants. In Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval (CHIIR \u201916). Diane Kelly, Robert Capra, Nicholas J. Belkin, Jaime Teevan, and Pertti Vakkari (Eds.), ACM, New York, NY, 121\u2013130. DOI: 10.1145\/2854946.2854961"},{"key":"e_1_3_1_25_2","first-page":"193","volume-title":"Proceedings of the 24th Annual Conference of the European Association for Machine Translation","author":"Kocmi Tom","year":"2023","unstructured":"Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation. Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, et al. (Eds.), European Association for Machine Translation, 193\u2013203. Retrieved from https:\/\/aclanthology.org\/2023.eamt-1.19\/"},{"key":"e_1_3_1_26_2","doi-asserted-by":"crossref","first-page":"60","DOI":"10.18653\/v1\/W18-5007","volume-title":"Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue","author":"Kreyssig Florian","year":"2018","unstructured":"Florian Kreyssig, I\u00f1igo Casanueva, Pawe\u0142 Budzianowski, and Milica Ga\u0161i\u0107. 2018. Neural user simulation for corpus-based policy optimisation of spoken dialogue systems. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. Kazunori Komatani, Diane Litman, Kai Yu, Alex Papangelis, Lawrence Cavedon, and Mikio Nakano (Eds.), Association for Computational Linguistics, 60\u201369. DOI: 10.18653\/v1\/W18-5007"},{"key":"e_1_3_1_27_2","unstructured":"Lorenz Kuhn Yarin Gal and Sebastian Farquhar. 2022. Clam: Selective clarification for ambiguous questions with generative language models. arXiv:2212.07769. Retrieved from https:\/\/arxiv.org\/abs\/2212.07769"},{"key":"e_1_3_1_28_2","unstructured":"Xiujun Li Zachary C. Lipton Bhuwan Dhingra Lihong Li Jianfeng Gao and Yun-Nung Chen. 2016. A user simulator for task-completion dialogues. arXiv:1612.05688. Retrieved from https:\/\/arxiv.org\/abs\/1612.05688"},{"key":"e_1_3_1_29_2","unstructured":"Yen-Ting Lin and Yun-Nung Chen. 2023. LLM-Eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv:2305.13711. Retrieved from https:\/\/arxiv.org\/abs\/2305.13711"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3736161"},{"key":"e_1_3_1_31_2","first-page":"2511","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201923)","author":"Liu Yang","year":"2023","unstructured":"Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP \u201923). Houda Bouamor, Juan Pino, and Kalika Bali (Eds.), Association for Computational Linguistics, 2511\u20132522. DOI: 10.18653\/V1\/2023.EMNLP-MAIN.153"},{"key":"e_1_3_1_32_2","first-page":"12688","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (ACL \u201924)","author":"Liu Yiqi","year":"2024","unstructured":"Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2024. LLMs as narcissistic evaluators: When ego inflates evaluation scores. In Proceedings of the Findings of the Association for Computational Linguistics (ACL \u201924). Association for Computational Linguistics, 12688\u201312701."},{"key":"e_1_3_1_33_2","first-page":"1101","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201919)","author":"MacAvaney Sean","year":"2019","unstructured":"Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: Contextualized embeddings for document ranking. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201919). Benjamin Piwowarski, Max Chevalier, \u00c9ric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.), ACM, New York, NY, 1101\u20131104. DOI: 10.1145\/3331184.3331317"},{"key":"e_1_3_1_34_2","unstructured":"Fangwen Mu Lin Shi Song Wang Zhuohao Yu Binquan Zhang Chenxue Wang Shichao Liu and Qing Wang. 2023. ClarifyGPT: Empowering LLM-based code generation with intention clarification. arXiv:2310.10996. Retrieved from https:\/\/arxiv.org\/abs\/2310.10996"},{"key":"e_1_3_1_35_2","unstructured":"Thong Nguyen and Andrew Yates. 2023. Generative retrieval as dense retrieval. arXiv:2306.11397. Retrieved from https:\/\/arxiv.org\/abs\/2306.11397"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1145\/3020165.3020183","volume-title":"Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (CHIIR \u201917)","author":"Radlinski Filip","year":"2017","unstructured":"Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (CHIIR \u201917). Ragnar Nordlie, Nils Pharo, Luanne Freund, Birger Larsen, and Dan Russel (Eds.), ACM, New York, NY, 117\u2013126. DOI: 10.1145\/3020165.3020183"},{"issue":"140","key":"e_1_3_1_37_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1\u201367.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","first-page":"1266","DOI":"10.18653\/v1\/2024.findings-eacl.84","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (EACL \u201924)","author":"Rahmani Hossein A.","year":"2024","unstructured":"Hossein A. Rahmani, Xi Wang, Mohammad Aliannejadi, Mohammadmehdi Naghiaei, and Emine Yilmaz. 2024. Clarifying the path to user satisfaction: An investigation into clarification usefulness. In Proceedings of the Findings of the Association for Computational Linguistics (EACL \u201924). Yvette Graham and Matthew Purver (Eds.), Association for Computational Linguistics, 1266\u20131277. Retrieved from https:\/\/aclanthology.org\/2024.findings-eacl.84\/"},{"key":"e_1_3_1_39_2","unstructured":"Kimia Ramezan Alireza Amiri Bavandpour Yifei Yuan Clemencia Siro and Mohammad Aliannejadi. 2025. Multi-turn multi-modal question clarification for enhanced conversational understanding. arXiv:2502.11442. Retrieved from https:\/\/arxiv.org\/abs\/2502.11442"},{"key":"e_1_3_1_40_2","first-page":"2737","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL \u201918)Long Papers","volume":"1","author":"Rao Sudha","year":"2018","unstructured":"Sudha Rao and Hal Daum\u00e9, III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL \u201918). Iryna Gurevych and Yusuke Miyao (Eds.), Long Papers, Vol. 1, Association for Computational Linguistics, 2737\u20132746. DOI: 10.18653\/V1\/P18-1255"},{"key":"e_1_3_1_41_2","first-page":"143","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (NAACL-HLT \u201919)","author":"Rao Sudha","year":"2019","unstructured":"Sudha Rao and Hal Daum\u00e9, III. 2019. Answer-based adversarial training for generating clarification questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (NAACL-HLT \u201919). Jill Burstein, Christy Doran, and Thamar Solorio (Eds.), Association for Computational Linguistics, 143\u2013155. DOI: 10.18653\/V1\/N19-1013"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3432726"},{"key":"e_1_3_1_43_2","first-page":"902","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201924)","author":"Ren Ruiyang","year":"2024","unstructured":"Ruiyang Ren, Peng Qiu, Yingqi Qu, Jing Liu, Xin Zhao, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2024. BASES: Large-scale web search user simulation with large language model based agents. In Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201924). Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.), Association for Computational Linguistics, 902\u2013917. DOI: 10.18653\/v1\/2024.findings-emnlp.50"},{"issue":"4","key":"e_1_3_1_44_2","first-page":"333","article-title":"The probabilistic relevance framework: BM25 and beyond","volume":"3","author":"Robertson Stephen","year":"2009","unstructured":"Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval 3, 4 (2009), 333\u2013389.","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"e_1_3_1_45_2","volume-title":"Proceedings of the Web Conference 2020 (WWW \u201920)","author":"Rosset Corbin","year":"2020","unstructured":"Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. 2020. Leading conversational search by suggesting useful questions. In Proceedings of the Web Conference 2020 (WWW \u201920). ACM, New York, NY. DOI: 10.1145\/3366423.3380193"},{"key":"e_1_3_1_46_2","doi-asserted-by":"crossref","first-page":"2087","DOI":"10.18653\/v1\/D18-1233","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Saeidi Marzieh","year":"2018","unstructured":"Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rockt\u00e4schel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun\u2019ichi Tsujii (Eds.), Association for Computational Linguistics, 2087\u20132097. DOI: 10.18653\/v1\/D18-1233"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","unstructured":"Jost Schatzmann Karl Weilhammer Matt Stuttle and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. 21 2 (2006) 97\u2013126. DOI: 10.1017\/S0269888906000944","DOI":"10.1017\/S0269888906000944"},{"key":"e_1_3_1_48_2","first-page":"167","volume-title":"Proceedings of the 2021 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR \u201921)","author":"Sekulic Ivan","year":"2021","unstructured":"Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards facet-driven generation of clarifying questions for conversational search. In Proceedings of the 2021 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR \u201921). Faegheh Hasibi, Yi Fang, and Akiko Aizawa (Eds.), ACM, New York, NY, 167\u2013175. DOI: 10.1145\/3471158.3472257"},{"key":"e_1_3_1_49_2","first-page":"888","volume-title":"Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM \u201922)","author":"Sekulic Ivan","year":"2022","unstructured":"Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. 2022. Evaluating mixed-initiative conversational search systems via user simulation. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM \u201922). K. Selcuk Candan, Huan Liu, Leman Akoglu, Xin Luna Dong, and Jiliang Tang (Eds.), ACM, New York, NY, 888\u2013896. DOI: 10.1145\/3488560.3498440"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650041"},{"key":"e_1_3_1_51_2","first-page":"384","volume-title":"Proceedings of the 46th European Conference on Information Retrieval (ECIR \u201924)","volume":"14610","author":"Sekulic Ivan","year":"2024","unstructured":"Ivan Sekulic, Weronika Lajewska, Krisztian Balog, and Fabio Crestani. 2024. Estimating the usefulness of clarifying questions and answers for conversational search. In Proceedings of the 46th European Conference on Information Retrieval (ECIR \u201924). Nazli Goharian, Nicola Tonellotto, Yulan He, Aldo Lipani, Graham McDonald, Craig Macdonald, and Iadh Ounis (Eds.), Lecture Notes in Computer Science, Vol. 14610, Springer, 384\u2013392. DOI: 10.1007\/978-3-031-56063-7_30"},{"key":"e_1_3_1_52_2","volume-title":"Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP \u201924)","author":"Sekuli\u0107 Ivan","year":"2024","unstructured":"Ivan Sekuli\u0107, Lili Lu, Navdeep Singh Bedi, and Fabio Crestani. 2024. Simulating conversational search users with parameterized behavior. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP \u201924). ACM, New York, NY. DOI: 10.1145\/3673791.3698425"},{"key":"e_1_3_1_53_2","doi-asserted-by":"crossref","first-page":"19","DOI":"10.18653\/v1\/2024.scichat-1.3","volume-title":"Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT \u201924)","author":"Sekulic Ivan","year":"2024","unstructured":"Ivan Sekulic, Silvia Terragni, Victor Guimar\u00e3es, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis. 2024. Reliable LLM-based user simulator for task-oriented dialogue systems. In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT \u201924). Yvette Graham, Qun Liu, Gerasimos Lampouras, Ignacio Iacobacci, Sinead Madden, Haider Khalid, and Rameez Qureshi (Eds.), Association for Computational Linguistics, 19\u201335. Retrieved from https:\/\/aclanthology.org\/2024.scichat-1.3\/"},{"key":"e_1_3_1_54_2","doi-asserted-by":"crossref","first-page":"4215","DOI":"10.18653\/v1\/2023.findings-emnlp.278","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201923)","author":"Shen Chenhui","year":"2023","unstructured":"Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. Large language models are not yet human-level evaluators for abstractive summarization. In Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201923). Houda Bouamor, Juan Pino, and Kalika Bali (Eds.), Association for Computational Linguistics, 4215\u20134233. DOI: 10.18653\/v1\/2023.findings-emnlp.278"},{"key":"e_1_3_1_55_2","doi-asserted-by":"crossref","first-page":"2442","DOI":"10.18653\/v1\/D19-1248","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Shen Tao","year":"2019","unstructured":"Tao Shen, Xiubo Geng, Tao Qin, Daya Guo, Duyu Tang, Nan Duan, Guodong Long, and Daxin Jiang. 2019. Multi-task learning for conversational question answering over a large-scale knowledge base. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.), Association for Computational Linguistics, 2442\u20132451. DOI: 10.18653\/v1\/D19-1248"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3624989"},{"key":"e_1_3_1_57_2","doi-asserted-by":"crossref","first-page":"1258","DOI":"10.18653\/v1\/2024.findings-naacl.80","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (NAACL \u201924)","author":"Siro Clemencia","year":"2024","unstructured":"Clemencia Siro, Mohammad Aliannejadi, and Maarten de Rijke. 2024. Context does matter: Implications for crowdsourced evaluation labels in task-oriented dialogue systems. In Proceedings of the Findings of the Association for Computational Linguistics (NAACL \u201924). Kevin Duh, Helena Gomez, and Steven Bethard (Eds.), Association for Computational Linguistics, 1258\u20131273. DOI: 10.18653\/v1\/2024.findings-naacl.80"},{"key":"e_1_3_1_58_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. LLaMA: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_1_59_2","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1145\/3437963.3441748","volume-title":"Proceedings of the 14th International Conference on Web Search and Data Mining (WSDM \u201921)","author":"Vakulenko Svitlana","year":"2021","unstructured":"Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th International Conference on Web Search and Data Mining (WSDM \u201921). ACM, New York, NY, 355\u2013363. DOI: 10.1145\/3437963.3441748"},{"key":"e_1_3_1_60_2","doi-asserted-by":"crossref","first-page":"3288","DOI":"10.1145\/3543507.3583420","volume-title":"Proceedings of the ACM Web Conference 2023 (WWW \u201923)","author":"Wang Zhenduo","year":"2023","unstructured":"Zhenduo Wang, Yuancheng Tu, Corby Rosset, Nick Craswell, Ming Wu, and Qingyao Ai. 2023. Zero-shot clarifying question generation for conversational search. In Proceedings of the ACM Web Conference 2023 (WWW \u201923). Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben (Eds.), ACM, New York, NY, 3288\u20133298. DOI: 10.1145\/3543507.3583420"},{"key":"e_1_3_1_61_2","first-page":"11","volume-title":"Dialogue & Discourse","volume":"3","author":"Yao Xuchen","year":"2012","unstructured":"Xuchen Yao, Gosse Bouma, and Yi Zhang. 2012. Semantics-based question generation and implementation. Dialogue & Discourse 3, 2 (2012), 11\u201342."},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.naacl-long.83"},{"key":"e_1_3_1_63_2","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201920)","author":"Zamani Hamed","year":"2020","unstructured":"Hamed Zamani and Nick Craswell. 2020. Macaw: An extensible conversational information seeking platform. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201920). ACM, New York, NY. DOI: 10.1145\/3397271.3401415"},{"key":"e_1_3_1_64_2","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1145\/3366423.3380126","volume-title":"Proceedings of the Web Conference 2020 (WWW \u201920)","author":"Zamani Hamed","year":"2020","unstructured":"Hamed Zamani, Susan T. Dumais, Nick Craswell, Paul N. Bennett, and Gord Lueck. 2020. Generating clarifying questions for information retrieval. In Proceedings of the Web Conference 2020 (WWW \u201920). Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen (Eds.), ACM\/IW3C2, 418\u2013428. DOI: 10.1145\/3366423.3380126"},{"key":"e_1_3_1_65_2","first-page":"1181","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201920)","author":"Zamani Hamed","year":"2020","unstructured":"Hamed Zamani, Bhaskar Mitra, Everest Chen, Gord Lueck, Fernando Diaz, Paul N. Bennett, Nick Craswell, and Susan T. Dumais. 2020. Analyzing and learning from user interactions for search clarification. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201920). Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.), ACM, New York, NY, 1181\u20131190. DOI: 10.1145\/3397271.3401160"},{"key":"e_1_3_1_66_2","first-page":"244","volume-title":"Foundations and Trends\u00ae in Information Retrieval","volume":"17","author":"Zamani Hamed","year":"2023","unstructured":"Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. 2023. Conversational information seeking. Foundations and Trends\u00ae in Information Retrieval 17, 3\u20134 (2023), 244\u2013456. DOI: 10.1561\/1500000081"},{"key":"e_1_3_1_67_2","doi-asserted-by":"crossref","first-page":"1796","DOI":"10.1145\/3626772.3657828","volume-title":"Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Zhao Yuyue","year":"2024","unstructured":"Yuyue Zhao, Jiancan Wu, Xiang Wang, Wei Tang, Dingxian Wang, and Maarten de Rijke. 2024. Let me do it for you: Towards LLM empowered recommendation via tool learning. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 1796\u20131806."},{"key":"e_1_3_1_68_2","first-page":"3384","volume-title":"Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM \u201924)","author":"Zhao Ziliang","year":"2024","unstructured":"Ziliang Zhao, Zhicheng Dou, and Yujia Zhou. 2024. Generating intent-aware clarifying questions in conversational information retrieval systems. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM \u201924). Edoardo Serra and Francesca Spezzano (Eds.), ACM, New York, NY, 3384\u20133394. DOI: 10.1145\/3627673.3679851"},{"key":"e_1_3_1_69_2","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS \u201923)","year":"2023","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. 2023. Judging LLM-as-a-judge with MT-bench and ChatBot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS \u201923). Curran Associates Inc., Article 2020."},{"key":"e_1_3_1_70_2","unstructured":"OpenAI. (2023). GPT-4 Technical Report. arXiv preprint. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_1_71_2","first-page":"27730","volume-title":"Advances in Neural Information Processing Systems","volume":"35","author":"Ouyang L.","year":"2022","unstructured":"L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35 (2022), 27730\u201327744. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/hash\/b1efde53be364a73914f58805a001731-Abstract-Conference.html"},{"key":"e_1_3_1_72_2","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The Llama 3 Herd of Models. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:271571434"},{"key":"e_1_3_1_73_2","unstructured":"A. Yang A. Li B. Yang B. Zhang B. Hui B. Zheng B. Yu C. Gao C. Huang C. Lv et al. 2025. Qwen3 Technical Report. arXiv:2505.09388. Retrieved from https:\/\/arxiv.org\/abs\/2505.09388"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3809182","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T13:57:11Z","timestamp":1781186231000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3809182"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,11]]},"references-count":72,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3809182"],"URL":"https:\/\/doi.org\/10.1145\/3809182","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,11]]},"assertion":[{"value":"2025-06-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-30","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}