{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T20:30:26Z","timestamp":1773520226432,"version":"3.50.1"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2025,12,1]]},"abstract":"<jats:p>The Cranfield paradigm has been the dominant approach to evaluate information retrieval systems for decades, but\u2014in its classical form\u2014has clear limitations when it comes to conversational search systems, which synthesize unique outputs in a dynamic multi-turn interaction with the user. User simulation, i.e., the interaction of a computer program with a retrieval system instead of a human user to generate plausible conversations as a basis for evaluation, was proposed several years ago as a way to integrate the dynamics of conversational systems into an evaluation framework. Seen as a distant vision for years, the advent of large language models has propelled this idea forward. In 2025, there were the first three shared tasks in information retrieval where user simulation was used for evaluation or was the participants' goal. In this article, the organizers of these three shared tasks report on their specific evaluation approaches, highlight differences in setup, report on insights gained, and look to the future to discuss how user simulation can be integrated into a new evaluation paradigm.<\/jats:p>","DOI":"10.1145\/3799914.3799917","type":"journal-article","created":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T18:19:03Z","timestamp":1772648343000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["User Simulation in Practice: Lessons Learned from Three Shared Tasks"],"prefix":"10.1145","volume":"59","author":[{"given":"Marcel","family":"Gohsen","sequence":"first","affiliation":[{"name":"Bauhaus-Universit\u00e4t Weimar, Weimar, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zahra","family":"Abbasiantaeb","sequence":"additional","affiliation":[{"name":"University of Amsterdam; Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammad","family":"Aliannejadi","sequence":"additional","affiliation":[{"name":"University of Amsterdam; Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Krisztian","family":"Balog","sequence":"additional","affiliation":[{"name":"University of Stavanger; Stavanger, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timo","family":"Breuer","sequence":"additional","affiliation":[{"name":"TH K\u00f6ln; Cologne, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeffrey","family":"Dalton","sequence":"additional","affiliation":[{"name":"University of Edinburgh; Edinburgh, Scotland, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maik","family":"Fr\u00f6be","sequence":"additional","affiliation":[{"name":"Friedrich-Schiller-Universit\u00e4t Jena; Jena, Germany;"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christin Katharina","family":"Kreutz","sequence":"additional","affiliation":[{"name":"TH Mittelhessen; Gie\u00dfen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Kruff","sequence":"additional","affiliation":[{"name":"TH K\u00f6ln; Cologne, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Simon","family":"Lupart","sequence":"additional","affiliation":[{"name":"University of Amsterdam; Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nailia","family":"Mirzakhmedova","sequence":"additional","affiliation":[{"name":"Bauhaus-Universit\u00e4t Weimar; Weimar, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Harrisen","family":"Scells","sequence":"additional","affiliation":[{"name":"University of T\u00fcbingen; T\u00fcbingen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philipp","family":"Schaer","sequence":"additional","affiliation":[{"name":"TH K\u00f6ln; Cologne, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benno","family":"Stein","sequence":"additional","affiliation":[{"name":"Bauhaus-Universit\u00e4t Weimar; Weimar, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Johannes","family":"Kiesel","sequence":"additional","affiliation":[{"name":"GESIS - Leibniz Institute for the Social Sciences; Cologne, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,3,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"crossref","unstructured":"Jafar Afzali Aleksander Mark Drzewiecki Krisztian Balog and Shuo Zhang. UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining WSDM '23 pages 1160\u20131163 2023.","DOI":"10.1145\/3539597.3573029"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572037"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009923"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148276"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3673791.3698427"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1561\/9781638283799"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462821"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348301"},{"key":"e_1_2_1_9_1","volume-title":"UserSimCRS v2: Simulation-Based Evaluation for Conversational Recommender Systems","author":"Bernard Nolwenn","year":"2025","unstructured":"Nolwenn Bernard and Krisztian Balog. UserSimCRS v2: Simulation-Based Evaluation for Conversational Recommender Systems, 2025."},{"key":"e_1_2_1_10_1","volume-title":"Krisztian Balog, and ChengXiang Zhai. Sim-Lab: A Platform for Simulation-based Evaluation of Conversational Information Access Systems","author":"Bernard Nolwenn","year":"2025","unstructured":"Nolwenn Bernard, Sharath Chandra Etagi Suresh, Krisztian Balog, and ChengXiang Zhai. Sim-Lab: A Platform for Simulation-based Evaluation of Conversational Information Access Systems, 2025."},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Charles R. Blunt. An Information Retrieval System Model. Technical Report HRB-352.14-R-1 January 1965.","DOI":"10.21236\/AD0623590"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99736-6\\_6"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531738"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3722449.3722460"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","unstructured":"Matteo Cancellieri Alaa El-Ebshihy Tobias Fink Maik Fr\u00f6be Petra Galusc\u00e1kov\u00e1 Gabriela Gonz\u00e1lez S\u00e1ez Lorraine Goeuriot David Iommi J\u00fcri Keller Petr Knoth Philippe Mulhem Fiorina Piroi David Pride and Philipp Schaer. LongEval at CLEF 2025: Longitudinal Evaluation of IR Systems on Web and Scientific Data. In Jorge Carrillo-de-Albornoz Alba Garc\u00eda Seco de Herrera Julio Gonzalo Laura Plaza Josiane Mothe Florina Piroi Paolo Rosso Damiano Spina Guglielmo Faggioli and Nicola Ferro editors Experimental IR Meets Multilinguality Multimodality and Interaction - 16th International Conference of the CLEF Association CLEF 2025 Madrid Spain September 9\u201312 2025 Proceedings volume 16089 of Lecture Notes in Computer Science pages 363\u2013387. Springer 2025. 10.1007\/978-3-032-04354-2\\_20","DOI":"10.1007\/978-3-032-04354-2\\_20"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(73)90004-1"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(99)00072-2"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3731120.3744588"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2407.21783"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","unstructured":"Bj\u00f6rn Engelmann Timo Breuer Jana Isabelle Friese Philipp Schaer and Norbert Fuhr. Context-Driven Interactive Query Simulations Based on Generative Large Language Models. In Advances in Information Retrieval: 46th European Conference on Information Retrieval ECIR 2024 Glasgow UK March 24\u201328 2024 Proceedings Part II pages 173\u2013188 Berlin Heidelberg March 2024. Springer-Verlag. ISBN 978-3-031-56059-0. 10.1007\/978-3-031-56060-6_12","DOI":"10.1007\/978-3-031-56060-6_12"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406522.3446056"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730093"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097"},{"key":"e_1_2_1_24_1","unstructured":"Jos\u00e9-Marie Griffiths. The Computer Simulation of Information Retrieval Systems. PhD thesis University of London 1978."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133167"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1108\/EB026672"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141753.1141818"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/S10791-007-9043-7"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-56060-6_25"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-71736-9_11"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-032-04354-2_25"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-023-02208-w"},{"key":"e_1_2_1_33_1","volume-title":"Timo Breuer, Philipp Schaer, and Krisztian Balog. Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction","author":"Kruff Andreas Konstantin","year":"2025","unstructured":"Andreas Konstantin Kruff, Christin Katharina Kreutz, Timo Breuer, Philipp Schaer, and Krisztian Balog. Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction, 2025."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/354756.354809"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591683"},{"key":"e_1_2_1_36_1","series-title":"CEUR Workshop Proceedings","volume-title":"Giuseppe Di Fabbrizio","author":"Penha Gustavo","year":"2020","unstructured":"Gustavo Penha and Claudia Hauff. Challenges in the Evaluation of Conversational Search Systems. In Giuseppe Di Fabbrizio, Surya Kallumadi, Utkarsh Porwal, and Thrivikrama Taula, editors, Proceedings of the KDD 2020 Workshop on Conversational Systems Towards Mainstream Adoption Co-Located with the 26TH ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD 2020), Virtual Workshop, August 24, 2020, volume 2666 of CEUR Workshop Proceedings. CEUR-WS.org, 2020."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657991"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3799914.3799927"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730363"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3498440"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650041"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.EACL-MAIN.147"},{"key":"e_1_2_1_43_1","first-page":"255","volume-title":"Proc. Joint ACM\/BCS Symposium in Information Storage and Retrieval","author":"Tague Jean","year":"1980","unstructured":"Jean Tague, Michael J. Nelson, and Harry Wu. Problems in the Simulation of Bibliographic Retrieval Systems. In Robert N. Oddy, Stephen E. Robertson, C. J. van Rijsbergen, and P. W. Williams, editors, Information Retrieval Research, Proc. Joint ACM\/BCS Symposium in Information Storage and Retrieval, Cambridge, UK, June 1980, pages 236\u2013255. Butterworths, 1980."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45691-0\\_34"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589334.3645447"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1080343.1080347"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3799914.3799917","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T17:19:36Z","timestamp":1773508776000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3799914.3799917"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,1]]},"references-count":47,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,12,1]]}},"alternative-id":["10.1145\/3799914.3799917"],"URL":"https:\/\/doi.org\/10.1145\/3799914.3799917","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2025,12,1]]},"assertion":[{"value":"2026-03-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}