{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:10:40Z","timestamp":1784178640385,"version":"3.55.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p>\n            Simulation is used as a low-cost and repeatable means of experimentation. As\n            <jats:italic>Information Retrieval (IR)<\/jats:italic>\n            researchers, we are no strangers to the idea of using simulation within our own field---such as the traditional means of IR system evaluation as manifested through the\n            <jats:italic>Cranfield paradigm.<\/jats:italic>\n            While simulation has been used in other areas of IR research (such as the study of user behaviours), we argue that the potential for using simulation has been recognised by relatively few IR researchers so far.\n          <\/jats:p>\n          <jats:p>\n            To this end, the\n            <jats:italic>Sim4IR<\/jats:italic>\n            workshop was held online on July 15\n            <jats:sup>th<\/jats:sup>\n            , 2021 in conjunction with ACM SIGIR 2021. Building on past efforts, the goal of the workshop was to create a forum for researchers and practitioners to promote methodology and development of more widespread use of simulation for IR evaluation. Around 80 participants took part over two sessions. A total of two keynotes, three original paper presentations, and eight\n            <jats:italic>'encore talks'<\/jats:italic>\n            were presented. The main conclusions from the resultant discussion were that simulation has the potential to offer solutions to the limitations of existing evaluation methodologies, but there is more research needed toward developing realistic user simulators; and the development and sharing of simulators, in the form of toolkits and online services, is critical for successful uptake.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Date<\/jats:bold>\n            : 15 July, 2021.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Website<\/jats:bold>\n            : https:\/\/sim4ir.org.\n          <\/jats:p>","DOI":"10.1145\/3527546.3527559","type":"journal-article","created":{"date-parts":[[2022,3,17]],"date-time":"2022-03-17T23:24:36Z","timestamp":1647559476000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Report on the 1st simulation for information retrieval workshop (Sim4IR 2021) at SIGIR 2021"],"prefix":"10.1145","volume":"55","author":[{"given":"Krisztian","family":"Balog","sequence":"first","affiliation":[{"name":"University of Stavanger, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Maxwell","sequence":"additional","affiliation":[{"name":"Delft University of Technology, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Thomas","sequence":"additional","affiliation":[{"name":"Microsoft Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuo","family":"Zhang","sequence":"additional","affiliation":[{"name":"Bloomberg London, England"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,3,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2215676.2215678"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009923"},{"key":"e_1_2_1_3_1","first-page":"35","volume-title":"Mark D Smucker. Report on the SIGIR 2010 workshop on the simulation of interaction. In ACM SIGIR Forum","volume":"44","author":"Azzopardi Leif","year":"2011","unstructured":"Leif Azzopardi , Kalervo J\u00e4rvelin , Jaap Kamps , and Mark D Smucker. Report on the SIGIR 2010 workshop on the simulation of interaction. In ACM SIGIR Forum , volume 44 , pages 35 -- 47 , 2011 . Leif Azzopardi, Kalervo J\u00e4rvelin, Jaap Kamps, and Mark D Smucker. Report on the SIGIR 2010 workshop on the simulation of interaction. In ACM SIGIR Forum, volume 44, pages 35--47, 2011."},{"key":"e_1_2_1_4_1","series-title":"CEUR Workshop Proceedings","volume-title":"Paul Thomas","author":"Balog Krisztian","year":"2021","unstructured":"Krisztian Balog , Xianjie Chen , Xu Chen David Maxwell , Paul Thomas , Shuo Zhang, Yi Zhang, and Yongfeng Zhang, editors. CSR-Sim4IR 2021 : Joint Proceedings of the Causality in Search and Recommendation (CSR) and Simulation of Information Retrieval Evaluation (Sim4IR) Workshops, volume 2911 of CEUR Workshop Proceedings , 2021. CEUR-WS.org. URL http:\/\/ceur-ws.org\/Vol-2911. Krisztian Balog, Xianjie Chen, Xu Chen David Maxwell, Paul Thomas, Shuo Zhang, Yi Zhang, and Yongfeng Zhang, editors. CSR-Sim4IR 2021: Joint Proceedings of the Causality in Search and Recommendation (CSR) and Simulation of Information Retrieval Evaluation (Sim4IR) Workshops, volume 2911 of CEUR Workshop Proceedings, 2021. CEUR-WS.org. URL http:\/\/ceur-ws.org\/Vol-2911."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348301"},{"key":"e_1_2_1_6_1","first-page":"2297","volume-title":"Proceedings of the 22nd ACM CIKM","author":"Baskaya Feza","year":"2013","unstructured":"Feza Baskaya , Heikki Keskustalo , and Kalervo J\u00e4rvelin . Modeling behavioral factors in interactive information retrieval . In Proceedings of the 22nd ACM CIKM , pages 2297 -- 2302 , 2013 . Feza Baskaya, Heikki Keskustalo, and Kalervo J\u00e4rvelin. Modeling behavioral factors in interactive information retrieval. In Proceedings of the 22nd ACM CIKM, pages 2297--2302, 2013."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287583"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000055"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809470"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00654ED1V01Y201507ICR043"},{"key":"e_1_2_1_11_1","series-title":"Cranfield Research Projects","volume-title":"Factors Determining the Performance of Indexing Systems","author":"Cleverdon Cyril W.","year":"1966","unstructured":"Cyril W. Cleverdon , Jack Mills , and Michael Keen . Factors Determining the Performance of Indexing Systems , volume 1: 2 of Cranfield Research Projects . 1966 . Cyril W. Cleverdon, Jack Mills, and Michael Keen. Factors Determining the Performance of Indexing Systems, volume 1:2 of Cranfield Research Projects. 1966."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(73)90004-1"},{"key":"e_1_2_1_13_1","volume-title":"Report from the Third Strategic Workshop on Information Retrieval (SWIRL). 52(1):34--90","author":"Culpepper J Shane","year":"2018","unstructured":"J Shane Culpepper , Fernando Diaz , and Mark D Smucker . Report from the Third Strategic Workshop on Information Retrieval (SWIRL). 52(1):34--90 , 2018 . J Shane Culpepper, Fernando Diaz, and Mark D Smucker. Report from the Third Strategic Workshop on Information Retrieval (SWIRL). 52(1):34--90, 2018."},{"key":"e_1_2_1_14_1","first-page":"32","volume-title":"Balog et al. [2021]","author":"Erbacher Pierre","unstructured":"Pierre Erbacher , Laure Soulier , and Ludovic Denoyer . State of the art of user simulation approaches for conversational information retrieval . In Balog et al. [2021] , pages 32 -- 37 . URL http:\/\/ceur-ws.org\/Vol-2911\/paper5.pdf. Pierre Erbacher, Laure Soulier, and Ludovic Denoyer. State of the art of user simulation approaches for conversational information retrieval. In Balog et al. [2021], pages 32--37. URL http:\/\/ceur-ws.org\/Vol-2911\/paper5.pdf."},{"key":"e_1_2_1_15_1","volume-title":"Computer simulation: The art and science of digital world construction. Technical report","author":"Fishwick Paul A.","year":"1995","unstructured":"Paul A. Fishwick . Computer simulation: The art and science of digital world construction. Technical report , University of Florida , 1995 . Paul A. Fishwick. Computer simulation: The art and science of digital world construction. Technical report, University of Florida, 1995."},{"key":"e_1_2_1_16_1","volume-title":"Neural Approaches to Conversational AI. Foundations and Trends in Information Retrieval, 13(2--3):127--298","author":"Gao Jianfeng","year":"2019","unstructured":"Jianfeng Gao , Michel Galley , and Lihong Li . Neural Approaches to Conversational AI. Foundations and Trends in Information Retrieval, 13(2--3):127--298 , 2019 . Jianfeng Gao, Michel Galley, and Lihong Li. Neural Approaches to Conversational AI. Foundations and Trends in Information Retrieval, 13(2--3):127--298, 2019."},{"key":"e_1_2_1_17_1","first-page":"38","volume-title":"Balog et al. [2021]","author":"G\u00fcnther Sebastian","unstructured":"Sebastian G\u00fcnther and Matthias Hagen . Assessing query suggestions for search session simulation . In Balog et al. [2021] , pages 38 -- 45 . URL http:\/\/ceur-ws.org\/Vol-2911\/paper6.pdf. Sebastian G\u00fcnther and Matthias Hagen. Assessing query suggestions for search session simulation. In Balog et al. [2021], pages 38--45. URL http:\/\/ceur-ws.org\/Vol-2911\/paper6.pdf."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/160688.160692"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.2200\/S01043ED1V01Y202008ICR071"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000051"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3239570"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412252"},{"key":"e_1_2_1_23_1","volume-title":"Recsim: A configurable simulation platform for recommender systems. Computing Research Repository, abs\/1909.04847","author":"Ie Eugene","year":"2019","unstructured":"Eugene Ie , Chih-Wei Hsu , Martin Mladenov , Vihan Jain , Sanmit Narvekar , Jing Wang , Rui Wu , and Craig Boutilier . Recsim: A configurable simulation platform for recommender systems. Computing Research Repository, abs\/1909.04847 , 2019 . Eugene Ie, Chih-Wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. Recsim: A configurable simulation platform for recommender systems. Computing Research Repository, abs\/1909.04847, 2019."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3239575"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(99)00003-5"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000012"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1017\/9781108653985"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463091"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the Data Science for Retail and E-Commerce Workshop","author":"Labhishetty Sahiti","year":"2020","unstructured":"Sahiti Labhishetty , Chengxiang Zhai , Suhas Ranganath , and Pradeep Ranganathan . A cognitive user model for e-commerce search . In Proceedings of the Data Science for Retail and E-Commerce Workshop , 2020 . Sahiti Labhishetty, Chengxiang Zhai, Suhas Ranganath, and Pradeep Ranganathan. A cognitive user model for e-commerce search. In Proceedings of the Data Science for Retail and E-Commerce Workshop, 2020."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3451160"},{"key":"e_1_2_1_31_1","first-page":"587","volume-title":"Proceedings of the 37th ACM SIGIR","author":"Luo Jiyun","year":"2014","unstructured":"Jiyun Luo , Sicong Zhang , and Hui Yang . Win-win search : Dual-agent stochastic game in session search . In Proceedings of the 37th ACM SIGIR , pages 587 -- 596 , 2014 . Jiyun Luo, Sicong Zhang, and Hui Yang. Win-win search: Dual-agent stochastic game in session search. In Proceedings of the 37th ACM SIGIR, pages 587--596, 2014."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-16354-3_58"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983805"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911469"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911469"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-76941-7_16"},{"key":"e_1_2_1_38_1","first-page":"178","volume-title":"Proceedings of the 6th CLEF","author":"P\u00e4\u00e4kk\u00f6nen Teemu","year":"2015","unstructured":"Teemu P\u00e4\u00e4kk\u00f6nen , Kalervo J\u00e4rvelin , Jaana Kek\u00e4l\u00e4inen , Heikki Keskustalo , Feza Baskaya , David Maxwell , and Leif Azzopardi . Exploring behavioral dimensions in session effectiveness . In Proceedings of the 6th CLEF , pages 178 -- 189 , 2015 . Teemu P\u00e4\u00e4kk\u00f6nen, Kalervo J\u00e4rvelin, Jaana Kek\u00e4l\u00e4inen, Heikki Keskustalo, Feza Baskaya, David Maxwell, and Leif Azzopardi. Exploring behavioral dimensions in session effectiveness. In Proceedings of the 6th CLEF, pages 178--189, 2015."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3471158.3472243"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72113-8_39"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240325"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 5th HCIR","author":"Smucker Mark D.","year":"2011","unstructured":"Mark D. Smucker . An analysis of user strategies for examining and processing ranked lists of documents . In Proceedings of the 5th HCIR , 2011 . Mark D. Smucker. An analysis of user strategies for examining and processing ranked lists of documents. In Proceedings of the 5th HCIR, 2011."},{"key":"e_1_2_1_44_1","volume-title":"Matthew Lease. Overview of the TREC 2012 crowdsourcing track. In Proceedings of the 21st TREC","author":"Smucker Mark D.","year":"2012","unstructured":"Mark D. Smucker , Gabriella Kazai , and Matthew Lease. Overview of the TREC 2012 crowdsourcing track. In Proceedings of the 21st TREC , 2012 . Mark D. Smucker, Gabriella Kazai, and Matthew Lease. Overview of the TREC 2012 crowdsourcing track. In Proceedings of the 21st TREC, 2012."},{"key":"e_1_2_1_45_1","first-page":"236","volume-title":"Proceedings of the 3rd ACM SIGIR","author":"Tague Jean","year":"1980","unstructured":"Jean Tague , Michael Nelson , and Harry Wu . Problems in the simulation of bibliographic retrieval systems . In Proceedings of the 3rd ACM SIGIR , pages 236 -- 255 , 1980 . Jean Tague, Michael Nelson, and Harry Wu. Problems in the simulation of bibliographic retrieval systems. In Proceedings of the 3rd ACM SIGIR, pages 236--255, 1980."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-16354-3_75"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1_2"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357945"},{"key":"e_1_2_1_49_1","first-page":"255","volume-title":"Proceedings of the 40th ACM SIGIR","author":"Zhang Shuo","year":"2017","unstructured":"Shuo Zhang and Krisztian Balog . EntiTables : Smart assistance for entity-focused tables . In Proceedings of the 40th ACM SIGIR , pages 255 -- 264 , 2017 . Shuo Zhang and Krisztian Balog. EntiTables: Smart assistance for entity-focused tables. In Proceedings of the 40th ACM SIGIR, pages 255--264, 2017."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403202"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121070"},{"key":"e_1_2_1_52_1","first-page":"46","volume-title":"Balog et al. [2021]","author":"Zhao Yurou","unstructured":"Yurou Zhao , Jiaxin Mao , and Qingyao Ai . ULTRE framework: a framework for unbiased learning to rank evaluation based on simulation of user behavior . In Balog et al. [2021] , pages 46 -- 51 . URL http:\/\/ceur-ws.org\/Vol-2911\/paper7.pdf. Yurou Zhao, Jiaxin Mao, and Qingyao Ai. ULTRE framework: a framework for unbiased learning to rank evaluation based on simulation of user behavior. In Balog et al. [2021], pages 46--51. URL http:\/\/ceur-ws.org\/Vol-2911\/paper7.pdf."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527559","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3527546.3527559","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:53Z","timestamp":1750191533000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527559"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":51,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1145\/3527546.3527559"],"URL":"https:\/\/doi.org\/10.1145\/3527546.3527559","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"2022-03-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}