{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T20:12:50Z","timestamp":1783541570806,"version":"3.55.0"},"reference-count":102,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T00:00:00Z","timestamp":1709683200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for small-scale platforms to validate information retrieval systems with real users. If enough user interaction data are available, then click models can be parameterized from historical sessions to evaluate systems before exposing users to experimental rankings. However, interaction data are sparse in living labs, and little is studied about how click models can be validated for reliable user simulations when click data are available in moderate amounts.<\/jats:p>\n          <jats:p>This work introduces an evaluation approach for validating synthetic usage data generated by click models in data-sparse human-in-the-loop environments like living labs. We ground our methodology on the click model\u2019s estimates about a system ranking compared to a reference ranking for which the relative performance is known. Our experiments compare different click models and their reliability and robustness as more session log data become available. In our setup, simple click models can reliably determine the relative system performance with already 20 logged sessions for 50 queries. In contrast, more complex click models require more session data for reliable estimates, but they are a better choice in simulated interleaving experiments when enough session data are available. While it is easier for click models to distinguish between more diverse systems, it is harder to reproduce the system ranking based on the same retrieval algorithm with different interpolation weights. Our setup is entirely open, and we share the code to reproduce the experiments.<\/jats:p>","DOI":"10.1145\/3623640","type":"journal-article","created":{"date-parts":[[2023,9,24]],"date-time":"2023-09-24T08:10:13Z","timestamp":1695543013000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Validating Synthetic Usage Data in Living Lab Environments"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1765-2449","authenticated-orcid":false,"given":"Timo","family":"Breuer","sequence":"first","affiliation":[{"name":"TH K\u00f6ln - University of Applied Sciences, K\u00f6ln, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0441-6949","authenticated-orcid":false,"given":"Norbert","family":"Fuhr","sequence":"additional","affiliation":[{"name":"University of Duisburg-Essen, Duisburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8817-4632","authenticated-orcid":false,"given":"Philipp","family":"Schaer","sequence":"additional","affiliation":[{"name":"TH K\u00f6ln - University of Applied Sciences, K\u00f6ln, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,6]]},"reference":[{"key":"e_1_3_3_2_2","first-page":"19","volume-title":"Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201906)","author":"Agichtein Eugene","year":"2006","unstructured":"Eugene Agichtein, Eric Brill, and Susan T. Dumais. 2006. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201906), Efthimis N. Efthimiadis, Susan T. Dumais, David Hawking, and Kalervo J\u00e4rvelin (Eds.). ACM, 19\u201326. DOI:10.1145\/1148170.1148177"},{"key":"e_1_3_3_3_2","doi-asserted-by":"crossref","first-page":"3801","DOI":"10.1145\/3511808.3557714","volume-title":"Proceedings of the 31st ACM International Conference on Information & Knowledge Management","author":"Althammer Sophia","year":"2022","unstructured":"Sophia Althammer, Sebastian Hofst\u00e4tter, Suzan Verberne, and Allan Hanbury. 2022. TripJudge: A relevance judgement test collection for TripClick health retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Mohammad Al Hasan and Li Xiong (Eds.). ACM, 3801\u20133805. DOI:10.1145\/3511808.3557714"},{"key":"e_1_3_3_4_2","series-title":"Advances in Information Retrieval: Proceedings of the 28th European Conference on IR Research (ECIR 2006),","first-page":"13","volume":"3936","author":"Amati Giambattista","year":"2006","unstructured":"Giambattista Amati. 2006. Frequentist and Bayesian approach to information retrieval. In Advances in Information Retrieval: Proceedings of the 28th European Conference on IR Research (ECIR 2006),Lecture Notes in Computer Science, Vol. 3936, Mounia Lalmas, Andy MacFarlane, Stefan M. R\u00fcger, Anastasios Tombros, Theodora Tsikrika, and Alexei Yavlinsky (Eds.). Springer, 13\u201324. DOI:10.1007\/11735106_3"},{"key":"e_1_3_3_5_2","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1007\/978-3-642-23708-9_5","volume-title":"Multilingual and Multimodal Information Access Evaluation: Proceedings of the 2nd International Conference of the Cross-Language Evaluation Forum (CLEF\u201911)","author":"Azzopardi Leif","year":"2011","unstructured":"Leif Azzopardi and Krisztian Balog. 2011. Towards a living lab for information retrieval research and development - A proposal for a living lab for product search tasks. In Multilingual and Multimodal Information Access Evaluation: Proceedings of the 2nd International Conference of the Cross-Language Evaluation Forum (CLEF\u201911). 26\u201337. DOI:10.1007\/978-3-642-23708-9_5"},{"key":"e_1_3_3_6_2","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1145\/2566486.2567967","volume-title":"Proceedings of the 23rd International World Wide Web Conference (WWW\u201914)","author":"Bakshy Eytan","year":"2014","unstructured":"Eytan Bakshy, Dean Eckles, and Michael S. Bernstein. 2014. Designing and deploying online field experiments. In Proceedings of the 23rd International World Wide Web Conference (WWW\u201914), Chin-Wan Chung, Andrei Z. Broder, Kyuseok Shim, and Torsten Suel (Eds.). ACM, 283\u2013292. DOI:10.1145\/2566486.2567967"},{"key":"e_1_3_3_7_2","first-page":"2557","volume-title":"Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM\u201913)","author":"Balog Krisztian","year":"2013","unstructured":"Krisztian Balog, David Elsweiler, Evangelos Kanoulas, Liadh Kelly, and Mark D. Smucker. 2013. CIKM 2013 workshop on living labs for information retrieval evaluation. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM\u201913). 2557\u20132558. DOI:10.1145\/2505515.2505815"},{"key":"e_1_3_3_8_2","first-page":"1815","volume-title":"Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM\u201914)","author":"Balog Krisztian","year":"2014","unstructured":"Krisztian Balog, Liadh Kelly, and Anne Schuth. 2014. Head first: Living labs for Ad-Hoc search evaluation. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM\u201914), Jianzhong Li, Xiaoyang Sean Wang, Minos N. Garofalakis, Ian Soboroff, Torsten Suel, and Min Wang (Eds.). ACM, 1815\u20131818. DOI:10.1145\/2661829.2661962"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3527546.3527559"},{"key":"e_1_3_3_10_2","first-page":"2297","volume-title":"Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM\u201913)","author":"Baskaya Feza","year":"2013","unstructured":"Feza Baskaya, Heikki Keskustalo, and Kalervo J\u00e4rvelin. 2013. Modeling behavioral factors in interactive information retrieval. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM\u201913), Qi He, Arun Iyengar, Wolfgang Nejdl, Jian Pei, and Rajeev Rastogi (Eds.). ACM, 2297\u20132302. DOI:10.1145\/2505515.2505660"},{"key":"e_1_3_3_11_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/978-3-030-15719-7_27","volume-title":"Advances in Information Retrieval: Proceedings of the 41st European Conference on IR Research (ECIR\u201919), Part II","author":"Beel J\u00f6ran","year":"2019","unstructured":"J\u00f6ran Beel, Andrew Collins, Oliver Kopp, Linus W. Dietz, and Petr Knoth. 2019. Online evaluations for everyone: Mr. DLib\u2019s living lab for scholarly recommendations. In Advances in Information Retrieval: Proceedings of the 41st European Conference on IR Research (ECIR\u201919), Part II. 213\u2013219. DOI:10.1007\/978-3-030-15719-7_27"},{"key":"e_1_3_3_12_2","first-page":"3613","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP\u201919)","author":"Beltagy Iz","year":"2019","unstructured":"Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP\u201919), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 3613\u20133618. DOI:10.18653\/v1\/D19-1371"},{"key":"e_1_3_3_13_2","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1145\/2872427.2883033","volume-title":"Proceedings of the 25th International Conference on World Wide Web (WWW\u201916)","author":"Borisov Alexey","year":"2016","unstructured":"Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A neural click model for web search. In Proceedings of the 25th International Conference on World Wide Web (WWW\u201916), Jacqueline Bourdeau, Jim Hendler, Roger Nkambou, Ian Horrocks, and Ben Y. Zhao (Eds.). ACM, 531\u2013541. DOI:10.1145\/2872427.2883033"},{"key":"e_1_3_3_14_2","first-page":"349","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201920)","author":"Breuer Timo","year":"2020","unstructured":"Timo Breuer, Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Philipp Schaer, and Ian Soboroff. 2020. How to measure the reproducibility of system-oriented IR experiments. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201920), Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.). ACM, 349\u2013358. DOI:10.1145\/3397271.3401036"},{"key":"e_1_3_3_15_2","first-page":"348","volume-title":"Information between Data and Knowledge: Information Science and Its Neighbors from Data Science to Digital Humanities: Proceedings of the 16th International Symposium of Information Science (ISI\u201921)","author":"Breuer Timo","year":"2021","unstructured":"Timo Breuer and Philipp Schaer. 2021. A living lab architecture for reproducible shared task experimentation. In Information between Data and Knowledge: Information Science and Its Neighbors from Data Science to Digital Humanities: Proceedings of the 16th International Symposium of Information Science (ISI\u201921), Christian Wolff and Thomas Schmidt (Eds.). Werner H\u00fclsbusch, 348\u2013362. DOI:10.5283\/epub.44953"},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1145\/2637002.2637028","volume-title":"Proceedings of the 5th Information Interaction in Context Symposium (IIiX\u201914)","author":"Brodt Torben","year":"2014","unstructured":"Torben Brodt and Frank Hopfgartner. 2014. Shedding light on a living lab: The CLEF NEWSREEL open recommendation platform. In Proceedings of the 5th Information Interaction in Context Symposium (IIiX\u201914). 223\u2013226. DOI:10.1145\/2637002.2637028"},{"key":"e_1_3_3_17_2","first-page":"25","volume-title":"Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201904)","author":"Buckley Chris","year":"2004","unstructured":"Chris Buckley and Ellen M. Voorhees. 2004. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201904), Mark Sanderson, Kalervo J\u00e4rvelin, James Allan, and Peter Bruza (Eds.). ACM, 25\u201332. DOI:10.1145\/1008992.1009000"},{"key":"e_1_3_3_18_2","first-page":"903","volume-title":"Proceeding of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR\u201911)","author":"Carterette Ben","year":"2011","unstructured":"Ben Carterette. 2011. System effectiveness, user models, and user utility: A conceptual framework for investigation. In Proceeding of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR\u201911). 903\u2013912. DOI:10.1145\/2009916.2010037"},{"key":"e_1_3_3_19_2","first-page":"91","volume-title":"Proceedings of the International Conference on the Theory of Information Retrieval (ICTIR\u201915)","author":"Carterette Ben","year":"2015","unstructured":"Ben Carterette, Ashraf Bah, and Mustafa Zengin. 2015. Dynamic test collections for retrieval evaluation. In Proceedings of the International Conference on the Theory of Information Retrieval (ICTIR\u201915), James Allan, W. Bruce Croft, Arjen P. de Vries, and Chengxiang Zhai (Eds.). ACM, 91\u2013100. DOI:10.1145\/2808194.2809470"},{"key":"e_1_3_3_20_2","first-page":"705","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR\u201918)","author":"Carterette Ben","year":"2018","unstructured":"Ben Carterette and Praveen Chandar. 2018. Offline comparative evaluation with incremental, minimally-invasive online feedback. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR\u201918), Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 705\u2013714. DOI:10.1145\/3209978.3210050"},{"key":"e_1_3_3_21_2","first-page":"217","volume-title":"Advances in Neural Information Processing Systems 20: Proceedings of the 21st Annual Conference on Neural Information Processing Systems","author":"Carterette Ben","year":"2007","unstructured":"Ben Carterette and Rosie Jones. 2007. Evaluating search engines by modeling the relationship between relevance and clicks. In Advances in Neural Information Processing Systems 20: Proceedings of the 21st Annual Conference on Neural Information Processing Systems, John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis (Eds.). Curran Associates, Inc., 217\u2013224."},{"key":"e_1_3_3_22_2","first-page":"1","volume-title":"Proceedings of the 18th International Conference on World Wide Web (WWW\u201909)","author":"Chapelle Olivier","year":"2009","unstructured":"Olivier Chapelle and Ya Zhang. 2009. A dynamic Bayesian network click model for web search ranking. In Proceedings of the 18th International Conference on World Wide Web (WWW\u201909), Juan Quemada, Gonzalo Le\u00f3n, Yo\u00eblle S. Maarek, and Wolfgang Nejdl (Eds.). ACM, 1\u201310. DOI:10.1145\/1526709.1526711"},{"key":"e_1_3_3_23_2","first-page":"15","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Chen Ye","year":"2017","unstructured":"Ye Chen, Ke Zhou, Yiqun Liu, Min Zhang, and Shaoping Ma. 2017. Meta-evaluation of online and offline web search evaluation metrics. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White (Eds.). ACM, 15\u201324. DOI:10.1145\/3077136.3080804"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.2200\/S00654ED1V01Y201507ICR043"},{"key":"e_1_3_3_25_2","first-page":"669","volume-title":"Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM\u201913)","author":"Chuklin Aleksandr","year":"2013","unstructured":"Aleksandr Chuklin, Anne Schuth, Katja Hofmann, Pavel Serdyukov, and Maarten de Rijke. 2013. Evaluating aggregated search using interleaving. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM\u201913), Qi He, Arun Iyengar, Wolfgang Nejdl, Jian Pei, and Rajeev Rastogi (Eds.). ACM, 669\u2013678. DOI:10.1145\/2505515.2505698"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2668120"},{"key":"e_1_3_3_27_2","first-page":"493","volume-title":"Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201913)","author":"Chuklin Aleksandr","year":"2013","unstructured":"Aleksandr Chuklin, Pavel Serdyukov, and Maarten de Rijke. 2013. Click model-based information retrieval metrics. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201913), Gareth J. F. Jones, Paraic Sheridan, Diane Kelly, Maarten de Rijke, and Tetsuya Sakai (Eds.). ACM, 493\u2013502. DOI:10.1145\/2484028.2484071"},{"key":"e_1_3_3_28_2","first-page":"3","volume-title":"Proceedings of the 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Cleverdon Cyril W.","year":"1991","unstructured":"Cyril W. Cleverdon. 1991. The significance of the cranfield tests on index languages. In Proceedings of the 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Abraham Bookstein, Yves Chiaramella, Gerard Salton, and Vijay V. Raghavan (Eds.). ACM, 3\u201312. DOI:10.1145\/122860.122861"},{"key":"e_1_3_3_29_2","doi-asserted-by":"crossref","first-page":"2270","DOI":"10.18653\/v1\/2020.acl-main.207","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL\u201920)","author":"Cohan Arman","year":"2020","unstructured":"Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL\u201920), Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 2270\u20132282. DOI:10.18653\/v1\/2020.acl-main.207"},{"key":"e_1_3_3_30_2","first-page":"2983","volume-title":"Proceedings of the 29th ACM International Conference on Information and Knowledge Management, Virtual Event (CIKM\u201920)","author":"Craswell Nick","year":"2020","unstructured":"Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. 2020. ORCAS: 20 million clicked query-document pairs for analyzing search. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, Virtual Event (CIKM\u201920), Mathieu d\u2019Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudr\u00e9-Mauroux (Eds.). ACM, 2983\u20132989. DOI:10.1145\/3340531.3412779"},{"key":"e_1_3_3_31_2","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1145\/1341531.1341545","volume-title":"Proceedings of the International Conference on Web Search and Web Data Mining (WSDM\u201908)","author":"Craswell Nick","year":"2008","unstructured":"Nick Craswell, Onno Zoeter, Michael J. Taylor, and Bill Ramsey. 2008. An experimental comparison of click position-bias models. In Proceedings of the International Conference on Web Search and Web Data Mining (WSDM\u201908), Marc Najork, Andrei Z. Broder, and Soumen Chakrabarti (Eds.). ACM, 87\u201394. DOI:10.1145\/1341531.1341545"},{"key":"e_1_3_3_32_2","first-page":"4171","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919), Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919), Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171\u20134186. DOI:10.18653\/v1\/n19-1423"},{"key":"e_1_3_3_33_2","first-page":"1399","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201919)","author":"Drutsa Alexey","year":"2019","unstructured":"Alexey Drutsa, Gleb Gusev, Eugene Kharitonov, Denis Kulemyakin, Pavel Serdyukov, and Igor Yashkov. 2019. Effective online evaluation for web search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201919), Benjamin Piwowarski, Max Chevalier, \u00c9ric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 1399\u20131400. DOI:10.1145\/3331184.3331378"},{"key":"e_1_3_3_34_2","first-page":"3393","volume-title":"Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM\u201920)","author":"Gingstad Kristian","year":"2020","unstructured":"Kristian Gingstad, \u00d8yvind Jekteberg, and Krisztian Balog. 2020. ArXivDigest: A living lab for personalized scientific literature recommendation. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM\u201920), Mathieu d\u2019Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudr\u00e9-Mauroux (Eds.). ACM, 3393\u20133396. DOI:10.1145\/3340531.3417417"},{"key":"e_1_3_3_35_2","series-title":"Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of the 6th International Conference of the CLEF Association,(CLEF 2015),","first-page":"78","volume":"9283","author":"Grotov Artem","year":"2015","unstructured":"Artem Grotov, Aleksandr Chuklin, Ilya Markov, Luka Stout, Finde Xumara, and Maarten de Rijke. 2015. A comparative study of click models for web search. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of the 6th International Conference of the CLEF Association,(CLEF 2015),Lecture Notes in Computer Science, Vol. 9283, Josiane Mothe, Jacques Savoy, Jaap Kamps, Karen Pinel-Sauvagnat, Gareth J. F. Jones, Eric SanJuan, Linda Cappellato, and Nicola Ferro (Eds.). Springer, 78\u201390. DOI:10.1007\/978-3-319-24027-5_7"},{"key":"e_1_3_3_36_2","first-page":"124","volume-title":"Proceedings of the 2nd International Conference on Web Search and Web Data Mining (WSDM\u201909)","author":"Guo Fan","year":"2009","unstructured":"Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient multiple-click models in web search. In Proceedings of the 2nd International Conference on Web Search and Web Data Mining (WSDM\u201909), Ricardo Baeza-Yates, Paolo Boldi, Berthier A. Ribeiro-Neto, and Berkant Barla Cambazoglu (Eds.). ACM, 124\u2013131. DOI:10.1145\/1498759.1498818"},{"key":"e_1_3_3_37_2","first-page":"2029","volume-title":"Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM 2009)","author":"He Jing","year":"2009","unstructured":"Jing He, Chengxiang Zhai, and Xiaoming Li. 2009. Evaluation of methods for relative comparison of retrieval systems based on clickthroughs. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM 2009), David Wai-Lok Cheung, Il-Yeol Song, Wesley W. Chu, Xiaohua Hu, and Jimmy Lin (Eds.). ACM, 2029\u20132032. DOI:10.1145\/1645953.1646293"},{"key":"e_1_3_3_38_2","first-page":"17","volume-title":"Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201900)","author":"Hersh William R.","year":"2000","unstructured":"William R. Hersh, Andrew Turpin, Susan Price, Benjamin Chan, Dale Kraemer, Lynetta Sacherek, and Daniel Olson. 2000. Do batch and user evaluation give the same results? In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201900), Emmanuel J. Yannakoudakis, Nicholas J. Belkin, Peter Ingwersen, and Mun-Kew Leong (Eds.). ACM, 17\u201324. DOI:10.1145\/345508.345539"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000051"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2536736.2536737"},{"key":"e_1_3_3_41_2","series-title":"Advances in Information Retrieval: Proceedings of the 44th European Conference on IR Research (ECIR\u201922), Part II,","first-page":"144","volume":"13186","author":"Hofst\u00e4tter Sebastian","year":"2022","unstructured":"Sebastian Hofst\u00e4tter, Sophia Althammer, Mete Sertkan, and Allan Hanbury. 2022. Establishing strong baselines for TripClick health retrieval. In Advances in Information Retrieval: Proceedings of the 44th European Conference on IR Research (ECIR\u201922), Part II,Lecture Notes in Computer Science, Vol. 13186, Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil N\u00f8rv\u00e5g, and Vinay Setty (Eds.). Springer, 144\u2013152. DOI:10.1007\/978-3-030-99739-7_17"},{"key":"e_1_3_3_42_2","series-title":"Information Access Evaluation. Multilinguality, Multimodality, and Interaction: Proceedings of the 5th International Conference of the CLEF Initiative (CLEF\u201914),","first-page":"250","volume":"8685","author":"Hopfgartner Frank","year":"2014","unstructured":"Frank Hopfgartner, Benjamin Kille, Andreas Lommatzsch, Till Plumbaum, Torben Brodt, and Tobias Heintz. 2014. Benchmarking news recommendations in a living lab. In Information Access Evaluation. Multilinguality, Multimodality, and Interaction: Proceedings of the 5th International Conference of the CLEF Initiative (CLEF\u201914),Lecture Notes in Computer Science, Vol. 8685, Evangelos Kanoulas, Mihai Lupu, Paul D. Clough, Mark Sanderson, Mark M. Hall, Allan Hanbury, and Elaine G. Toms (Eds.). Springer, 250\u2013267. DOI:10.1007\/978-3-319-11382-1_21"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3239575"},{"key":"e_1_3_3_44_2","first-page":"285","volume-title":"Proceedings of the ACM Conference on Human Information Interaction and Retrieval (CHIIR\u201916)","author":"Jiang Jiepu","year":"2016","unstructured":"Jiepu Jiang and James Allan. 2016. Correlation between system and user metrics in a session. In Proceedings of the ACM Conference on Human Information Interaction and Retrieval (CHIIR\u201916), Diane Kelly, Robert Capra, Nicholas J. Belkin, Jaime Teevan, and Pertti Vakkari (Eds.). ACM, 285\u2013288. DOI:10.1145\/2854946.2855005"},{"key":"e_1_3_3_45_2","first-page":"154","volume-title":"Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201905)","author":"Joachims Thorsten","year":"2005","unstructured":"Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201905), Ricardo A. Baeza-Yates, Nivio Ziviani, Gary Marchionini, Alistair Moffat, and John Tait (Eds.). ACM, 154\u2013161. DOI:10.1145\/1076034.1076063"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1145\/1507509.1507522","volume-title":"Proceedings of the Workshop on Web Search Click Data (WSCD@WSDM\u201909)","author":"Kamps Jaap","year":"2009","unstructured":"Jaap Kamps, Marijn Koolen, and Andrew Trotman. 2009. Comparative analysis of clicks and judgments for IR evaluation. In Proceedings of the Workshop on Web Search Click Data (WSCD@WSDM\u201909), Nick Craswell, Rosie Jones, Georges Dupret, and Evelyne Viegas (Eds.). ACM, 80\u201387. DOI:10.1145\/1507509.1507522"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2009.82"},{"key":"e_1_3_3_49_2","first-page":"392","volume-title":"Proceedings of the Web Conference (WWW\u201922)","author":"Khandel Pooya","year":"2022","unstructured":"Pooya Khandel, Ilya Markov, Andrew Yates, and Ana Lucia Varbanescu. 2022. ParClick: A scalable algorithm for EM-based click models. In Proceedings of the Web Conference (WWW\u201922), Fr\u00e9d\u00e9rique Laforest, Rapha\u00ebl Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel M\u00e9dini (Eds.). ACM, 392\u2013400. DOI:10.1145\/3485447.3511967"},{"key":"e_1_3_3_50_2","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1145\/2766462.2767706","volume-title":"Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Kharitonov Eugene","year":"2015","unstructured":"Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. 2015. Optimised scheduling of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Ricardo Baeza-Yates, Mounia Lalmas, Alistair Moffat, and Berthier A. Ribeiro-Neto (Eds.). ACM, 453\u2013462. DOI:10.1145\/2766462.2767706"},{"key":"e_1_3_3_51_2","doi-asserted-by":"crossref","first-page":"1168","DOI":"10.1145\/2487575.2488217","volume-title":"Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201913)","author":"Kohavi Ron","year":"2013","unstructured":"Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201913), Inderjit S. Dhillon, Yehuda Koren, Rayid Ghani, Ted E. Senator, Paul Bradley, Rajesh Parekh, Jingrui He, Robert L. Grossman, and Ramasamy Uthurusamy (Eds.). ACM, 1168\u20131176. DOI:10.1145\/2487575.2488217"},{"key":"e_1_3_3_52_2","first-page":"1598","volume-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201921)","author":"Labhishetty Sahiti","year":"2021","unstructured":"Sahiti Labhishetty and Chengxiang Zhai. 2021. An exploration of tester-based evaluation of user simulators for comparing interactive retrieval systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201921), Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (Eds.). ACM, 1598\u20131602. DOI:10.1145\/3404835.3463091"},{"key":"e_1_3_3_53_2","series-title":"Advances in Information Retrieval: Proceedings of the 44th European Conference on IR Research (ECIR\u201922), Part I,","first-page":"336","volume":"13185","author":"Labhishetty Sahiti","year":"2022","unstructured":"Sahiti Labhishetty and ChengXiang Zhai. 2022. RATE: A reliability-aware tester-based evaluation framework of user simulators. In Advances in Information Retrieval: Proceedings of the 44th European Conference on IR Research (ECIR\u201922), Part I,Lecture Notes in Computer Science, Vol. 13185, Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil N\u00f8rv\u00e5g, and Vinay Setty (Eds.). Springer, 336\u2013350. DOI:10.1007\/978-3-030-99736-6_23"},{"key":"e_1_3_3_54_2","first-page":"37","volume-title":"Proceedings of the 8th ACM International Conference on Web Search and Data Mining (WSDM\u201915)","author":"Li Lihong","year":"2015","unstructured":"Lihong Li, Jinyoung Kim, and Imed Zitouni. 2015. Toward predicting the outcome of an A\/B experiment for search relevance. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining (WSDM\u201915), Xueqi Cheng, Hang Li, Evgeniy Gabrilovich, and Jie Tang (Eds.). ACM, 37\u201346. DOI:10.1145\/2684822.2685311"},{"key":"e_1_3_3_55_2","first-page":"1685","volume-title":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD\u201918)","author":"Li Shuai","year":"2018","unstructured":"Shuai Li, Yasin Abbasi-Yadkori, Branislav Kveton, S. Muthukrishnan, Vishwa Vinay, and Zheng Wen. 2018. Offline evaluation of ranking policies with click models. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD\u201918), Yike Guo and Faisal Farooq (Eds.). ACM, 1685\u20131694. DOI:10.1145\/3219819.3220028"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.2200\/S01123ED1V01Y202108HLT053"},{"key":"e_1_3_3_57_2","first-page":"923","volume-title":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD\u201919)","author":"Liu Mengyang","year":"2019","unstructured":"Mengyang Liu, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. Investigating cognitive effects in session-level search user satisfaction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD\u201919), Ankur Teredesai, Vipin Kumar, Ying Li, R\u00f3mer Rosales, Evimaria Terzi, and George Karypis (Eds.). ACM, 923\u2013931. DOI:10.1145\/3292500.3330981"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2988230"},{"key":"e_1_3_3_59_2","first-page":"2429","volume-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201921)","author":"MacAvaney Sean","year":"2021","unstructured":"Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. 2021. Simplified data wrangling with Ir_datasets. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201921), Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (Eds.). ACM, 2429\u20132436. DOI:10.1145\/3404835.3463254"},{"key":"e_1_3_3_60_2","first-page":"161","volume-title":"Proceedings of the ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR\u201920)","author":"Macdonald Craig","year":"2020","unstructured":"Craig Macdonald and Nicola Tonellotto. 2020. Declarative experimentation in information retrieval using PyTerrier. In Proceedings of the ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR\u201920), Krisztian Balog, Vinay Setty, Christina Lioma, Yiqun Liu, Min Zhang, and Klaus Berberich (Eds.). ACM, 161\u2013168. DOI:10.1145\/3409256.3409829"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2023.103332"},{"key":"e_1_3_3_62_2","first-page":"281","volume-title":"Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR\u201917)","author":"Malkevich Stepan","year":"2017","unstructured":"Stepan Malkevich, Ilya Markov, Elena Michailova, and Maarten de Rijke. 2017. Evaluating and analyzing click simulation in web search. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR\u201917), Jaap Kamps, Evangelos Kanoulas, Maarten de Rijke, Hui Fang, and Emine Yilmaz (Eds.). ACM, 281\u2013284. DOI:10.1145\/3121050.3121096"},{"key":"e_1_3_3_63_2","first-page":"125","volume-title":"Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR\u201919)","author":"Mao Jiaxin","year":"2019","unstructured":"Jiaxin Mao, Zhumin Chu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. Investigating the reliability of click models. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR\u201919), Yi Fang, Yi Zhang, James Allan, Krisztian Balog, Ben Carterette, and Jiafeng Guo (Eds.). ACM, 125\u2013128. DOI:10.1145\/3341981.3344242"},{"key":"e_1_3_3_64_2","first-page":"2195","volume-title":"Proceedings of the ACM on Conference on Information and Knowledge Management (CIKM\u201917)","author":"Markov Ilya","year":"2017","unstructured":"Ilya Markov, Alexey Borisov, and Maarten de Rijke. 2017. Online expectation-maximization for click models. In Proceedings of the ACM on Conference on Information and Knowledge Management (CIKM\u201917), Ee-Peng Lim, Marianne Winslett, Mark Sanderson, Ada Wai-Chee Fu, Jimeng Sun, J. Shane Culpepper, Eric Lo, Joyce C. Ho, Debora Donato, Rakesh Agrawal, Yu Zheng, Carlos Castillo, Aixin Sun, Vincent S. Tseng, and Chenliang Li (Eds.). ACM, 2195\u20132198. DOI:10.1145\/3132847.3133053"},{"key":"e_1_3_3_65_2","series-title":"Proceedings of the 1st Biennial Conference on Design of Experimental Search & Information Retrieval Systems","first-page":"47","volume":"2167","author":"Marrero M\u00f3nica","year":"2018","unstructured":"M\u00f3nica Marrero. 2018. APONE: Academic platform for ONline experiments. In Proceedings of the 1st Biennial Conference on Design of Experimental Search & Information Retrieval SystemsCEUR Workshop Proceedings, Vol. 2167, Omar Alonso and Gianmaria Silvello (Eds.). CEUR-WS.org, 47\u201353."},{"key":"e_1_3_3_66_2","first-page":"1269","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR\u201918)","author":"Marrero M\u00f3nica","year":"2018","unstructured":"M\u00f3nica Marrero and Claudia Hauff. 2018. A\/B testing with APONE. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR\u201918), Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 1269\u20131272. DOI:10.1145\/3209978.3210164"},{"key":"e_1_3_3_67_2","volume-title":"Modelling Search and Stopping in Interactive Information Retrieval","author":"Maxwell David","year":"2019","unstructured":"David Maxwell. 2019. Modelling Search and Stopping in Interactive Information Retrieval. Ph. D. Dissertation. University of Glasgow, UK."},{"key":"e_1_3_3_68_2","first-page":"731","volume-title":"Proceedings of the 25th ACM International Conference on Information and Knowledge Management (CIKM\u201916)","author":"Maxwell David","year":"2016","unstructured":"David Maxwell and Leif Azzopardi. 2016. Agents, simulated users and humans: An analysis of performance and behaviour. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management (CIKM\u201916), Snehasis Mukhopadhyay, ChengXiang Zhai, Elisa Bertino, Fabio Crestani, Javed Mostafa, Jie Tang, Luo Si, Xiaofang Zhou, Yi Chang, Yunyao Li, and Parikshit Sondhi (Eds.). ACM, 731\u2013740. DOI:10.1145\/2983323.2983805"},{"key":"e_1_3_3_69_2","first-page":"1141","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201916)","author":"Maxwell David","year":"2016","unstructured":"David Maxwell and Leif Azzopardi. 2016. Simulating interactive information retrieval: SimIIR: A framework for the simulation of interaction. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201916), Raffaele Perego, Fabrizio Sebastiani, Javed A. Aslam, Ian Ruthven, and Justin Zobel (Eds.). ACM, 1141\u20131144. DOI:10.1145\/2911451.2911469"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3052768"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_3_72_2","series-title":"Advances in Information Retrieval: Proceedings of the 27th European Conference on IR Research (ECIR\u201905),","first-page":"517","volume":"3408","author":"Ounis Iadh","year":"2005","unstructured":"Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and Douglas Johnson. 2005. Terrier information retrieval platform. In Advances in Information Retrieval: Proceedings of the 27th European Conference on IR Research (ECIR\u201905),Lecture Notes in Computer Science, Vol. 3408, David E. Losada and Juan M. Fern\u00e1ndez-Luna (Eds.). Springer, 517\u2013519. DOI:10.1007\/978-3-540-31865-1_37"},{"key":"e_1_3_3_73_2","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1145\/1963405.1963463","volume-title":"Proceedings of the 20th International Conference on World Wide Web (WWW\u201911)","author":"Ozertem Umut","year":"2011","unstructured":"Umut Ozertem, Rosie Jones, and Beno\u00eet Dumoulin. 2011. Evaluating new search engine configurations with pre-existing judgments and clicks. In Proceedings of the 20th International Conference on World Wide Web (WWW\u201911), Sadagopan Srinivasan, Krithi Ramamritham, Arun Kumar, M. P. Ravindra, Elisa Bertino, and Ravi Kumar (Eds.). ACM, 397\u2013406. DOI:10.1145\/1963405.1963463"},{"key":"e_1_3_3_74_2","first-page":"175","volume-title":"Proceedings of the 16th ACM Conference on Information and Knowledge Management (CIKM\u201907)","author":"Piwowarski Benjamin","year":"2007","unstructured":"Benjamin Piwowarski and Hugo Zaragoza. 2007. Predictive user click models based on click-through history. In Proceedings of the 16th ACM Conference on Information and Knowledge Management (CIKM\u201907), M\u00e1rio J. Silva, Alberto H. F. Laender, Ricardo A. Baeza-Yates, Deborah L. McGuinness, Bj\u00f8rn Olstad, \u00d8ystein Haug Olsen, and Andr\u00e9 O. Falc\u00e3o (Eds.). ACM, 175\u2013182. DOI:10.1145\/1321440.1321467"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-017-9301-2"},{"key":"e_1_3_3_76_2","first-page":"175","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201916)","author":"Qian Xin","year":"2016","unstructured":"Xin Qian, Jimmy Lin, and Adam Roegiest. 2016. Interleaved evaluation for retrospective summarization and prospective notification on document streams. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201916), Raffaele Perego, Fabrizio Sebastiani, Javed A. Aslam, Ian Ruthven, and Justin Zobel (Eds.). ACM, 175\u2013184. DOI:10.1145\/2911451.2911494"},{"key":"e_1_3_3_77_2","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1145\/1458082.1458092","volume-title":"Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM\u201908)","author":"Radlinski Filip","year":"2008","unstructured":"Filip Radlinski, Madhu Kurup, and Thorsten Joachims. 2008. How does clickthrough data reflect retrieval quality? In Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM\u201908), James G. Shanahan, Sihem Amer-Yahia, Ioana Manolescu, Yi Zhang, David A. Evans, Aleksander Kolcz, Key-Sun Choi, and Abdur Chowdhury (Eds.). ACM, 43\u201352. DOI:10.1145\/1458082.1458092"},{"key":"e_1_3_3_78_2","first-page":"2507","volume-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201921)","author":"Rekabsaz Navid","year":"2021","unstructured":"Navid Rekabsaz, Oleg Lesota, Markus Schedl, Jon Brassey, and Carsten Eickhoff. 2021. TripClick: The log files of a large health web search engine. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201921), Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (Eds.). ACM, 2507\u20132513. DOI:10.1145\/3404835.3463242"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000019"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009"},{"key":"e_1_3_3_81_2","first-page":"555","volume-title":"Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201910)","author":"Sanderson Mark","year":"2010","unstructured":"Mark Sanderson, Monica Lestari Paramita, Paul D. Clough, and Evangelos Kanoulas. 2010. Do user preferences and evaluation measures line up? In Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201910), Fabio Crestani, St\u00e9phane Marchand-Maillet, Hsin-Hsi Chen, Efthimis N. Efthimiadis, and Jacques Savoy (Eds.). ACM, 555\u2013562. DOI:10.1145\/1835449.1835542"},{"key":"e_1_3_3_82_2","series-title":"Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of the 12th International Conference of the CLEF Association (CLEF\u201921),","first-page":"394","volume":"12880","author":"Schaer Philipp","year":"2021","unstructured":"Philipp Schaer, Timo Breuer, Leyla Jael Castro, Benjamin Wolff, Johann Schaible, and Narges Tavakolpoursaleh. 2021. Overview of LiLAS 2021\u2014Living labs for academic search. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of the 12th International Conference of the CLEF Association (CLEF\u201921),Lecture Notes in Computer Science, Vol. 12880, K. Sel\u00e7uk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning M\u00fcller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro (Eds.). Springer, 394\u2013418. DOI:10.1007\/978-3-030-85251-1_25"},{"key":"e_1_3_3_83_2","series-title":"Advances in Information Retrieval: Proceedings of the 30th European Conference on IR Research (ECIR\u201908),","first-page":"28","volume":"4956","author":"Scholer Falk","year":"2008","unstructured":"Falk Scholer, Milad Shokouhi, Bodo Billerbeck, and Andrew Turpin. 2008. Using clicks as implicit judgments: Expectations versus observations. In Advances in Information Retrieval: Proceedings of the 30th European Conference on IR Research (ECIR\u201908),Lecture Notes in Computer Science, Vol. 4956, Craig Macdonald, Iadh Ounis, Vassilis Plachouras, Ian Ruthven, and Ryen W. White (Eds.). Springer, 28\u201339. DOI:10.1007\/978-3-540-78646-7_6"},{"key":"e_1_3_3_84_2","series-title":"Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of the 6th International Conference of the CLEF Association (CLEF\u201915),","first-page":"484","volume":"9283","author":"Schuth Anne","year":"2015","unstructured":"Anne Schuth, Krisztian Balog, and Liadh Kelly. 2015. Overview of the living labs for information retrieval evaluation (LL4IR) CLEF Lab 2015. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of the 6th International Conference of the CLEF Association (CLEF\u201915),Lecture Notes in Computer Science, Vol. 9283, Josiane Mothe, Jacques Savoy, Jaap Kamps, Karen Pinel-Sauvagnat, Gareth J. F. Jones, Eric SanJuan, Linda Cappellato, and Nicola Ferro (Eds.). Springer, 484\u2013496. DOI:10.1007\/978-3-319-24027-5_47"},{"key":"e_1_3_3_85_2","first-page":"457","volume-title":"Proceedings of the 9th ACM International Conference on Web Search and Data Mining","author":"Schuth Anne","year":"2016","unstructured":"Anne Schuth, Harrie Oosterhuis, Shimon Whiteson, and Maarten de Rijke. 2016. Multileave gradient descent for fast online learning to rank. In Proceedings of the 9th ACM International Conference on Web Search and Data Mining, Paul N. Bennett, Vanja Josifovski, Jennifer Neville, and Filip Radlinski (Eds.). ACM, 457\u2013466. DOI:10.1145\/2835776.2835804"},{"key":"e_1_3_3_86_2","doi-asserted-by":"crossref","unstructured":"Amanpreet Singh Mike D\u2019Arcy Arman Cohan Doug Downey and Sergey Feldman. 2022. SciRepEval: A multi-format benchmark for scientific document representations. arXiv:2211.13308. Retrieved from https:\/\/arxiv.org\/abs\/2211.13308","DOI":"10.18653\/v1\/2023.emnlp-main.338"},{"key":"e_1_3_3_87_2","first-page":"66","volume-title":"Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201901)","author":"Soboroff Ian","year":"2001","unstructured":"Ian Soboroff, Charles K. Nicholas, and Patrick Cahan. 2001. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201901), W. Bruce Croft, David J. Harper, Donald H. Kraft, and Justin Zobel (Eds.). ACM, 66\u201373. DOI:10.1145\/383952.383961"},{"key":"e_1_3_3_88_2","first-page":"66","volume-title":"Theoretical Issues in Information Retrieval: Proceedings of the 4th International Conference on Information Storage and Retrieval","author":"Tague Jean","year":"1981","unstructured":"Jean Tague and Michael J. Nelson. 1981. Simulation of user judgments in bibliographic retrieval systems. In Theoretical Issues in Information Retrieval: Proceedings of the 4th International Conference on Information Storage and Retrieval, Carolyn J. Crouch (Ed.). ACM, 66\u201371. DOI:10.1145\/511754.511764"},{"key":"e_1_3_3_89_2","first-page":"236","volume-title":"Information Retrieval Research: Proceedings of the Joint ACM\/BCS Symposium in Information Storage and Retrieval","author":"Tague Jean","year":"1980","unstructured":"Jean Tague, Michael J. Nelson, and Harry Wu. 1980. Problems in the simulation of bibliographic retrieval systems. In Information Retrieval Research: Proceedings of the Joint ACM\/BCS Symposium in Information Storage and Retrieval, Robert N. Oddy, Stephen E. Robertson, C. J. van Rijsbergen, and P. W. Williams (Eds.). Butterworths, 236\u2013255."},{"key":"e_1_3_3_90_2","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1145\/2637002.2637032","volume-title":"Proceedings of the 5th Information Interaction in Context Symposium (IIiX\u201914)","author":"Thomas Paul","year":"2014","unstructured":"Paul Thomas, Alistair Moffat, Peter Bailey, and Falk Scholer. 2014. Modeling decision points in user search behavior. In Proceedings of the 5th Information Interaction in Context Symposium (IIiX\u201914), David Elsweiler, Bernd Ludwig, Leif Azzopardi, and Max L. Wilson (Eds.). ACM, 239\u2013242. DOI:10.1145\/2637002.2637032"},{"key":"e_1_3_3_91_2","first-page":"225","volume-title":"Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201901)","author":"Turpin Andrew","year":"2001","unstructured":"Andrew Turpin and William R. Hersh. 2001. Why batch and user evaluations do not give the same results. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201901), W. Bruce Croft, David J. Harper, Donald H. Kraft, and Justin Zobel (Eds.). ACM, 225\u2013231. DOI:10.1145\/383952.383992"},{"key":"e_1_3_3_92_2","first-page":"431","volume-title":"Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201902)","author":"Turpin Andrew","year":"2002","unstructured":"Andrew Turpin and William R. Hersh. 2002. User interface effects in past batch versus user experiments. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201902), Kalervo J\u00e4rvelin, Micheline Beaulieu, Ricardo A. Baeza-Yates, and Sung-Hyon Myaeng (Eds.). ACM, 431\u2013432. DOI:10.1145\/564376.564479"},{"key":"e_1_3_3_93_2","series-title":"Proceedings of the 15th Australasian Database Conference (ADC\u201904),","first-page":"85","volume":"27","author":"Turpin Andrew","year":"2004","unstructured":"Andrew Turpin and William R. Hersh. 2004. Do clarity scores for queries correlate with user performance? In Proceedings of the 15th Australasian Database Conference (ADC\u201904),CRPIT, Vol. 27, Klaus-Dieter Schewe and Hugh E. Williams (Eds.). Australian Computer Society, 85\u201391."},{"key":"e_1_3_3_94_2","first-page":"11","volume-title":"Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201906)","author":"Turpin Andrew","year":"2006","unstructured":"Andrew Turpin and Falk Scholer. 2006. User performance versus precision measures for simple search tasks. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201906), Efthimis N. Efthimiadis, Susan T. Dumais, David Hawking, and Kalervo J\u00e4rvelin (Eds.). ACM, 11\u201318. DOI:10.1145\/1148170.1148176"},{"key":"e_1_3_3_95_2","first-page":"315","volume-title":"Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201998)","author":"Voorhees Ellen M.","year":"1998","unstructured":"Ellen M. Voorhees. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201998), W. Bruce Croft, Alistair Moffat, C. J. van Rijsbergen, Ross Wilkinson, and Justin Zobel (Eds.). ACM, 315\u2013323. DOI:10.1145\/290941.291017"},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_3_3_97_2","first-page":"3","volume-title":"Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201913)","author":"White Ryen","year":"2013","unstructured":"Ryen White. 2013. Beliefs and biases in web search. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201913), Gareth J. F. Jones, Paraic Sheridan, Diane Kelly, Maarten de Rijke, and Tetsuya Sakai (Eds.). ACM, 3\u201312. DOI:10.1145\/2484028.2484053"},{"key":"e_1_3_3_98_2","first-page":"627","volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM\u201918)","author":"Zhang Junqi","year":"2018","unstructured":"Junqi Zhang, Yiqun Liu, Shaoping Ma, and Qi Tian. 2018. Relevance estimation with multiple information sources on search engine result pages. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM\u201918), Alfredo Cuzzocrea, James Allan, Norman W. Paton, Divesh Srivastava, Rakesh Agrawal, Andrei Z. Broder, Mohammed J. Zaki, K. Sel\u00e7uk Candan, Alexandros Labrinidis, Assaf Schuster, and Haixun Wang (Eds.). ACM, 627\u2013636. DOI:10.1145\/3269206.3271673"},{"key":"e_1_3_3_99_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511469"},{"key":"e_1_3_3_100_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1145\/3485447.3511950","volume-title":"Proceedings of the ACM Web Conference (WWW\u201922)","author":"Zhang Junqi","year":"2022","unstructured":"Junqi Zhang, Yiqun Liu, Jiaxin Mao, Xiaohui Xie, Min Zhang, Shaoping Ma, and Qi Tian. 2022. Global or local: Constructing personalized click models for web search. In Proceedings of the ACM Web Conference (WWW\u201922), Fr\u00e9d\u00e9rique Laforest, Rapha\u00ebl Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel M\u00e9dini (Eds.). ACM, 213\u2013223. DOI:10.1145\/3485447.3511950"},{"key":"e_1_3_3_101_2","first-page":"193","volume-title":"Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR\u201917)","author":"Zhang Yinan","year":"2017","unstructured":"Yinan Zhang, Xueqing Liu, and ChengXiang Zhai. 2017. Information retrieval evaluation as search simulation: A general formal framework for IR evaluation. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR\u201917), Jaap Kamps, Evangelos Kanoulas, Maarten de Rijke, Hui Fang, and Emine Yilmaz (Eds.). ACM, 193\u2013200. DOI:10.1145\/3121050.3121070"},{"key":"e_1_3_3_102_2","first-page":"1117","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR\u201918)","author":"Zheng Yukun","year":"2018","unstructured":"Yukun Zheng, Zhen Fan, Yiqun Liu, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. Sogou-QCL: A new dataset with click relevance label. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR\u201918), Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 1117\u20131120. DOI:10.1145\/3209978.3210092"},{"key":"e_1_3_3_103_2","first-page":"307","volume-title":"Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201998)","author":"Zobel Justin","year":"1998","unstructured":"Justin Zobel. 1998. How reliable are the results of large-scale information retrieval experiments?. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201998), W. Bruce Croft, Alistair Moffat, C. J. van Rijsbergen, Ross Wilkinson, and Justin Zobel (Eds.). ACM, 307\u2013314. DOI:10.1145\/290941.291014"}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3623640","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3623640","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:00Z","timestamp":1750178220000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3623640"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,6]]},"references-count":102,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3623640"],"URL":"https:\/\/doi.org\/10.1145\/3623640","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"value":"1936-1955","type":"print"},{"value":"1936-1963","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,6]]},"assertion":[{"value":"2023-03-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}