{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:17:19Z","timestamp":1784179039013,"version":"3.55.0"},"reference-count":210,"publisher":"Emerald","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,6,22]]},"abstract":"<jats:p>Online evaluation is one of the most common approaches to measure the effectiveness of an information retrieval system. It involves fielding the information retrieval system to real users, and observing these users\u2019 interactions in-situ while they engage with the system. This allows actual users with real world information needs to play an important part in assessing retrieval quality. As such, online evaluation complements the common alternative offline evaluation approaches which may provide more easily interpretable outcomes, yet are often less realistic when measuring of quality and actual user experience.<\/jats:p>\n                  <jats:p>In this survey, we provide an overview of online evaluation techniques for information retrieval. We show how online evaluation is used for controlled experiments, segmenting them into experiment designs that allow absolute or relative quality assessments. Our presentation of different metrics further partitions online evaluation based on different sized experimental units commonly of interest: documents, lists and sessions. Additionally, we include an extensive discussion of recent work on data re-use, and experiment estimation based on historical data.<\/jats:p>\n                  <jats:p>A substantial part of this work focuses on practical issues: How to run evaluations in practice, how to select experimental parameters, how to take into account ethical considerations inherent in online evaluations, and limitations. While most published work on online experimentation today is at large scale in systems with millions of users, we also emphasize that the same techniques can be applied at small scale. To this end, we emphasize recent work that makes it easier to use at smaller scales and encourage studying real-world information seeking in a wide range of scenarios. Finally, we present a summary of the most recent work in the area, and describe open problems, as well as postulating future directions.<\/jats:p>","DOI":"10.1561\/1500000051","type":"journal-article","created":{"date-parts":[[2016,6,22]],"date-time":"2016-06-22T16:41:10Z","timestamp":1466613670000},"page":"1-117","source":"Crossref","is-referenced-by-count":84,"title":["Online Evaluation for Information Retrieval"],"prefix":"10.1108","volume":"10","author":[{"given":"Katja","family":"Hofmann","sequence":"first","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lihong","family":"Li","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Filip","family":"Radlinski","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","published-online":{"date-parts":[[2016,6,22]]},"reference":[{"key":"2026040314520145900_ref001","author":"Agarwal","year":"2016"},{"key":"2026040314520145900_ref002","first-page":"132","article-title":"Click shaping to optimize multiple objectives","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Agarwal","year":"2011"},{"key":"2026040314520145900_ref003","first-page":"485","article-title":"Personalized click shaping through Lagrangian duality for online recommendation","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Agarwal","year":"2012"},{"key":"2026040314520145900_ref004","doi-asserted-by":"crossref","DOI":"10.1145\/2009916.2009965","article-title":"Find it if you can: A game for modeling different types of web search success using interaction data","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Ageev","year":"2011"},{"key":"2026040314520145900_ref005","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1145\/1498759.1498824","article-title":"Generating labels from clicks","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Agrawal","year":"2009"},{"key":"2026040314520145900_ref006","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/s10791-012-9204-1","article-title":"Implementing crowdsourcing-based relevance experimentation: An industrial perspective","volume":"16","author":"Alonso","year":"2013","journal-title":"Information Retrieval"},{"key":"2026040314520145900_ref007","first-page":"16","article-title":"Can we get rid of TREC assessors? using Mechanical Turk for relevance assessment","volume":"15","author":"Alonso","year":"2009","journal-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation"},{"key":"2026040314520145900_ref008","volume-title":"Exploratory analysis of spatial and temporal data: A systematic approach","author":"Andrienko","year":"2006"},{"key":"2026040314520145900_ref009","first-page":"723","article-title":"Search engine evaluation based on search engine switching prediction","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Arkhipova","year":"2015"},{"key":"2026040314520145900_ref010","first-page":"723","article-title":"Search engine evaluation based on search engine switching prediction","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Arkhipova","year":"2015"},{"key":"2026040314520145900_ref011","unstructured":"Arthur\n              Asuncion\n             and David J.Newman. UCI machine learning repository, 2007. http:\/\/www.ics.uci.edu\/\u223cmlearn\/MLRepository.html."},{"issue":"1","key":"2026040314520145900_ref012","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1137\/S0097539701398375","article-title":"The nonstochastic multiarmed bandit problem","volume":"32","author":"Auer","year":"2002","journal-title":"SIAM Journal on Computing"},{"key":"2026040314520145900_ref013","first-page":"3","article-title":"Modelling interaction with economic models of search","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Azzopardi","year":"2014"},{"key":"2026040314520145900_ref014","first-page":"455","article-title":"Building simulated queries for known-item topics: An analysis using six european languages","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Azzopardi","year":"2007"},{"issue":"4","key":"2026040314520145900_ref015","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1016\/j.jml.2007.12.005","article-title":"Mixed-effects modeling with crossed random effects for subjects and items","volume":"59","author":"Baayen","year":"2008","journal-title":"Journal of Memory and Language"},{"key":"2026040314520145900_ref016","first-page":"75","article-title":"Optimising trade-offs among stakeholders in ad auctions","volume-title":"Proceedings of the ACM Conference on Economics and Computation (EC-14)","author":"Bachrach","year":"2014"},{"key":"2026040314520145900_ref017","doi-asserted-by":"crossref","DOI":"10.1145\/2736277.2741082","article-title":"Design and analysis of benchmarking experiments for distributed internet services","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Bakshy","year":"2015"},{"key":"2026040314520145900_ref018","first-page":"283","article-title":"Designing and deploying online field experiments","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Bakshy","year":"2014"},{"key":"2026040314520145900_ref019","doi-asserted-by":"crossref","DOI":"10.1145\/2661829.2661962","article-title":"Head first: Living labs for ad-hoc search evaluation","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Balog","year":"2014"},{"key":"2026040314520145900_ref020","article-title":"Fitting linear mixed-effects models using lme4","volume-title":"arXiv preprint arXiv:1406.5823","author":"Bates","year":"2014"},{"key":"2026040314520145900_ref021","doi-asserted-by":"crossref","first-page":"1769","DOI":"10.1145\/2623330.2623344","article-title":"Up next: Retrieval methods for large scale related video suggestion","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Bendersky","year":"2014"},{"key":"2026040314520145900_ref022","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1145\/1557019.1557040","article-title":"The offset tree for learning with partial labels","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Beygelzimer","year":"2009"},{"issue":"4","key":"2026040314520145900_ref023","doi-asserted-by":"crossref","first-page":"71","DOI":"10.4018\/jmhci.2011100105","article-title":"My app is an experiment: Experience from user studies in mobile app stores","volume":"3","author":"Boll","year":"2011","journal-title":"International Journal of Mobile Human Computer Interaction (IJMHCI)"},{"key":"2026040314520145900_ref024","first-page":"3207","article-title":"Counterfactual reasoning and learning systems: The example of computational advertising","volume":"14","author":"Bottou","year":"2013","journal-title":"Journal of Machine Learning Research (JMLR)"},{"key":"2026040314520145900_ref025","first-page":"1","article-title":"A machine learning architecture for optimizing Web search engines","volume-title":"AAAI Workshop on Internet Based Information Systems","author":"Boyan","year":"1996"},{"issue":"1","key":"2026040314520145900_ref026","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/2200000024","article-title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","volume":"5","author":"Bubeck","year":"2012","journal-title":"Foundations and Trends in Machine Learning"},{"key":"2026040314520145900_ref027","article-title":"A survey of online experiment design with the stochastic multi-armed bandit","volume-title":"arXiv preprint arXiv:1510.00757","author":"Burtini","year":"2015"},{"key":"2026040314520145900_ref028","first-page":"42","article-title":"The good, the bad, and the random: An eye-tracking study of ad quality in web search","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Buscher","year":"2010"},{"key":"2026040314520145900_ref029","volume-title":"Experimental and Quasi-Experimental Designs for Research","author":"Campbell","year":"1966"},{"key":"2026040314520145900_ref030","doi-asserted-by":"crossref","DOI":"10.1145\/2499178.2499204","article-title":"Statistical significance testing in information retrieval: Theory and practice","volume-title":"Proceedings of the International Conference on The Theory of Information Retrieval (ICTIR)","author":"Carterette","year":"2013"},{"key":"2026040314520145900_ref031","first-page":"217","article-title":"Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS)","author":"Carterette","year":"2007"},{"key":"2026040314520145900_ref032","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1145\/1718487.1718516","article-title":"Measuring the reusability of test collections","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Carterette","year":"2010"},{"key":"2026040314520145900_ref033","volume-title":"Statistical Inference","author":"Casella","year":"2001","edition":"2nd"},{"key":"2026040314520145900_ref034","first-page":"1163","article-title":"On correlation of absence time and search effectiveness","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Chakraborty","year":"2014"},{"key":"2026040314520145900_ref035","first-page":"1","article-title":"Yahoo! learning to rank challenge overview","volume":"14","author":"Chapelle","year":"2011","journal-title":"Journal of Machine Learning Research - Proceedings Track"},{"key":"2026040314520145900_ref036","first-page":"1","article-title":"A dynamic bayesian network click model for web search ranking","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Chapelle","year":"2009"},{"issue":"1","key":"2026040314520145900_ref037","first-page":"6:1","article-title":"Large-scale validation and analysis of interleaved search evaluation","volume":"30","author":"Chapelle","year":"2012","journal-title":"Transactions on Information System (TOIS)"},{"key":"2026040314520145900_ref038","first-page":"1581","article-title":"Does vertical bring more satisfaction?: Predicting search satisfaction in a heterogeneous environment","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Chen","year":"2015"},{"key":"2026040314520145900_ref039","first-page":"421","article-title":"Automatic evaluation of world wide web search services","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Chowdhury","year":"2002"},{"key":"2026040314520145900_ref040","doi-asserted-by":"crossref","DOI":"10.1145\/2505515.2505698","article-title":"Evaluating Aggregated Search Using Interleaving","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Chuklin","year":"2013"},{"key":"2026040314520145900_ref041","first-page":"493","article-title":"Click modelbased information retrieval metrics","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Chuklin","year":"2013"},{"key":"2026040314520145900_ref042","article-title":"A comparative analysis of interleaving methods for aggregated search","volume-title":"Transactions on Information System (TOIS)","author":"Chuklin","year":"2014"},{"key":"2026040314520145900_ref043","volume-title":"Click Models for Web Search. Synthesis Lectures on Information Concepts, Retrieval, and Services","author":"Chuklin","year":"2015"},{"key":"2026040314520145900_ref044","first-page":"135","article-title":"The influence of caption features on clickthrough patterns in web search","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Clarke","year":"2007"},{"issue":"6","key":"2026040314520145900_ref045","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1108\/eb050097","article-title":"The Cranfield tests on index language devices","volume":"19","author":"Cleverdon","year":"1967","journal-title":"Aslib Proceedings"},{"key":"2026040314520145900_ref046","first-page":"87","article-title":"An experimental comparison of click position-bias models","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Craswell","year":"2008"},{"key":"2026040314520145900_ref047","first-page":"923","article-title":"Objective Bayesian two sample hypothesis testing for online controlled experiments","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Deng","year":"2015"},{"key":"2026040314520145900_ref048","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1145\/2433396.2433413","article-title":"Improving the sensitivity of online controlled experiments by utilizing pre-experiment data","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Deng","year":"2013"},{"key":"2026040314520145900_ref049","first-page":"248","article-title":"ImageNet: A large-scale hierarchical image database","volume-title":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Deng","year":"2009"},{"key":"2026040314520145900_ref050","first-page":"1451","article-title":"Robust models of mouse movement on dynamic web search results pages","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Diaz","year":"2013"},{"key":"2026040314520145900_ref051","doi-asserted-by":"crossref","first-page":"1025","DOI":"10.1145\/2396761.2398399","article-title":"Leaving so soon? Understanding and predicting web search abandonment rationales","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Diriye","year":"2012"},{"key":"2026040314520145900_ref052","doi-asserted-by":"crossref","first-page":"2129","DOI":"10.1145\/2063576.2063907","article-title":"User action interpretation for personalized content optimization in recommender systems","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Dong","year":"2011"},{"key":"2026040314520145900_ref053","first-page":"256","article-title":"Future user engagement prediction and its application to improve the sensitivity of online experiments","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Drutsa","year":"2015"},{"key":"2026040314520145900_ref054","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1145\/2684822.2685318","article-title":"Engagement periodicity in search engine usage: Analysis and its application to search quality evaluation","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Drutsa","year":"2015"},{"key":"2026040314520145900_ref055","first-page":"763","article-title":"Practical aspects of sensitivity in online experimentation with user engagement metrics","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Drutsa","year":"2015"},{"key":"2026040314520145900_ref056","first-page":"1097","article-title":"Doubly robust policy evaluation and learning","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Dud\u00edk","year":"2011"},{"key":"2026040314520145900_ref057","first-page":"247","article-title":"Sampleefficient nonstationary-policy evaluation for contextual bandits","volume-title":"Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)","author":"Dud\u00edk","year":"2012"},{"key":"2026040314520145900_ref058","first-page":"563","article-title":"Contextual dueling bandits","volume-title":"Proceedings of the Annual Conference on Learning Theory (COLT)","author":"Dud\u00edk","year":"2015"},{"key":"2026040314520145900_ref059","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1145\/2433396.2433418","article-title":"Absence time and user engagement: Evaluating ranking functions","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Dupret","year":"2013"},{"key":"2026040314520145900_ref060","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1145\/1718487.1718510","article-title":"A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Dupret","year":"2010"},{"key":"2026040314520145900_ref061","first-page":"331","article-title":"A user browsing model to predict search engine click data from past observations","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Dupret","year":"2008"},{"key":"2026040314520145900_ref062","volume-title":"Chapman & Hall\/CRC Monographs on Statistics & Applied Probability","author":"Efron","year":"1993"},{"key":"2026040314520145900_ref063","first-page":"871","article-title":"Quality through flow and immersion: gamifying crowdsourced relevance assessments","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Eickhoff","year":"2012"},{"key":"2026040314520145900_ref064","first-page":"34","article-title":"Predicting searcher frustration","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Feild","year":"2010"},{"key":"2026040314520145900_ref065","first-page":"375","article-title":"Crowdlogging: distributed, private, and anonymous search logging","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Feild","year":"2011"},{"issue":"2","key":"2026040314520145900_ref066","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1145\/1059981.1059982","article-title":"Evaluating implicit measures to improve Web search","volume":"23","author":"Fox","year":"2005","journal-title":"Transactions on Information System (TOIS)"},{"key":"2026040314520145900_ref067","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/978-3-642-20291-9_13","article-title":"Web user profiling on proxy logs and its evaluation in personalization","volume-title":"Proceedings of the Asia-Pacific web conference on Web technologies and applications (APWeb)","author":"Fujimoto","year":"2011"},{"key":"2026040314520145900_ref068","volume-title":"Analytical Methods for Social Research","author":"Gelman","year":"2006"},{"issue":"1","key":"2026040314520145900_ref069","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1214\/009053604000001048","article-title":"Analysis of variance \u2013 why it is more important than ever","volume":"33","author":"Gelman","year":"2005","journal-title":"The Annals of Statistics"},{"key":"2026040314520145900_ref070","first-page":"478","article-title":"Eye-tracking analysis of user behavior in www search","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Granka","year":"2004"},{"key":"2026040314520145900_ref071","first-page":"273","article-title":"Bayesian ranker comparison based on historical user interactions","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Grotov","year":"2015"},{"key":"2026040314520145900_ref072","first-page":"407","article-title":"What are you looking for? An eye-tracking study of information usage in web search","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI)","author":"Guan","year":"2007"},{"key":"2026040314520145900_ref073","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1145\/1240624.1240691","article-title":"An eye tracking study of the effect of target rank on web search","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI)","author":"Guan","year":"2007"},{"key":"2026040314520145900_ref074","first-page":"399","article-title":"Network A\/B testing: From sampling to estimation","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Gui","year":"2015"},{"key":"2026040314520145900_ref075","first-page":"11","article-title":"Click chain model in Web search","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Guo","year":"2009"},{"key":"2026040314520145900_ref076","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1145\/1498759.1498818","article-title":"Efficient multiple-click models in Web search","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Guo","year":"2009"},{"key":"2026040314520145900_ref077","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1145\/2187836.2187914","article-title":"Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Guo","year":"2012"},{"key":"2026040314520145900_ref078","first-page":"1229","article-title":"Detecting success in mobile search from interaction","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Guo","year":"2011"},{"key":"2026040314520145900_ref079","first-page":"153","article-title":"Mining touch interaction data on mobile devices to predict web search result relevance","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Guo","year":"2013"},{"key":"2026040314520145900_ref080","first-page":"1821","article-title":"Towards estimating web search result relevance from touch interactions on mobile devices","volume-title":"CHI\u201913 Extended Abstracts on Human Factors in Computing Systems","author":"Guo","year":"2013"},{"issue":"2","key":"2026040314520145900_ref081","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v033.i02","article-title":"Mcmc methods for multi-response generalized linear mixed models: the mcmcglmm r package","volume":"33","author":"Hadfield","year":"2010","journal-title":"Journal of Statistical Software"},{"key":"2026040314520145900_ref082","volume-title":"The Morgan Kaufmann Series in Data Management Systems","author":"Han","year":"2011"},{"key":"2026040314520145900_ref083","article-title":"Demonstration of improved search result relevancy using real-time implicit relevance feedback","volume-title":"Understanding the User \u2013 Workshop in conjunction with SIGIR","author":"Hardtke","year":"2009"},{"key":"2026040314520145900_ref084","first-page":"2009","article-title":"Personalized models of search satisfaction","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Hassan","year":"2013"},{"key":"2026040314520145900_ref085","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1145\/1718487.1718515","article-title":"Beyond dcg: user behavior as a predictor of a successful search","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Hassan","year":"2010"},{"key":"2026040314520145900_ref086","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1145\/2063576.2063599","article-title":"A task level user satisfaction metric and its application on improving relevance estimation","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Hassan","year":"2011"},{"key":"2026040314520145900_ref087","first-page":"2019","article-title":"Beyond clicks: Query reformulation as a predictor of search satisfaction","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Hassan","year":"2013"},{"key":"2026040314520145900_ref088","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The Elements of Statistical Learning: Data Mining, Inference, and Prediction","author":"Hastie","year":"2009","edition":"2nd"},{"key":"2026040314520145900_ref089","first-page":"17","article-title":"Do batch and user evaluations give the same results?","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Hersh","year":"2000"},{"issue":"4","key":"2026040314520145900_ref090","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1111\/1468-0262.00442","article-title":"Efficient estimation of average treatment effects using the estimated propensity score","volume":"71","author":"Hirano","year":"2003","journal-title":"Econometrica"},{"key":"2026040314520145900_ref091","first-page":"761","article-title":"Comparing click-through data to purchase decisions for retrieval evaluation","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Hofmann","year":"2010"},{"key":"2026040314520145900_ref092","first-page":"251","article-title":"Balancing exploration and exploitation in learning to rank online","volume-title":"Proceedings of the European Conference on Information Retrieval (ECIR), volume 6611 of Lecture Notes in Computer Science","author":"Hofmann","year":"2011"},{"key":"2026040314520145900_ref093","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1145\/2063576.2063618","article-title":"A probabilistic method for inferring preferences from clicks","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Hofmann","year":"2011"},{"key":"2026040314520145900_ref094","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1145\/2396761.2396780","article-title":"On caption bias in interleaving experiments","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Hofmann","year":"2012"},{"key":"2026040314520145900_ref095","doi-asserted-by":"crossref","first-page":"1779","DOI":"10.1145\/2396761.2398516","article-title":"Estimating interleaved comparison outcomes from historical click data","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Hofmann","year":"2012"},{"key":"2026040314520145900_ref096","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1145\/2433396.2433419","article-title":"Reusing historical interaction data for faster online learning to rank for IR","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Hofmann","year":"2013"},{"issue":"1","key":"2026040314520145900_ref097","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1007\/s10791-012-9197-9","article-title":"Balancing exploration and exploitation in listwise and pairwise online learning to rank for information retrieval","volume":"16","author":"Hofmann","year":"2013","journal-title":"Information Retrieval"},{"issue":"4","key":"2026040314520145900_ref098","article-title":"Fidelity, soundness, and efficiency of interleaved comparison methods","volume":"31","author":"Hofmann","year":"2013","journal-title":"Transactions on Information System (TOIS)"},{"key":"2026040314520145900_ref099","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1145\/2661829.2661922","article-title":"An eye-tracking study of user interactions with query auto completion","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Hofmann","year":"2014"},{"key":"2026040314520145900_ref100","first-page":"1849","article-title":"Focusing on the long-term: It\u2019s good for users and business","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Hohnhold","year":"2015"},{"issue":"6","key":"2026040314520145900_ref101","doi-asserted-by":"crossref","first-page":"945","DOI":"10.1080\/01621459.1986.10478354","article-title":"Statistics and causal inference","volume":"81","author":"Holland","year":"1986","journal-title":"Journal of the American Statistical Association"},{"issue":"5","key":"2026040314520145900_ref102","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1109\/TDSC.2014.2369034","article-title":"Collaborative search log sanitization: Toward differential privacy and boosted utility","volume":"12","author":"Hong","year":"2015","journal-title":"IEEE Transactions on Dependable and Secure Computing"},{"issue":"2","key":"2026040314520145900_ref103","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1198\/106186008X320456","article-title":"Truncated importance sampling","volume":"17","author":"Ionides","year":"2008","journal-title":"Journal of Computational and Graphical Statistics"},{"issue":"3","key":"2026040314520145900_ref104","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1016\/j.lisr.2006.06.005","article-title":"Search log analysis: What it is, what\u2019s been done, how to do it","volume":"28","author":"Jansen","year":"2006","journal-title":"Library & Information Science Research"},{"issue":"4","key":"2026040314520145900_ref105","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/582415.582418","article-title":"Cumulated gain-based evaluation of ir techniques","volume":"20","author":"J\u00e4rvelin","year":"2002","journal-title":"Transactions on Information System (TOIS)"},{"key":"2026040314520145900_ref106","first-page":"506","article-title":"Automatic online evaluation of intelligent personal assistants","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Jiang","year":"2015"},{"key":"2026040314520145900_ref107","first-page":"506","article-title":"Automatic online evaluation of intelligent assistants","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Jiang","year":"2015"},{"key":"2026040314520145900_ref108","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1145\/2684822.2685319","article-title":"Understanding and predicting graded search satisfaction","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Jiang","year":"2015"},{"key":"2026040314520145900_ref109","article-title":"Doubly robust off-policy evaluation for reinforcement learning","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Jiang","year":"2016"},{"key":"2026040314520145900_ref110","first-page":"133","article-title":"Optimizing search engines using clickthrough data","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Joachims","year":"2002"},{"issue":"2","key":"2026040314520145900_ref111","article-title":"Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search","volume":"25","author":"Joachims","year":"2007","journal-title":"Transactions on Information System (TOIS)"},{"key":"2026040314520145900_ref112","article-title":"Always valid inference: Bringing sequential analysis to A\/B testing","volume-title":"arXiv Preprint arXiv:1512.04922v1 [math.ST]","author":"Johari","year":"2015"},{"key":"2026040314520145900_ref113","first-page":"205","article-title":"Crowdsourcing for book search evaluation: Impact of hit design on comparative system ranking","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Kazai","year":"2011"},{"issue":"1\u20132","key":"2026040314520145900_ref114","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000012","article-title":"Methods for evaluating interactive information retrieval systems with users","volume":"3","author":"Kelly","year":"2009","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"2026040314520145900_ref115","first-page":"183","article-title":"How many results per page? a study of SERP size, search behavior and user experience","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Kelly","year":"2015"},{"key":"2026040314520145900_ref116","doi-asserted-by":"crossref","first-page":"1531","DOI":"10.1145\/1978942.1979166","article-title":"An examination of two delivery modes for interactive search system experiments","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI)","author":"Kelly","year":"2011"},{"issue":"2","key":"2026040314520145900_ref117","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1145\/959258.959260","article-title":"Implicit feedback for inferring user preference: a bibliography","volume":"37","author":"Kelly","year":"2003","journal-title":"SIGIR Forum"},{"key":"2026040314520145900_ref118","article-title":"Choices and constraints: Research goals and approaches in information retrieval","volume-title":"Tutorial at Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)","author":"Kelly","year":"2014"},{"key":"2026040314520145900_ref119","first-page":"679","article-title":"Using historical click data to increase interleaving sensitivity","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Kharitonov","year":"2013"},{"key":"2026040314520145900_ref120","first-page":"453","article-title":"Optimised scheduling of online experiments","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Kharitonov","year":"2015"},{"key":"2026040314520145900_ref121","first-page":"773","article-title":"Generalized team draft interleaving","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Kharitonov","year":"2015"},{"key":"2026040314520145900_ref122","first-page":"473","article-title":"Sequential testing for early stopping of online experiments","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Kharitonov","year":"2015"},{"key":"2026040314520145900_ref123","first-page":"193","article-title":"Modeling dwell time to predict click-level satisfaction","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Kim","year":"2014"},{"key":"2026040314520145900_ref124","first-page":"895","article-title":"Comparing client and server dwell time estimates for click-level satisfaction prediction","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Kim","year":"2014"},{"key":"2026040314520145900_ref125","doi-asserted-by":"crossref","DOI":"10.1145\/2911451.2911521","article-title":"Predicting user satisfaction with intelligent assistants","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Kiseleva","year":"2016"},{"key":"2026040314520145900_ref126","volume-title":"Data Stream Management: Processing High-Speed Data Streams","author":"Kleinberg","year":"2004"},{"issue":"1","key":"2026040314520145900_ref127","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1007\/s10618-008-0114-1","article-title":"Controlled experiments on the web: Survey and practical guide","volume":"18","author":"Kohavi","year":"2009","journal-title":"Data Mining and Knowledge Discovery"},{"key":"2026040314520145900_ref128","first-page":"786","article-title":"Trustworthy online controlled experiments: Five puzzling outcomes explained","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Kohavi","year":"2012"},{"key":"2026040314520145900_ref129","first-page":"1168","article-title":"Online controlled experiments at large scale","volume-title":"Proceedings of the ACM SIGKDD International Conference on Know ledge Discovery and Data Mining (KDD)","author":"Kohavi","year":"2013"},{"key":"2026040314520145900_ref130","doi-asserted-by":"crossref","first-page":"1857","DOI":"10.1145\/2623330.2623341","article-title":"Seven rules of thumb for web site experimenters","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Kohavi","year":"2014"},{"issue":"24","key":"2026040314520145900_ref131","doi-asserted-by":"crossref","first-page":"8788","DOI":"10.1073\/pnas.1320040111","article-title":"Experimental evidence of massive-scale emotional contagion through social networks","volume":"111","author":"Kramer","year":"2014","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2026040314520145900_ref132","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff","year":"2012"},{"key":"2026040314520145900_ref133","first-page":"113","article-title":"Towards better measurement of attention and satisfaction in mobile search","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Lagun","year":"2014"},{"key":"2026040314520145900_ref134","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1145\/1390156.1390223","article-title":"Exploration scavenging","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Langford","year":"2008"},{"key":"2026040314520145900_ref135","volume-title":"Design and Analysis of Experiments with R. Chapman & Hall\/CRC Texts in Statistical Science","author":"Lawson","year":"2014"},{"key":"2026040314520145900_ref136","first-page":"43","article-title":"Good abandonment in mobile and PC Internet search","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Li","year":"2009"},{"key":"2026040314520145900_ref137","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1145\/1772690.1772758","article-title":"A contextual-bandit approach to personalized news article recommendation","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Li","year":"2010"},{"key":"2026040314520145900_ref138","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1145\/1935826.1935878","article-title":"Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Li","year":"2011"},{"key":"2026040314520145900_ref139","first-page":"929","article-title":"Counterfactual analysis of click metrics for search engine optimization: A case study","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Li","year":"2015"},{"key":"2026040314520145900_ref140","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1145\/2684822.2685311","article-title":"Toward predicting the outcome of an A\/B experiment for search relevance","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Li","year":"2015"},{"key":"2026040314520145900_ref141","first-page":"608","article-title":"Toward minimax off-policy value estimation","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)","author":"Li","year":"2015"},{"key":"2026040314520145900_ref142","volume-title":"Springer Series in Statistics","author":"Liu","year":"2001"},{"issue":"1","key":"2026040314520145900_ref143","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3758\/s13428-011-0124-6","article-title":"Conducting behavioral research on Amazon\u2019s Mechanical Turk","volume":"44","author":"Mason","year":"2012","journal-title":"Behavior Research Methods"},{"key":"2026040314520145900_ref144","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1145\/1935826.1935840","article-title":"Personalizing web search using long term browsing history","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Matthijs","year":"2011"},{"key":"2026040314520145900_ref145","volume-title":"Infer.NET 2.6","author":"Minka","year":"2014"},{"issue":"4","key":"2026040314520145900_ref146","article-title":"An online learning framework for refining recency search results with user click feedback","volume":"30","author":"Moon","year":"2012","journal-title":"Transactions on Information System (TOIS)"},{"key":"2026040314520145900_ref147","first-page":"272","article-title":"Information filtering based on user behavior analysis and best match text retrieval","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Morita","year":"1994"},{"issue":"456","key":"2026040314520145900_ref148","doi-asserted-by":"crossref","first-page":"1410","DOI":"10.1198\/016214501753382327","article-title":"Marginal mean models for dynamic regimes","volume":"96","author":"Murphy","year":"2001","journal-title":"Journal of the American Statistical Association"},{"issue":"3","key":"2026040314520145900_ref149","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1016\/j.ipm.2011.01.004","article-title":"User k-anonymity for privacy preserving data mining of query logs","volume":"48","author":"Navarro-Arribas","year":"2012","journal-title":"Information Processing & Management"},{"issue":"2","key":"2026040314520145900_ref150","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1037\/1089-2680.2.2.175","article-title":"Confirmation bias: A ubiquitous phenomenon in many guises","volume":"2","author":"Nickerson","year":"1998","journal-title":"Review of General Psychology"},{"key":"2026040314520145900_ref151","first-page":"172","article-title":"Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Nicol","year":"2014"},{"key":"2026040314520145900_ref152","first-page":"845","article-title":"Extreme states distribution decomposition method for search engine online evaluation","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Nikolaev","year":"2015"},{"key":"2026040314520145900_ref153","first-page":"1551","article-title":"Struggling and success in web search","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Odijk","year":"2015"},{"key":"2026040314520145900_ref154","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1145\/1963405.1963463","article-title":"Evaluating new search engine configurations with pre-existing judgments and clicks","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Ozertem","year":"2011"},{"key":"2026040314520145900_ref155","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511803161","volume-title":"Causality: Models, Reasoning, and Inference","author":"Pearl","year":"2009","edition":"2nd"},{"key":"2026040314520145900_ref156","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1145\/1963405.1963419","article-title":"Model characterization curves for federated search using click-logs: Predicting user engagement metrics for the span of feasible operating points","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Ponnuswami","year":"2011"},{"key":"2026040314520145900_ref157","first-page":"759","article-title":"Eligibility traces for off-policy policy evaluation","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Precup","year":"2000"},{"issue":"4","key":"2026040314520145900_ref158","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1007\/s10791-009-9123-y","article-title":"LETOR: A benchmark collection for research on learning to rank for information retrieval","volume":"13","author":"Qin","year":"2010","journal-title":"Information Retrieval"},{"key":"2026040314520145900_ref159","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9780262170055.001.0001","volume-title":"Dataset Shift in Machine Learning","author":"Qui\u00f1onero-Candela","year":"2008"},{"key":"2026040314520145900_ref160","first-page":"667","article-title":"Comparing the sensitivity of information retrieval metrics","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Radlinski","year":"2010"},{"key":"2026040314520145900_ref161","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1145\/2433396.2433429","article-title":"Optimized Interleaving for Online Retrieval Evaluation","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Radlinski","year":"2013"},{"key":"2026040314520145900_ref162","first-page":"1406","article-title":"Minimally invasive randomization for collecting unbiased preferences from clickthrough logs","volume-title":"Proceedings of the National Conference on Artificial Intelligence (AAAI)","author":"Radlinski","year":"2006"},{"key":"2026040314520145900_ref163","first-page":"403","article-title":"Optimizing relevance and revenue in ad search: A query substitution approach","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Radlinski","year":"2008"},{"key":"2026040314520145900_ref164","doi-asserted-by":"crossref","first-page":"784","DOI":"10.1145\/1390156.1390255","article-title":"Learning diverse rankings with multi-armed bandits","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Radlinski","year":"2008"},{"key":"2026040314520145900_ref165","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1145\/1458082.1458092","article-title":"How does clickthrough data reflect retrieval quality?","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Radlinski","year":"2008"},{"issue":"4","key":"2026040314520145900_ref166","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1093\/biomet\/82.4.805","article-title":"Semiparametric regression estimation in the presence of dependent censoring","volume":"82","author":"Rotnitzky","year":"1995","journal-title":"Biometrika"},{"issue":"4","key":"2026040314520145900_ref167","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1561\/1500000009","article-title":"Test Collection Based Evaluation of Information Retrieval Systems","volume":"4","author":"","year":"2010","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"2026040314520145900_ref168","first-page":"28","article-title":"Using Clicks as Implicit Judgments: Expectations Versus Observations","volume-title":"Proceedings of the European Conference on Information Retrieval (ECIR)","author":"Scholer","year":"2008"},{"key":"2026040314520145900_ref169","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1145\/2661829.2661952","article-title":"Multileaved comparisons for fast online evaluation","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Schuth","year":"2014"},{"key":"2026040314520145900_ref170","first-page":"955","article-title":"Probabilistic multileave for online retrieval evaluation","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Schuth","year":"2015"},{"key":"2026040314520145900_ref171","first-page":"463","article-title":"Predicting search satisfaction metrics with interleaved comparisons","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Schuth","year":"2015"},{"key":"2026040314520145900_ref172","volume-title":"Experimental and quasi-experimental designs for generalized causal inference","author":"Shadish","year":"2002"},{"issue":"1\u20132","key":"2026040314520145900_ref173","first-page":"1","article-title":"Mining query logs: Turning search usage data into knowledge","volume":"4","author":"Silvestri","year":"2010","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"2026040314520145900_ref174","first-page":"238","article-title":"The interpretation of interaction in contingency tables","volume-title":"Journal of the Royal Statistical Society. Series B (Methodological)","author":"Simpson","year":"1951"},{"issue":"1","key":"2026040314520145900_ref175","first-page":"399","article-title":"Ranked bandits in metric spaces: Learning diverse rankings over large document collections","volume":"14","author":"Slivkins","year":"2013","journal-title":"Journal of Machine Learning Research (JMLR)"},{"key":"2026040314520145900_ref176","first-page":"1201","article-title":"Exploring and exploiting user search behavior on mobile and tablet devices to improve search relevance","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Song","year":"2013"},{"key":"2026040314520145900_ref177","first-page":"1213","article-title":"Evaluating and predicting user engagement change with degraded search relevance","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Song","year":"2013"},{"key":"2026040314520145900_ref178","first-page":"2217","article-title":"Learning from logged implicit exploration data","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS)","author":"Strehl","year":"2011"},{"key":"2026040314520145900_ref179","first-page":"814","article-title":"Counterfactual risk minimization: Learning from logged bandit feedback","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Swaminathan","year":"2015"},{"key":"2026040314520145900_ref180","first-page":"3231","article-title":"The self-normalized estimator for counterfactual learning","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS)","author":"Swaminathan","year":"2015"},{"key":"2026040314520145900_ref181","volume-title":"Off-policy evaluation for slate recommendation","author":"Swaminathan","year":"2016"},{"key":"2026040314520145900_ref182","volume-title":"Using Multivariate Statistics: Pearson New International Edition","author":"Tabachnick","year":"2013"},{"key":"2026040314520145900_ref183","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1145\/1835804.1835810","article-title":"Overlapping experiment infrastructure: More, better, faster experimentation","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Tang","year":"2010"},{"key":"2026040314520145900_ref184","first-page":"1587","article-title":"Automatic ad format selection via contextual bandits","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Tang","year":"2013"},{"key":"2026040314520145900_ref185","article-title":"Data-efficient off-policy policy evaluation for reinforcement learning","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Thomas","year":"2016"},{"key":"2026040314520145900_ref186","first-page":"2380","article-title":"High confidence policy improvement","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Thomas","year":"2015"},{"key":"2026040314520145900_ref187","volume-title":"The visual display of quantitative information","author":"Tufte","year":"1983"},{"key":"2026040314520145900_ref188","first-page":"11","article-title":"User performance versus precision measures for simple search tasks","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Turpin","year":"2006"},{"key":"2026040314520145900_ref189","first-page":"225","article-title":"Why batch and user evaluations do not give the same results","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Turpin","year":"2001"},{"key":"2026040314520145900_ref190","first-page":"329","article-title":"Graph cluster randomization: Network exposure to multiple universes","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Ugander","year":"2013"},{"key":"2026040314520145900_ref191","doi-asserted-by":"crossref","DOI":"10.1201\/b18305","volume-title":"Graphical Data Analysis with R","author":"Unwin","year":"2015"},{"key":"2026040314520145900_ref192","volume-title":"Statistical Rules of Thumb","author":"van Belle","year":"2008"},{"key":"2026040314520145900_ref193","first-page":"1119","article-title":"The effect of sampling strategy on inferred measures","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Voorhees","year":"2014"},{"key":"2026040314520145900_ref194","volume-title":"Digital Libraries and Electronic Publishing","author":"Voorhees","year":"2005"},{"issue":"12","key":"2026040314520145900_ref195","doi-asserted-by":"crossref","first-page":"1940","DOI":"10.1109\/JPROC.2014.2363674","article-title":"Design of randomized experiments in networks","volume":"102","author":"Walker","year":"2014","journal-title":"Proceedings of the IEEE"},{"key":"2026040314520145900_ref196","first-page":"123","article-title":"Modeling action-level satisfaction for search task satisfaction prediction","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Wang","year":"2014"},{"key":"2026040314520145900_ref197","doi-asserted-by":"crossref","first-page":"1355","DOI":"10.1145\/1557019.1557164","article-title":"PSkip: estimating relevance ranking quality from web search clickthrough data","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Wang","year":"2009"},{"key":"2026040314520145900_ref198","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1145\/1718487.1718514","article-title":"Inferring search behaviors using partially observable markov (pom) model","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Wang","year":"2010"},{"key":"2026040314520145900_ref199","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1145\/2872427.2883074","article-title":"Detecting good abandonment in mobile search","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Williams","year":"2016"},{"key":"2026040314520145900_ref200","volume-title":"Technical Report MSR-TR-2015-34","author":"Yankov","year":"2015"},{"key":"2026040314520145900_ref201","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1145\/2661829.2661953","article-title":"Relevance and Effort: An Analysis of Document Utility","volume-title":"Proceedings of ACM Conference on Information and Knowledge Management (CIKM)","author":"Yilmaz","year":"2014"},{"key":"2026040314520145900_ref202","first-page":"1201","article-title":"Interactively optimizing information retrieval systems as a dueling bandits problem","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Yue","year":"2009"},{"key":"2026040314520145900_ref203","first-page":"507","article-title":"Learning more powerful test statistics for click-based retrieval evaluation","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Yue","year":"2010"},{"key":"2026040314520145900_ref204","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.1145\/1772690.1772793","article-title":"Beyond position bias: examining result attractiveness as a source of presentation bias in clickthrough data","volume-title":"Proceedings of the International World Wide Web Conference (WWW)","author":"Yue","year":"2010"},{"issue":"5","key":"2026040314520145900_ref205","doi-asserted-by":"crossref","first-page":"1538","DOI":"10.1016\/j.jcss.2011.12.028","article-title":"The k-armed dueling bandits problem","volume":"78","author":"Yue","year":"2012","journal-title":"Journal of Computer and System Sciences"},{"key":"2026040314520145900_ref206","first-page":"1388","article-title":"User-click modeling for understanding and predicting search-behavior","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","author":"Zhang","year":"2011"},{"key":"2026040314520145900_ref207","first-page":"262","article-title":"Aggregating ordinal labels from crowds by minimax conditional entropy","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Zhou","year":"2014"},{"key":"2026040314520145900_ref208","first-page":"307","article-title":"How reliable are the results of large-scale information retrieval experiments?","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Zobel","year":"1998"},{"key":"2026040314520145900_ref209","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1145\/2684822.2685290","article-title":"Mergerucb: A method for large-scale online ranker evaluation","volume-title":"Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM)","author":"Zoghi","year":"2015"},{"key":"2026040314520145900_ref210","doi-asserted-by":"crossref","DOI":"10.1145\/2911451.2911500","article-title":"Click-based hot fixes for underperforming torso queries","volume-title":"Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR)","author":"Zoghi","year":"2016"}],"container-title":["Foundations and Trends\u00ae in Information Retrieval"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/ftinr\/article-pdf\/10\/1\/1\/11098384\/1500000051en.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.emerald.com\/ftinr\/article-pdf\/10\/1\/1\/11098384\/1500000051en.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T14:29:55Z","timestamp":1777472995000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/ftinr\/article\/10\/1\/1\/1330378\/Online-Evaluation-for-Information-Retrieval"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,6,22]]},"references-count":210,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2016,6,22]]}},"URL":"https:\/\/doi.org\/10.1561\/1500000051","relation":{},"ISSN":["1554-0669","1554-0677"],"issn-type":[{"value":"1554-0669","type":"print"},{"value":"1554-0677","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,6,22]]}}}