{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:10:03Z","timestamp":1784178603884,"version":"3.55.0"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2012,2,1]],"date-time":"2012-02-01T00:00:00Z","timestamp":1328054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000145","name":"Division of Information and Intelligent Systems","doi-asserted-by":"publisher","award":["IIS-0905467"],"award-info":[{"award-number":["IIS-0905467"]}],"id":[{"id":"10.13039\/100000145","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2012,2]]},"abstract":"<jats:p>Interleaving is an increasingly popular technique for evaluating information retrieval systems based on implicit user feedback. While a number of isolated studies have analyzed how this technique agrees with conventional offline evaluation approaches and other online techniques, a complete picture of its efficiency and effectiveness is still lacking. In this paper we extend and combine the body of empirical evidence regarding interleaving, and provide a comprehensive analysis of interleaving using data from two major commercial search engines and a retrieval system for scientific literature. In particular, we analyze the agreement of interleaving with manual relevance judgments and observational implicit feedback measures, estimate the statistical efficiency of interleaving, and explore the relative performance of different interleaving variants. We also show how to learn improved credit-assignment functions for clicks that further increase the sensitivity of interleaving.<\/jats:p>","DOI":"10.1145\/2094072.2094078","type":"journal-article","created":{"date-parts":[[2012,3,6]],"date-time":"2012-03-06T13:18:22Z","timestamp":1331039902000},"page":"1-41","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":136,"title":["Large-scale validation and analysis of interleaved search evaluation"],"prefix":"10.1145","volume":"30","author":[{"given":"Olivier","family":"Chapelle","sequence":"first","affiliation":[{"name":"Yahoo! Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thorsten","family":"Joachims","sequence":"additional","affiliation":[{"name":"Cornell University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Filip","family":"Radlinski","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yisong","family":"Yue","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2012,3,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148175"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498824"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Conference on Data-Mining and Information Engineering. 213--222","author":"Ali K.","unstructured":"Ali , K. and Chang , C . 2006. On the relationship between click-rate and relevance for search engines . In Proceedings of the Conference on Data-Mining and Information Engineering. 213--222 . Ali, K. and Chang, C. 2006. On the relationship between click-rate and relevance for search engines. In Proceedings of the Conference on Data-Mining and Information Engineering. 213--222."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of TREC.","author":"Allan J.","unstructured":"Allan , J. , Aslam , J. A. , Carterette , B. , Pavlu , V. , and Kanoulas , E . 2008. Million query track 2008 overview . In Proceedings of TREC. Allan, J., Aslam, J. A., Carterette, B., Pavlu, V., and Kanoulas, E. 2008. Million query track 2008 overview. In Proceedings of TREC."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the ICML Workshop on Learning with Partially Classified Training Data.","author":"Aslam J. A.","unstructured":"Aslam , J. A. , Pavlu , V. , and Yilmaz , E . 2005. A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments . In Proceedings of the ICML Workshop on Learning with Partially Classified Training Data. Aslam, J. A., Pavlu, V., and Yilmaz, E. 2005. A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments. In Proceedings of the ICML Workshop on Learning with Partially Classified Training Data."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the National Conference on Artificial Intelligence (AAAI). AAAI Press, 1310--1315","author":"Becker H.","unstructured":"Becker , H. , Meek , C. , and Chickering , D. M . 2007. Modeling contextual factors of click rates . In Proceedings of the National Conference on Artificial Intelligence (AAAI). AAAI Press, 1310--1315 . Becker, H., Meek, C., and Chickering, D. M. 2007. Modeling contextual factors of click rates. In Proceedings of the National Conference on Artificial Intelligence (AAAI). AAAI Press, 1310--1315."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the AAAI Workshop on Internet Based Information Systems. 1--8.","author":"Boyan J.","unstructured":"Boyan , J. , Freitag , D. , and Joachims , T . 1996. A machine learning architecture for optimizing web search engines . In Proceedings of the AAAI Workshop on Internet Based Information Systems. 1--8. Boyan, J., Freitag, D., and Joachims, T. 1996. A machine learning architecture for optimizing web search engines. In Proceedings of the AAAI Workshop on Internet Based Information Systems. 1--8."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009000"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1358628.1358796"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the European Conference on Information Retrieval (ECIR). 16--27","author":"Carterette B.","unstructured":"Carterette , B. , Bennett , P. N. , Chickering , D. M. , and Dumais , S. T . 2008. Here or there: Preference judgements for relevance . In Proceedings of the European Conference on Information Retrieval (ECIR). 16--27 . Carterette, B., Bennett, P. N., Chickering, D. M., and Dumais, S. T. 2008. Here or there: Preference judgements for relevance. In Proceedings of the European Conference on Information Retrieval (ECIR). 16--27."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 217--224","author":"Carterette B.","unstructured":"Carterette , B. and Jones , R . 2007. Evaluating search engines by modeling the relationship between relevance and clicks . In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 217--224 . Carterette, B. and Jones, R. 2007. Evaluating search engines by modeling the relationship between relevance and clicks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 217--224."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526711"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390446"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/359784.359836"},{"key":"e_1_2_1_17_1","volume-title":"Cranfield: College of Aeronautics.","author":"Cleverdon C. W.","year":"1966","unstructured":"Cleverdon , C. W. , Mills , J. , and Keen , E. M . 1966 . Factors Determining the Performance of Indexing Systems . Cranfield: College of Aeronautics. Cleverdon, C. W., Mills, J., and Keen, E. M. 1966. Factors Determining the Performance of Indexing Systems. Cranfield: College of Aeronautics."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the WWW Workshop on Query Log Analysis.","author":"Dupret G.","unstructured":"Dupret , G. , Murdock , V. , and Piwowarski , B . 2007. Web search engine evaluation using clickthrough data and a user model . In Proceedings of the WWW Workshop on Query Log Analysis. Dupret, G., Murdock, V., and Piwowarski, B. 2007. Web search engine evaluation using clickthrough data and a user model. In Proceedings of the WWW Workshop on Query Log Analysis."},{"key":"e_1_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Efron B. and Tibshirani R. 1993. An Introduction to the Bootstrap. Chapman & Hall\/CRC.  Efron B. and Tibshirani R. 1993. An Introduction to the Bootstrap. Chapman & Hall\/CRC.","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539791195877"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1059981.1059982"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646293"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063618"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277839"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_2_1_27_1","volume-title":"Evaluating Retrieval Performance using Clickthrough Data","author":"Joachims T.","unstructured":"Joachims , T. 2003. Evaluating Retrieval Performance using Clickthrough Data . In Text Mining, J. Franke, G. Nakhaeizadeh, and I. Renz, Eds., Physica\/Springer Verlag , 79--96. Joachims, T. 2003. Evaluating Retrieval Performance using Clickthrough Data. In Text Mining, J. Franke, G. Nakhaeizadeh, and I. Renz, Eds., Physica\/Springer Verlag, 79--96."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI).","volume":"1","author":"Joachims T.","unstructured":"Joachims , T. , Freitag , D. , and Mitchell , T . 1997. WebWatcher: a tour guide for the World Wide Web . In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Vol. 1 . Morgan Kaufmann, 770--777. Joachims, T., Freitag, D., and Mitchell, T. 1997. WebWatcher: a tour guide for the World Wide Web. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Vol. 1. Morgan Kaufmann, 770--777."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1229179.1229181"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the International Conference on Computer-Assisted Information Retrieval (RIAO). 139--147","author":"Kantor P.","year":"1988","unstructured":"Kantor , P. 1988 . National, language-specific evaluation sites for retrieval systems and interfaces . In Proceedings of the International Conference on Computer-Assisted Information Retrieval (RIAO). 139--147 . Kantor, P. 1988. National, language-specific evaluation sites for retrieval systems and interfaces. In Proceedings of the International Conference on Computer-Assisted Information Retrieval (RIAO). 139--147."},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Kelly D. 2005. Implicit feedback: Using behavior to infer relevance. In New Directions in Cognitive Information Retrieval 169--186.  Kelly D. 2005. Implicit feedback: Using behavior to infer relevance. In New Directions in Cognitive Information Retrieval 169--186.","DOI":"10.1007\/1-4020-4014-8_9"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/959258.959260"},{"key":"e_1_2_1_33_1","volume-title":"Psychological Decision Theory","author":"Kozielecki J.","unstructured":"Kozielecki , J. 1981. Psychological Decision Theory . Kluwer . Kozielecki, J. 1981. Psychological Decision Theory. Kluwer."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935862"},{"key":"e_1_2_1_35_1","volume-title":"Sensory Analysis","author":"Laming D.","unstructured":"Laming , D. 1986. Sensory Analysis . Academic Press , London . Laming, D. 1986. Sensory Analysis. Academic Press, London."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Morgan Kaufmann, 924--929","author":"Lieberman H.","year":"1995","unstructured":"Lieberman , H. 1995 . Letizia: An agent that assists Web browsing . In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Morgan Kaufmann, 924--929 . Lieberman, H. 1995. Letizia: An agent that assists Web browsing. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Morgan Kaufmann, 924--929."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242731"},{"key":"e_1_2_1_38_1","doi-asserted-by":"crossref","unstructured":"Manning C. D. Raghavan P. and Schuetze H. 2008. Introduction to Information Retrieval. Cambridge University Press.   Manning C. D. Raghavan P. and Schuetze H. 2008. Introduction to Information Retrieval. Cambridge University Press.","DOI":"10.1017\/CBO9780511809071"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Workshop on Context-Based Information Retrieval, B.-L. Doan, J. M. Jose, and M. Melucci, Eds., CEUR Workshop Proceedings","volume":"326","author":"Moe K. K.","unstructured":"Moe , K. K. , Jensen , J. M. , and Larsen , B . 2007. A qualitative look at eye-tracking for implicit relevance feedback . In Proceedings of the Workshop on Context-Based Information Retrieval, B.-L. Doan, J. M. Jose, and M. Melucci, Eds., CEUR Workshop Proceedings , vol. 326 , CEUR-WS.org. Moe, K. K., Jensen, J. M., and Larsen, B. 2007. A qualitative look at eye-tracking for implicit relevance feedback. In Proceedings of the Workshop on Context-Based Information Retrieval, B.-L. Doan, J. M. Jose, and M. Melucci, Eds., CEUR Workshop Proceedings, vol. 326, CEUR-WS.org."},{"key":"e_1_2_1_40_1","unstructured":"Mood A. Graybill F. and Boes D. 1974. Introduction to the Theory of Statistics 3rd Ed. McGraw-Hill New York NY.  Mood A. Graybill F. and Boes D. 1974. Introduction to the Theory of Statistics 3rd Ed. McGraw-Hill New York NY."},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 272--281","author":"Morita M.","unstructured":"Morita , M. and Shinoda , Y . 1994. Information filtering based on user behavior analysis and best match text retrieval . In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 272--281 . Morita, M. and Shinoda, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 272--281."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the Annual Meeting of the American Society for Information Science and Technology. 28--45","author":"Oard D. W.","unstructured":"Oard , D. W. and Kim , J . 2001. Modeling information content using observable behavior . In Proceedings of the Annual Meeting of the American Society for Information Science and Technology. 28--45 . Oard, D. W. and Kim, J. 2001. Modeling information content using observable behavior. In Proceedings of the Annual Meeting of the American Society for Information Science and Technology. 28--45."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1670564.1670572"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835560"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1081870.1081899"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI). 1406--1412","author":"Radlinski F.","unstructured":"Radlinski , F. and Joachims , T . 2006. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs . In Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI). 1406--1412 . Radlinski, F. and Joachims, T. 2006. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI). 1406--1412."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458092"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"Radlinski F. Kurup M. and Joachims T. 2010a. Evaluating search engine relevance with click-based metrics. In Preference Learning J. Fuernkranz and E. Huellermeier Eds. Springer 337--362.  Radlinski F. Kurup M. and Joachims T. 2010a. Evaluating search engine relevance with click-based metrics. In Preference Learning J. Fuernkranz and E. Huellermeier Eds. Springer 337--362.","DOI":"10.1007\/978-3-642-14125-6_16"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772859"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/11550822_80"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076064"},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"Shao J. and Tu D. 1995. The Jackknife and Bootstrap. Springer.  Shao J. and Tu D. 1995. The Jackknife and Bootstrap. Springer.","DOI":"10.1007\/978-1-4612-0795-5"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277894"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148176"},{"key":"e_1_2_1_56_1","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Voorhees E.","year":"2005","unstructured":"Voorhees , E. and Harman , D . 2005 . TREC: Experiment and Evaluation in Information Retrieval . MIT press . Voorhees, E. and Harman, D. 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT press."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564432"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557164"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1080343.1080347"},{"key":"e_1_2_1_60_1","doi-asserted-by":"crossref","unstructured":"White R. Ruthven I. and \n      Jose J. M\n  . \n  2002\n  . The use of implicit evidence for relevance feedback in web retrieval. In Proceedings of the European Conference on Information Retrieval (ECIR). F. Crestani M. Girolami and C. J. van Rijsbergen Eds. Lecture Notes in Computer Science vol. \n  2291 Springer 93--109.   White R. Ruthven I. and Jose J. M. 2002. The use of implicit evidence for relevance feedback in web retrieval. In Proceedings of the European Conference on Information Retrieval (ECIR). F. Crestani M. Girolami and C. J. van Rijsbergen Eds. Lecture Notes in Computer Science vol. 2291 Springer 93--109.","DOI":"10.1007\/3-540-45886-7_7"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the Annual Conference on Learning Theory (COLT).","author":"Yue Y.","unstructured":"Yue , Y. , Broder , J. , Kleinberg , R. , and Joachims , T . 2009. The K-armed Dueling Bandits Problem . In Proceedings of the Annual Conference on Learning Theory (COLT). Yue, Y., Broder, J., Kleinberg, R., and Joachims, T. 2009. The K-armed Dueling Bandits Problem. In Proceedings of the Annual Conference on Learning Theory (COLT)."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835534"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553527"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML). 241--248","author":"Yue Y.","unstructured":"Yue , Y. and Joachims , T . 2011. Beat the mean bandit . In Proceedings of the International Conference on Machine Learning (ICML). 241--248 . Yue, Y. and Joachims, T. 2011. Beat the mean bandit. In Proceedings of the International Conference on Machine Learning (ICML). 241--248."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772793"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2094072.2094078","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2094072.2094078","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:48:41Z","timestamp":1750240121000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2094072.2094078"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,2]]},"references-count":65,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2012,2]]}},"alternative-id":["10.1145\/2094072.2094078"],"URL":"https:\/\/doi.org\/10.1145\/2094072.2094078","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,2]]},"assertion":[{"value":"2011-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-03-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}