{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T06:19:26Z","timestamp":1775283566044,"version":"3.50.1"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2013,11,1]],"date-time":"2013-11-01T00:00:00Z","timestamp":1383264000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"European Union's ICT Policy Support Programme as part of the Competitiveness and Innovation Framework Programme"},{"name":"Center for Creation, Content and Technology"},{"name":"CLARIN-nl program"},{"name":"CIP ICT-PSP","award":["250430"],"award-info":[{"award-number":["250430"]}]},{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["612.061.814, 612.061.815, 640.004.802, 727.011.005, 612.001.116, HOR-11-10"],"award-info":[{"award-number":["612.061.814, 612.061.815, 640.004.802, 727.011.005, 612.001.116, HOR-11-10"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004895","name":"European Social Fund","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004895","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Dutch national program COMMIT"},{"DOI":"10.13039\/501100004963","name":"Seventh Framework Programme","doi-asserted-by":"publisher","award":["258191 (PROMISE Network of Excellence), 288024 (LiMoSINe project)"],"award-info":[{"award-number":["258191 (PROMISE Network of Excellence), 288024 (LiMoSINe project)"]}],"id":[{"id":"10.13039\/501100004963","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001722","name":"Royal Netherlands Academy of Arts and Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001722","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2013,11]]},"abstract":"<jats:p>\n            Ranker evaluation is central to the research into search engines, be it to compare rankers or to provide feedback for learning to rank. Traditional evaluation approaches do not scale well because they require explicit relevance judgments of document-query pairs, which are expensive to obtain. A promising alternative is the use of\n            <jats:italic>interleaved comparison<\/jats:italic>\n            methods, which compare rankers using click data obtained when interleaving their rankings.\n          <\/jats:p>\n          <jats:p>\n            In this article, we propose a framework for analyzing interleaved comparison methods. An interleaved comparison method has\n            <jats:italic>fidelity<\/jats:italic>\n            if the expected outcome of ranker comparisons properly corresponds to the true relevance of the ranked documents. It is\n            <jats:italic>sound<\/jats:italic>\n            if its estimates of that expected outcome are unbiased and consistent. It is\n            <jats:italic>efficient<\/jats:italic>\n            if those estimates are accurate with only little data.\n          <\/jats:p>\n          <jats:p>\n            We analyze existing interleaved comparison methods and find that, while sound, none meet our criteria for fidelity. We propose a\n            <jats:italic>probabilistic interleave<\/jats:italic>\n            method, which is sound and has fidelity. We show empirically that, by marginalizing out variables that are known, it is more efficient than existing interleaved comparison methods. Using importance sampling we derive a sound extension that is able to reuse historical data collected in previous comparisons of other ranker pairs.\n          <\/jats:p>","DOI":"10.1145\/2536736.2536737","type":"journal-article","created":{"date-parts":[[2013,12,4]],"date-time":"2013-12-04T14:04:47Z","timestamp":1386165887000},"page":"1-43","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["Fidelity, Soundness, and Efficiency of Interleaved Comparison Methods"],"prefix":"10.1145","volume":"31","author":[{"given":"Katja","family":"Hofmann","sequence":"first","affiliation":[{"name":"University of Amsterdam"}]},{"given":"Shimon","family":"Whiteson","sequence":"additional","affiliation":[{"name":"University of Amsterdam"}]},{"given":"Maarten De","family":"Rijke","sequence":"additional","affiliation":[{"name":"University of Amsterdam"}]}],"member":"320","published-online":{"date-parts":[[2013,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148177"},{"key":"e_1_2_1_2_1","unstructured":"Carterette B. and Jones R. 2008. Evaluating search engines by modeling the relationship between relevance and clicks. In Advances in Neural Information Processing Systems 20 (NIPS\u201907). J. Platt D. Koller Y. Singer and S. Roweis Eds. MIT Press Cambridge MA 217--224.  Carterette B. and Jones R. 2008. Evaluating search engines by modeling the relationship between relevance and clicks. In Advances in Neural Information Processing Systems 20 (NIPS\u201907) . J. Platt D. Koller Y. Singer and S. Roweis Eds. MIT Press Cambridge MA 217--224."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526711"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.spl.2005.01.002"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484071"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201911)","author":"Dud\u00edk M."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718510"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the Workshop on Query Log Analysis.","author":"Dupret G."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1059981.1059982"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.2307\/2527652"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498818"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177731020"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646293"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835603"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063618"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396780"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398516"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433419"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1571950"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_2_1_24_1","unstructured":"Joachims T. 2003. Evaluating retrieval performance using clickthrough data. In Text Mining J. Franke G. Nakhaeizadeh and I. Renz Eds. Springer 79--96.  Joachims T. 2003. Evaluating retrieval performance using clickthrough data. In Text Mining J. Franke G. Nakhaeizadeh and I. Renz Eds. Springer 79--96."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.07.021"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1507509.1507522"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390223"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Lehmann E. L. 1999. Elements of Large-Sample Theory. Springer.  Lehmann E. L. 1999. Elements of Large-Sample Theory . Springer.","DOI":"10.1007\/b98855"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935878"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/35.41401"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/11880592_51"},{"key":"e_1_2_1_32_1","volume-title":"Learning in Graphical Models","author":"Mackay D. J. C."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1963405.1963463"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201900)","author":"Precup D."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835560"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433429"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390255"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458092"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935859"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/988672.988675"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the European Conference on Information Retrieval (ECIR\u201908)","author":"Scholer F."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076045"},{"key":"e_1_2_1_44_1","unstructured":"Strehl A. M. Langford J. Li L. and Kakade S. M. 2010. Learning from logged implicit exploration data. In Advances in Neural Information Processing Systems 23 (NIPS\u201910). J. Lafferty C. K. I. Williams J. Shawe-Taylor R. Zemel and A. Culotta Eds. 2217--2225.  Strehl A. M. Langford J. Li L. and Kakade S. M. 2010. Learning from logged implicit exploration data. In Advances in Neural Information Processing Systems 23 (NIPS\u201910) . J. Lafferty C. K. I. Williams J. Shawe-Taylor R. Zemel and A. Culotta Eds. 2217--2225."},{"key":"e_1_2_1_45_1","unstructured":"Sutton R. S. and Barto A. G. 1998. Introduction to Reinforcement Learning. MIT Press Cambridge MA.   Sutton R. S. and Barto A. G. 1998. Introduction to Reinforcement Learning . MIT Press Cambridge MA."},{"key":"e_1_2_1_46_1","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Voorhees E. M.","year":"2005"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031171.1031192"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835534"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 14th European Conference on Research and Advanced Technology for Digital Libraries (ECDL\u201910)","author":"Zhang J."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2536736.2536737","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2536736.2536737","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:14:42Z","timestamp":1750277682000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2536736.2536737"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,11]]},"references-count":49,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2013,11]]}},"alternative-id":["10.1145\/2536736.2536737"],"URL":"https:\/\/doi.org\/10.1145\/2536736.2536737","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,11]]},"assertion":[{"value":"2012-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-11-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}