{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T05:21:36Z","timestamp":1784092896385,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":49,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T00:00:00Z","timestamp":1603065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["Discovery grant"],"award-info":[{"award-number":["Discovery grant"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,19]]},"DOI":"10.1145\/3340531.3411915","type":"proceedings-article","created":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T05:31:05Z","timestamp":1603085465000},"page":"225-234","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["Offline Evaluation by Maximum Similarity to an Ideal Ranking"],"prefix":"10.1145","author":[{"given":"Charles L. A.","family":"Clarke","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark D.","family":"Smucker","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandra","family":"Vtyurina","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,10,19]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"28th Text REtrieval Conference.","author":"Abualsaud Mustafa","unstructured":"Mustafa Abualsaud , Fuat C. Beyluniovglu , Mark D. Smucker , and P. Robert Duimering . 2019. UWaterlooMDS at the TREC 2019 Decision Track . In 28th Text REtrieval Conference. Mustafa Abualsaud, Fuat C. Beyluniovglu, Mark D. Smucker, and P. Robert Duimering. 2019. UWaterlooMDS at the TREC 2019 Decision Track. In 28th Text REtrieval Conference."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148177"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498766"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-15712-8_39"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646031"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210027"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348301"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/725"},{"key":"e_1_3_2_2_9_1","volume-title":"24th International Conference on Neural Information Processing Systems. 2546--2554","author":"Bergstra James","year":"2011","unstructured":"James Bergstra , R\u00e9mi Bardenet , Yoshua Bengio , and Bal\u00e1zs K\u00e9gl . 2011 . Algorithms for hyper-parameter optimization . In 24th International Conference on Neural Information Processing Systems. 2546--2554 . James Bergstra, R\u00e9mi Bardenet, Yoshua Bengio, and Bal\u00e1zs K\u00e9gl. 2011. Algorithms for hyper-parameter optimization. In 24th International Conference on Neural Information Processing Systems. 2546--2554."},{"key":"e_1_3_2_2_10_1","volume-title":"3rd Text REtrieval Conference.","author":"Buckley Chris","year":"1994","unstructured":"Chris Buckley , Gerard Salton , James Allan , and Amit Singhal . 1994 . Automatic query expansion using SMART: TREC 3 . In 3rd Text REtrieval Conference. Chris Buckley, Gerard Salton, James Allan, and Amit Singhal. 1994. Automatic query expansion using SMART: TREC 3. In 3rd Text REtrieval Conference."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291025"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04417-5_18"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"key":"e_1_3_2_2_15_1","volume-title":"31st ACM SIGIR Conference on Research and Development in Information Retrieval. 685--686","author":"Carterette Ben","unstructured":"Ben Carterette and Paul N. Bennett . 2008. Evaluation measures for preference judgments . In 31st ACM SIGIR Conference on Research and Development in Information Retrieval. 685--686 . Ben Carterette and Paul N. Bennett. 2008. Evaluation measures for preference judgments. In 31st ACM SIGIR Conference on Research and Development in Information Retrieval. 685--686."},{"key":"e_1_3_2_2_16_1","volume-title":"Proceedings of the Learning to Rank Challenge (Proceedings of Machine Learning Research), Olivier Chapelle, Yi Chang, and Tie-Yan Liu (Eds.)","volume":"14","author":"Chapelle Olivier","year":"2011","unstructured":"Olivier Chapelle and Yi Chang . 2011 . Yahoo! Learning to Rank Challenge overview . In Proceedings of the Learning to Rank Challenge (Proceedings of Machine Learning Research), Olivier Chapelle, Yi Chang, and Tie-Yan Liu (Eds.) , Vol. 14 . Proceedings of Machine Learning Research, 1--24. Olivier Chapelle and Yi Chang. 2011. Yahoo! Learning to Rank Challenge overview. In Proceedings of the Learning to Rank Challenge (Proceedings of Machine Learning Research), Olivier Chapelle, Yi Chang, and Tie-Yan Liu (Eds.), Vol. 14. Proceedings of Machine Learning Research, 1--24."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_3_2_2_18_1","unstructured":"Charles L. A. Clarke Alexandra Vtyurina and Mark D. Smucker. 2020 a. Assessing top-k preferences. arxiv: cs.IR\/2007.11682  Charles L. A. Clarke Alexandra Vtyurina and Mark D. Smucker. 2020 a. Assessing top-k preferences. arxiv: cs.IR\/2007.11682"},{"key":"e_1_3_2_2_19_1","volume-title":"ACM SIGIR International Conference on the Theory of Information Retrieval.","author":"Clarke Charles L. A.","unstructured":"Charles L. A. Clarke , Alexandra Vtyurina , and Mark D. Smucker . 2020 b. Offline evaluation without gain . In ACM SIGIR International Conference on the Theory of Information Retrieval. Charles L. A. Clarke, Alexandra Vtyurina, and Mark D. Smucker. 2020 b. Offline evaluation without gain. In ACM SIGIR International Conference on the Theory of Information Retrieval."},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026683"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274784.3274788"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_2_2_23_1","volume-title":"28th Text REtrieval Conference.","author":"Dalton Jeffrey","year":"2019","unstructured":"Jeffrey Dalton , Chenyan Xiong , and Jamie Callan . 2019 . CAsT 2019: The Conversational Assistance Track overview . In 28th Text REtrieval Conference. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2019. CAsT 2019: The Conversational Assistance Track overview. In 28th Text REtrieval Conference."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1561\/9781680831627"},{"key":"e_1_3_2_2_25_1","volume-title":"ACM Transactions on Information Systems","volume":"20","author":"Jaana Kalervo","year":"2002","unstructured":"Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002 . Cumulated gain-based evaluation of IR techniques . ACM Transactions on Information Systems , Vol. 20 , 4 (2002), 422--446. Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, Vol. 20, 4 (2002), 422--446."},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556195.2556220"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308774.3308781"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2523648"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2507665"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935850"},{"key":"e_1_3_2_2_32_1","volume-title":"22nd Text REtrieval Conference.","author":"Raiber Fiana","year":"2013","unstructured":"Fiana Raiber and Oren Kurland . 2013 . The Technion at TREC 2013 Web Track: Cluster-based document retrieval . In 22nd Text REtrieval Conference. Fiana Raiber and Oren Kurland. 2013. The Technion at TREC 2013 Web Track: Cluster-based document retrieval. In 22nd Text REtrieval Conference."},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026866"},{"key":"e_1_3_2_2_34_1","volume-title":"17th ACM SIGIR Conference on Research and Development in Information Retrieval. 232--241","author":"Robertson S. E.","unstructured":"S. E. Robertson and S. Walker . 1994. Some simple effective Approximations to the 2-Poisson model for probabilistic weighted retrieval . In 17th ACM SIGIR Conference on Research and Development in Information Retrieval. 232--241 . S. E. Robertson and S. Walker. 1994. Some simple effective Approximations to the 2-Poisson model for probabilistic weighted retrieval. In 17th ACM SIGIR Conference on Research and Development in Information Retrieval. 232--241."},{"key":"e_1_3_2_2_35_1","volume-title":"3rd Text REtrieval Conference.","author":"Robertson S. E.","unstructured":"S. E. Robertson , S. Walker , S. Jones , M. M. Hancock-Beaulieu , and M. Gatford . 1994. Okapi at TREC-3 . In 3rd Text REtrieval Conference. S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1994. Okapi at TREC-3. In 3rd Text REtrieval Conference."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199012)41:8<590::AID-ASI5>3.0.CO;2-T"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888903000638"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484031"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010055"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331215"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243206"},{"key":"e_1_3_2_2_42_1","volume-title":"35th ACM SIGIR Conference on Research and Development in Information Retrieval. 95--104","author":"Mark","unstructured":"Mark D. Smucker and Charles L.A. Clarke. 2012. Time-based calibration of effectiveness measures . In 35th ACM SIGIR Conference on Research and Development in Information Retrieval. 95--104 . Mark D. Smucker and Charles L.A. Clarke. 2012. Time-based calibration of effectiveness measures. In 35th ACM SIGIR Conference on Research and Development in Information Retrieval. 95--104."},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2682862.2682863"},{"key":"e_1_3_2_2_44_1","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Voorhees Ellen M.","year":"2005","unstructured":"Ellen M. Voorhees and Donna K. Harman ( Eds .) . 2005 . TREC: Experiment and Evaluation in Information Retrieval . MIT Press . Ellen M. Voorhees and Donna K. Harman (Eds.). 2005. TREC: Experiment and Evaluation in Information Retrieval .MIT Press."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331340"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3291992.3291995"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.5555\/208098.208105"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871672"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860440"}],"event":{"name":"CIKM '20: The 29th ACM International Conference on Information and Knowledge Management","location":"Virtual Event Ireland","acronym":"CIKM '20","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 29th ACM International Conference on Information &amp; Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340531.3411915","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3340531.3411915","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:22Z","timestamp":1750197682000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340531.3411915"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,19]]},"references-count":49,"alternative-id":["10.1145\/3340531.3411915","10.1145\/3340531"],"URL":"https:\/\/doi.org\/10.1145\/3340531.3411915","relation":{},"subject":[],"published":{"date-parts":[[2020,10,19]]},"assertion":[{"value":"2020-10-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}