{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,3]],"date-time":"2025-12-03T18:01:36Z","timestamp":1764784896028,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":28,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T00:00:00Z","timestamp":1657065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"funder":[{"name":"CIFAR"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,6]]},"DOI":"10.1145\/3477495.3532033","type":"proceedings-article","created":{"date-parts":[[2022,7,7]],"date-time":"2022-07-07T15:12:08Z","timestamp":1657206728000},"page":"599-609","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Offline Retrieval Evaluation Without Evaluation Metrics"],"prefix":"10.1145","author":[{"given":"Fernando","family":"Diaz","sequence":"first","affiliation":[{"name":"Google, Montr\u00e9al, PQ, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andres","family":"Ferraro","sequence":"additional","affiliation":[{"name":"Mila - Quebec Artificial Intelligence Institute, Montr\u00e9al, PQ, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,7,7]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498766"},{"key":"e_1_3_2_2_2_1","volume-title":"Atomized Search Length: Beyond User Models. CoRR","author":"Alex John","year":"2022","unstructured":"John Alex , Keith Hall , and Donald Metzler . 2022. Atomized Search Length: Beyond User Models. CoRR , Vol. abs\/ 2201 .01745 ( 2022 ). showeprint[arXiv]2201.01745 https:\/\/arxiv.org\/abs\/2201.01745 John Alex, Keith Hall, and Donald Metzler. 2022. Atomized Search Length: Beyond User Models. CoRR , Vol. abs\/2201.01745 (2022). showeprint[arXiv]2201.01745 https:\/\/arxiv.org\/abs\/2201.01745"},{"key":"e_1_3_2_2_3_1","volume-title":"Alonzo","author":"Braun Thomas M.","year":"2007","unstructured":"Thomas M. Braun and Todd A . Alonzo . 2007 . A modified sign test for comparing paired ROC curves . Biostatistics , Vol. 9 , 2 (10 2007), 364--372. Thomas M. Braun and Todd A. Alonzo. 2007. A modified sign test for comparing paired ROC curves . Biostatistics , Vol. 9, 2 (10 2007), 364--372."},{"volume-title":"Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '04)","author":"Buckley Chris","key":"e_1_3_2_2_4_1","unstructured":"Chris Buckley and Ellen M. Voorhees . 2004. Retrieval Evaluation with Incomplete Information . In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '04) . ACM, New York, NY, USA, 25--32. Chris Buckley and Ellen M. Voorhees. 2004. Retrieval Evaluation with Incomplete Information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '04). ACM, New York, NY, USA, 25--32."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"volume-title":"SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM","author":"Carterette Ben","key":"e_1_3_2_2_6_1","unstructured":"Ben Carterette and Paul N. Bennett . 2008. Evaluation measures for preference judgments . In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM , New York, NY, USA, 685--686. Ben Carterette and Paul N. Bennett. 2008. Evaluation measures for preference judgments. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, New York, NY, USA, 685--686."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3451161"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.5090190108"},{"volume-title":"Proceedings of the Twenty-Eighth Text REtrieval Conference (TREC 2019)","author":"Craswell Nick","key":"e_1_3_2_2_11_1","unstructured":"Nick Craswell , Bhaskar Mitra , Daniel Campos , Emine Yilmaz , and Ellen M. Voorhees . 2020. Overview of the TREC 2019 Deep Learning Track . In Proceedings of the Twenty-Eighth Text REtrieval Conference (TREC 2019) . Nick Craswell, Bhaskar Mitra, Daniel Campos, Emine Yilmaz, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 Deep Learning Track. In Proceedings of the Twenty-Eighth Text REtrieval Conference (TREC 2019) ."},{"volume-title":"Proceedings of the 10th International Conference on World Wide Web (WWW '01)","author":"Dwork Cynthia","key":"e_1_3_2_2_12_1","unstructured":"Cynthia Dwork , Ravi Kumar , Moni Naor , and D. Sivakumar . 2001. Rank Aggregation Methods for the Web . In Proceedings of the 10th International Conference on World Wide Web (WWW '01) . Association for Computing Machinery, New York, NY, USA, 613--622. Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. 2001. Rank Aggregation Methods for the Web. In Proceedings of the 10th International Conference on World Wide Web (WWW '01). Association for Computing Machinery, New York, NY, USA, 613--622."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661913"},{"key":"e_1_3_2_2_14_1","volume-title":"TOIS","volume":"20","author":"Jaana Kalervo","year":"2002","unstructured":"Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002 . Cumulated gain-based evaluation of IR techniques . TOIS , Vol. 20 , 4 (2002), 422--446. Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002. Cumulated gain-based evaluation of IR techniques. TOIS , Vol. 20, 4 (2002), 422--446."},{"key":"e_1_3_2_2_15_1","unstructured":"Thorsten Joachims. 2003. Evaluating Retrieval Performance Using Clickthrough Data .Physica\/Springer Verlag 79--96.  Thorsten Joachims. 2003. Evaluating Retrieval Performance Using Clickthrough Data .Physica\/Springer Verlag 79--96."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484168"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403226"},{"volume-title":"Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '10)","author":"Magdy Walid","key":"e_1_3_2_2_19_1","unstructured":"Walid Magdy and Gareth J.F. Jones . 2010. PRES: A Score Metric for Evaluating Recall-Oriented Information Retrieval Applications . In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '10) . Association for Computing Machinery, New York, NY, USA, 611--618. Walid Magdy and Gareth J.F. Jones. 2010. PRES: A Score Metric for Evaluating Recall-Oriented Information Retrieval Applications. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '10). Association for Computing Machinery, New York, NY, USA, 611--618."},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390453"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835550"},{"key":"e_1_3_2_2_24_1","volume-title":"Bridging Between Information Retrieval and Databases - PROMISE Winter School","author":"Sakai Tetsuya","year":"2013","unstructured":"Tetsuya Sakai . 2014. Metrics , statistics, tests . In Bridging Between Information Retrieval and Databases - PROMISE Winter School 2013 , Revised Tutorial Lectures (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Springer Verlag , 116--163. Tetsuya Sakai. 2014. Metrics, statistics, tests. In Bridging Between Information Retrieval and Databases - PROMISE Winter School 2013, Revised Tutorial Lectures (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Springer Verlag, 116--163."},{"key":"e_1_3_2_2_25_1","volume-title":"Proceedings of The Second International Workshop on Evaluating Information Access (EVIA) .","author":"Sakai Tetsuya","year":"2008","unstructured":"Tetsuya Sakai and Stephen Robertson . 2008 . Modelling A User Population for Designing Information Retrieval Metrics . In Proceedings of The Second International Workshop on Evaluating Information Access (EVIA) . Tetsuya Sakai and Stephen Robertson. 2008. Modelling A User Population for Designing Information Retrieval Metrics. In Proceedings of The Second International Workshop on Evaluating Information Access (EVIA) ."},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835542"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183632"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09377-x"},{"volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18)","author":"Zhang Haotian","key":"e_1_3_2_2_29_1","unstructured":"Haotian Zhang , Mustafa Abualsaud , Nimesh Ghelani , Mark D. Smucker , Gordon V. Cormack , and Maura R. Grossman . 2018. Effective User Interaction for High-Recall Retrieval: Less is More . In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18) . Association for Computing Machinery, New York, NY, USA, 187--196. Haotian Zhang, Mustafa Abualsaud, Nimesh Ghelani, Mark D. Smucker, Gordon V. Cormack, and Maura R. Grossman. 2018. Effective User Interaction for High-Recall Retrieval: Less is More. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18). Association for Computing Machinery, New York, NY, USA, 187--196."}],"event":{"name":"SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Madrid Spain","acronym":"SIGIR '22"},"container-title":["Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477495.3532033","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477495.3532033","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:10:34Z","timestamp":1750183834000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477495.3532033"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,6]]},"references-count":28,"alternative-id":["10.1145\/3477495.3532033","10.1145\/3477495"],"URL":"https:\/\/doi.org\/10.1145\/3477495.3532033","relation":{},"subject":[],"published":{"date-parts":[[2022,7,6]]},"assertion":[{"value":"2022-07-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}