{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:13:36Z","timestamp":1755839616818,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":59,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T00:00:00Z","timestamp":1697846400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010801","name":"Xunta de Galicia","doi-asserted-by":"publisher","award":["ED431-B 2022\/33"],"award-info":[{"award-number":["ED431-B 2022\/33"]}],"id":[{"id":"10.13039\/501100010801","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100021856","name":"Ministero dell'Universit\u00e0 e della Ricerca","doi-asserted-by":"publisher","award":["CAMEO, PRIN 2022 n. 2022ZLL7MW"],"award-info":[{"award-number":["CAMEO, PRIN 2022 n. 2022ZLL7MW"]}],"id":[{"id":"10.13039\/501100021856","id-type":"DOI","asserted-by":"publisher"}]},{"name":"MCIN","award":["PLEC2021-007662 and PID2022-137061OB-C21"],"award-info":[{"award-number":["PLEC2021-007662 and PID2022-137061OB-C21"]}]},{"DOI":"10.13039\/501100023561","name":"Ministerio de Universidades","doi-asserted-by":"publisher","award":["FPU20\/02659"],"award-info":[{"award-number":["FPU20\/02659"]}],"id":[{"id":"10.13039\/501100023561","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,10,21]]},"DOI":"10.1145\/3583780.3614916","type":"proceedings-article","created":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T07:45:26Z","timestamp":1697874326000},"page":"1960-1970","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication Methods"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1139-0449","authenticated-orcid":false,"given":"David","family":"Otero","sequence":"first","affiliation":[{"name":"Universidade da Coru\u00f1a, A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5997-8252","authenticated-orcid":false,"given":"Javier","family":"Parapar","sequence":"additional","affiliation":[{"name":"Univesidade da Coru\u00f1a, A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9219-6239","authenticated-orcid":false,"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"University of Padua, Padova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,10,21]]},"reference":[{"volume-title":"TREC 2017 common core track overview. In Proceedings of TREC 2017. NIST Special Publication 500--324","author":"Allan James","key":"e_1_3_2_1_1_1","unstructured":"James Allan , Donna K. Harman , Evangelos Kanoulas , Dan Li , Christophe Van Gysel , and Ellen M. Voorhees . 2017 . TREC 2017 common core track overview. In Proceedings of TREC 2017. NIST Special Publication 500--324 , Gaithersburg, Maryland, USA. https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/Overview-CC.pdf (cited on p. 2). James Allan, Donna K. Harman, Evangelos Kanoulas, Dan Li, Christophe Van Gysel, and Ellen M. Voorhees. 2017. TREC 2017 common core track overview. In Proceedings of TREC 2017. NIST Special Publication 500--324, Gaithersburg, Maryland, USA. https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/Overview-CC.pdf (cited on p. 2)."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412121"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/956863.956953"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009984519381"},{"key":"e_1_3_2_1_5_1","volume-title":"Proceedings of ACM SIGIR","author":"Buckley Chris","year":"2000","unstructured":"Chris Buckley and Ellen M. Voorhees . 2000. Evaluating evaluation measure stability . In Proceedings of ACM SIGIR 2000 . ACM, New York, NY, USA, 33--40. doi: 10.1145\/345508.345543 (cited on p. 6). 10.1145\/345508.345543 Chris Buckley and Ellen M. Voorhees. 2000. Evaluating evaluation measure stability. In Proceedings of ACM SIGIR 2000. ACM, New York, NY, USA, 33--40. doi: 10.1145\/345508.345543 (cited on p. 6)."},{"key":"e_1_3_2_1_6_1","volume-title":"Voorhees","author":"Buckley Chris","year":"2005","unstructured":"Chris Buckley and Ellen M . Voorhees . 2005 . Retrieval system evaluation. In TREC: Experiment and Evaluation in Information Retrieval. Ellen M. Voorhees and Donna K. Harman, (Eds.) The MIT Press , 53--78 (cited on p. 4). Chris Buckley and Ellen M. Voorhees. 2005. Retrieval system evaluation. In TREC: Experiment and Evaluation in Information Retrieval. Ellen M. Voorhees and Donna K. Harman, (Eds.) The MIT Press, 53--78 (cited on p. 4)."},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of ACM SIGIR","author":"Carterette Ben","year":"2017","unstructured":"Ben Carterette . 2017 . But is it statistically significant? statistical significance in ir research, 1995--2014 . In Proceedings of ACM SIGIR 2017. ACM, New York, NY, USA, 1125--1128. doi: 10.1145\/3077136.3080738 (cited on p. 3). 10.1145\/3077136.3080738 Ben Carterette. 2017. But is it statistically significant? statistical significance in ir research, 1995--2014. In Proceedings of ACM SIGIR 2017. ACM, New York, NY, USA, 1125--1128. doi: 10.1145\/3077136.3080738 (cited on p. 3)."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of ACM SIGIR","author":"Gordon","year":"2006","unstructured":"Gordon V. Cormack and Thomas R. Lynam. 2006. Statistical precision of information retrieval evaluation . In Proceedings of ACM SIGIR 2006 . ACM, New York, NY, USA, 533--540. doi: 10.1145\/1148170.1148262 (cited on p. 3). 10.1145\/1148170.1148262 Gordon V. Cormack and Thomas R. Lynam. 2006. Statistical precision of information retrieval evaluation. In Proceedings of ACM SIGIR 2006. ACM, New York, NY, USA, 533--540. doi: 10.1145\/1148170.1148262 (cited on p. 3)."},{"key":"e_1_3_2_1_10_1","volume-title":"Proceedings of ACM SIGIR","author":"Gordon","year":"2007","unstructured":"Gordon V. Cormack and Thomas R. Lynam. 2007. Validity and power of t-test for comparing map and gmap . In Proceedings of ACM SIGIR 2007 . ACM, New York, NY, USA, 753--754. doi: 10.1145\/1277741.1277892 (cited on p. 3). 10.1145\/1277741.1277892 Gordon V. Cormack and Thomas R. Lynam. 2007. Validity and power of t-test for comparing map and gmap. In Proceedings of ACM SIGIR 2007. ACM, New York, NY, USA, 753--754. doi: 10.1145\/1277741.1277892 (cited on p. 3)."},{"key":"e_1_3_2_1_11_1","first-page":"2","volume-title":"Proceedings of ACM SIGIR","author":"Cormack Gordon V.","year":"1998","unstructured":"Gordon V. Cormack , Christopher R. Palmer , and Charles L. A. Clarke . 1998. Efficient construction of large test collections . In Proceedings of ACM SIGIR 1998 . ACM, New York, NY, USA, 282--289. doi: 10.1145\/290941.291009 (cited on pp. 2 , 4). 10.1145\/290941.291009 Gordon V. Cormack, Christopher R. Palmer, and Charles L. A. Clarke. 1998. Efficient construction of large test collections. In Proceedings of ACM SIGIR 1998. ACM, New York, NY, USA, 282--289. doi: 10.1145\/290941.291009 (cited on pp. 2, 4)."},{"key":"e_1_3_2_1_12_1","volume-title":"Overview of the TREC 2021 deep learning track. In Proceedings of TREC 2021. NIST Special Publication 500--335","author":"Craswell Nick","year":"2021","unstructured":"Nick Craswell , Bhaskar Mitra , Emine Yilmaz , Daniel Campos , and Jimmy Lin . 2021 . Overview of the TREC 2021 deep learning track. In Proceedings of TREC 2021. NIST Special Publication 500--335 , Gaithersburg, Maryland, USA. https:\/\/trec.nist.gov\/pubs\/trec30\/papers\/Overview-DL.pdf (cited on p. 4). Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2021. Overview of the TREC 2021 deep learning track. In Proceedings of TREC 2021. NIST Special Publication 500--335, Gaithersburg, Maryland, USA. https:\/\/trec.nist.gov\/pubs\/trec30\/papers\/Overview-DL.pdf (cited on p. 4)."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463249"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72240-1_3"},{"key":"e_1_3_2_1_15_1","volume-title":"Proceedings of ACM WSDM","author":"Ferro Nicola","year":"2022","unstructured":"Nicola Ferro and Mark Sanderson . 2022 . How do you test a test? a multifaceted examination of significance tests . In Proceedings of ACM WSDM 2022. ACM, New York, NY, USA, 280--288. doi: 10.1145\/3488560.3498406 (cited on p. 3). 10.1145\/3488560.3498406 Nicola Ferro and Mark Sanderson. 2022. How do you test a test? a multifaceted examination of significance tests. In Proceedings of ACM WSDM 2022. ACM, New York, NY, USA, 280--288. doi: 10.1145\/3488560.3498406 (cited on p. 3)."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3338062"},{"key":"e_1_3_2_1_17_1","volume-title":"Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum, 51, 3, (Feb","author":"Fuhr Norbert","year":"2018","unstructured":"Norbert Fuhr . 2018. Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum, 51, 3, (Feb . 2018 ). doi: 10.1145\/3190580.3190586 (cited on p. 2). 10.1145\/3190580.3190586 Norbert Fuhr. 2018. Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum, 51, 3, (Feb. 2018). doi: 10.1145\/3190580.3190586 (cited on p. 2)."},{"key":"e_1_3_2_1_18_1","volume-title":"Information retrieval evaluation. Synthesis Lectures on Information Concepts, Retrieval, and Services, 3, 2, (May","author":"Harman Donna","year":"2011","unstructured":"Donna Harman . 2011. Information retrieval evaluation. Synthesis Lectures on Information Concepts, Retrieval, and Services, 3, 2, (May 2011 ), 1--119. doi: 10.2200\/S00368ED1V01Y201105ICR019 (cited on p. 1). 10.2200\/S00368ED1V01Y201105ICR019 Donna Harman. 2011. Information retrieval evaluation. Synthesis Lectures on Information Concepts, Retrieval, and Services, 3, 2, (May 2011), 1--119. doi: 10.2200\/S00368ED1V01Y201105ICR019 (cited on p. 1)."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646249"},{"key":"e_1_3_2_1_20_1","volume-title":"Tamhane","author":"Hochberg Yosef","year":"1987","unstructured":"Yosef Hochberg and Ajit C . Tamhane . 1987 . Multiple Comparison Procedures. John Wiley & Sons , USA. doi: 10.1002\/9780470316672 (cited on p. 4). 10.1002\/9780470316672 Yosef Hochberg and Ajit C. Tamhane. 1987. Multiple Comparison Procedures. John Wiley & Sons, USA. doi: 10.1002\/9780470316672 (cited on p. 4)."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348403"},{"volume-title":"Theory and methods","author":"Hsu Jason C.","key":"e_1_3_2_1_22_1","unstructured":"Jason C. Hsu . 1996. Multiple Comparisons . Theory and methods . Chapman and Hall\/CRC , USA. doi: 10.1201\/b15074 (cited on p. 4). 10.1201\/b15074 Jason C. Hsu. 1996. Multiple Comparisons. Theory and methods. Chapman and Hall\/CRC, USA. doi: 10.1201\/b15074 (cited on p. 4)."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/160688.160758"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"volume-title":"A new measure of rank correlation. Biometrika, 30, 1\/2, 81--93. doi: 10.2307\/2332226 (cited on","author":"Kendall Maurice G.","key":"e_1_3_2_1_25_1","unstructured":"Maurice G. Kendall . 1938. A new measure of rank correlation. Biometrika, 30, 1\/2, 81--93. doi: 10.2307\/2332226 (cited on p. 2). 10.2307\/2332226 Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30, 1\/2, 81--93. doi: 10.2307\/2332226 (cited on p. 2)."},{"volume-title":"Rank Correlation Methods","author":"Kendall Maurice G.","key":"e_1_3_2_1_26_1","unstructured":"Maurice G. Kendall . 1948. Rank Correlation Methods . Chales Griffin and Company Limited ( cited on p. 2). Maurice G. Kendall. 1948. Rank Correlation Methods. Chales Griffin and Company Limited (cited on p. 2)."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3133015"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851613.2851692"},{"key":"e_1_3_2_1_29_1","first-page":"2","volume-title":"Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing & Management, 53, 5, (Sept","author":"Losada David E.","year":"2017","unstructured":"David E. Losada , Javier Parapar , and \u00c1lvaro Barreiro . 2017. Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing & Management, 53, 5, (Sept . 2017 ), 1005--1025. doi: 10.1016\/j.ipm.2017.04.005 (cited on pp. 2 , 4). 10.1016\/j.ipm.2017.04.005 David E. Losada, Javier Parapar, and \u00c1lvaro Barreiro. 2017. Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing & Management, 53, 5, (Sept. 2017), 1005--1025. doi: 10.1016\/j.ipm.2017.04.005 (cited on pp. 2, 4)."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277824"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2407085.2407092"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277806"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3412841.3441945"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.24203"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313675"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-019-09357-w"},{"key":"e_1_3_2_1_37_1","series-title":"The Information Retrieval Series","volume-title":"Laboratory Experiments in Information Retrieval","author":"Sakai Tetsuya","unstructured":"Tetsuya Sakai . 2018. Laboratory Experiments in Information Retrieval . The Information Retrieval Series . Vol. 40 . Springer . isbn: 978--981--13--1198--7. doi: 10.1007\/978--981--13--1199--4 (cited on p. 4). 10.1007\/978--981--13--1199--4 Tetsuya Sakai. 2018. Laboratory Experiments in Information Retrieval. The Information Retrieval Series. Vol. 40. Springer. isbn: 978--981--13--1198--7. doi: 10.1007\/978--981--13--1199--4 (cited on p. 4)."},{"key":"e_1_3_2_1_38_1","volume-title":"Article 12","author":"Sakai Tetsuya","year":"2021","unstructured":"Tetsuya Sakai . 2021 . On Fuhr's guideline for IR evaluation. SIGIR Forum, 54, 1 , Article 12 , (Feb. 2021). doi: 10.1145\/3451964.3451976 (cited on p. 2). 10.1145\/3451964.3451976 Tetsuya Sakai. 2021. On Fuhr's guideline for IR evaluation. SIGIR Forum, 54, 1, Article 12, (Feb. 2021). doi: 10.1145\/3451964.3451976 (cited on p. 2)."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911492"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9273-z"},{"key":"e_1_3_2_1_41_1","volume-title":"Proceedings of the NTCIR-7 (cited on","author":"Sakai Tetsuya","year":"2008","unstructured":"Tetsuya Sakai , N. Kando , Chuan-Jie Lin , Teruko Mitamura , Hideki Shima , Dong-Hong Ji , Kuang-hua Chen, and Eric Nyberg . 2008 . Overview of the NTCIR-7 ACLIA IR4QA Task . In Proceedings of the NTCIR-7 (cited on p. 4). Tetsuya Sakai, N. Kando, Chuan-Jie Lin, Teruko Mitamura, Hideki Shima, Dong-Hong Ji, Kuang-hua Chen, and Eric Nyberg. 2008. Overview of the NTCIR-7 ACLIA IR4QA Task. In Proceedings of the NTCIR-7 (cited on p. 4)."},{"key":"e_1_3_2_1_42_1","volume-title":"Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4, 4, (June","author":"Sanderson Mark","year":"2010","unstructured":"Mark Sanderson . 2010. Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4, 4, (June 2010 ), 247--375. doi: 10.1561\/1500000009 (cited on p. 1). 10.1561\/1500000009 Mark Sanderson. 2010. Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4, 4, (June 2010), 247--375. doi: 10.1561\/1500000009 (cited on p. 1)."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076064"},{"key":"e_1_3_2_1_44_1","volume-title":"Statistical inference in retrieval effectiveness evaluation. Information Processing & Management, 33, 4, (July","author":"Savoy Jacques","year":"1997","unstructured":"Jacques Savoy . 1997. Statistical inference in retrieval effectiveness evaluation. Information Processing & Management, 33, 4, (July 1997 ), 495--512. doi: 10.1016\/S0306--4573(97)00027--7 (cited on p. 3). 10.1016\/S0306--4573(97)00027--7 Jacques Savoy. 1997. Statistical inference in retrieval effectiveness evaluation. Information Processing & Management, 33, 4, (July 1997), 495--512. doi: 10.1016\/S0306--4573(97)00027--7 (cited on p. 3)."},{"key":"e_1_3_2_1_45_1","volume-title":"van Rijsbergen","author":"Sp\u00e4rck Jones K.","year":"1975","unstructured":"K. Sp\u00e4rck Jones and Cornelis J . van Rijsbergen . 1975 . Report on the need for and provision of an 'ideal' information retrieval test collection. Computer Laboratory ( cited on p. 1). K. Sp\u00e4rck Jones and Cornelis J. van Rijsbergen. 1975. Report on the need for and provision of an 'ideal' information retrieval test collection. Computer Laboratory (cited on p. 1)."},{"key":"e_1_3_2_1_46_1","volume-title":"Barto","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G . Barto . 2018 . Reinforcement Learning : An Introduction. ( Second ed.). Adaptive computation and Machine Learning. MIT Press . isbn: 9780262039246 (cited on p. 2). Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. (Second ed.). Adaptive computation and Machine Learning. MIT Press. isbn: 9780262039246 (cited on p. 2)."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331259"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484163"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271766"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45691-0_34"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572138"},{"key":"e_1_3_2_1_52_1","volume-title":"Proceedings of ACM SIGIR","author":"Ellen","year":"2002","unstructured":"Ellen M. Voorhees and Chris Buckley. 2002. The effect of topic set size on retrieval experiment error . In Proceedings of ACM SIGIR 2002 . ACM, New York, NY, USA, 316--323. doi: 10.1145\/564376.564432 (cited on p. 6). 10.1145\/564376.564432 Ellen M. Voorhees and Chris Buckley. 2002. The effect of topic set size on retrieval experiment error. In Proceedings of ACM SIGIR 2002. ACM, New York, NY, USA, 316--323. doi: 10.1145\/564376.564432 (cited on p. 6)."},{"key":"e_1_3_2_1_53_1","volume-title":"Proceedings of ACM SIGIR","author":"Voorhees Ellen M.","year":"2022","unstructured":"Ellen M. Voorhees , Nick Craswell , and Jimmy Lin . 2022 . Too many relevants, whither cranfield test collections? In Proceedings of ACM SIGIR 2022. ACM, New York, NY, USA, 11 pages. doi: 10.1145\/3477495.3531728 (cited on p. 2). 10.1145\/3477495.3531728 Ellen M. Voorhees, Nick Craswell, and Jimmy Lin. 2022. Too many relevants, whither cranfield test collections? In Proceedings of ACM SIGIR 2022. ACM, New York, NY, USA, 11 pages. doi: 10.1145\/3477495.3531728 (cited on p. 2)."},{"volume-title":"Proceedings of TREC-8. NIST Special Publication 500--246","author":"Ellen","key":"e_1_3_2_1_54_1","unstructured":"Ellen M. Voorhees and Donna Harman. 2000. Overview of the eighth text retrieval conference (TREC-8) . In Proceedings of TREC-8. NIST Special Publication 500--246 , Gaithersburg, Maryland, USA, 1--24. doi: 10.6028\/NIST.SP.500--246 (cited on p. 4). 10.6028\/NIST.SP.500--246 Ellen M. Voorhees and Donna Harman. 2000. Overview of the eighth text retrieval conference (TREC-8). In Proceedings of TREC-8. NIST Special Publication 500--246, Gaithersburg, Maryland, USA, 1--24. doi: 10.6028\/NIST.SP.500--246 (cited on p. 4)."},{"key":"e_1_3_2_1_55_1","first-page":"1","volume-title":"Harman","author":"Voorhees Ellen M.","year":"2005","unstructured":"Ellen M. Voorhees and Donna K . Harman . 2005 . TREC : Experiment and Evaluation in Information Retrieval. The MIT Press . isbn: 0262220733 (cited on pp. 1 , 4). Ellen M. Voorhees and Donna K. Harman. 2005. TREC: Experiment and Evaluation in Information Retrieval. The MIT Press. isbn: 0262220733 (cited on pp. 1, 4)."},{"key":"e_1_3_2_1_56_1","first-page":"2","volume-title":"Can old TREC collections reliably evaluate modern neural retrieval models? https:\/\/arxiv.org\/abs\/2201.11086 arXiv: 2201.11086 (cited on","author":"Voorhees Ellen M.","unstructured":"Ellen M. Voorhees , Ian Soboroff , and Jimmy Lin . 2022. Can old TREC collections reliably evaluate modern neural retrieval models? https:\/\/arxiv.org\/abs\/2201.11086 arXiv: 2201.11086 (cited on pp. 2 , 4). Ellen M. Voorhees, Ian Soboroff, and Jimmy Lin. 2022. Can old TREC collections reliably evaluate modern neural retrieval models? https:\/\/arxiv.org\/abs\/2201.11086 arXiv: 2201.11086 (cited on pp. 2, 4)."},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458158"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390435"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"event":{"name":"CIKM '23: The 32nd ACM International Conference on Information and Knowledge Management","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Birmingham United Kingdom","acronym":"CIKM '23"},"container-title":["Proceedings of the 32nd ACM International Conference on Information and Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583780.3614916","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3583780.3614916","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:43Z","timestamp":1750178203000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583780.3614916"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,21]]},"references-count":59,"alternative-id":["10.1145\/3583780.3614916","10.1145\/3583780"],"URL":"https:\/\/doi.org\/10.1145\/3583780.3614916","relation":{},"subject":[],"published":{"date-parts":[[2023,10,21]]},"assertion":[{"value":"2023-10-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}