{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T20:01:10Z","timestamp":1768593670799,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":48,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,10,17]],"date-time":"2018-10-17T00:00:00Z","timestamp":1539734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Qatar National Research Fund","award":["NPRP 7-1313-1-245"],"award-info":[{"award-number":["NPRP 7-1313-1-245"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,10,17]]},"DOI":"10.1145\/3269206.3271751","type":"proceedings-article","created":{"date-parts":[[2018,10,22]],"date-time":"2018-10-22T12:08:27Z","timestamp":1540210107000},"page":"397-406","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["When Rank Order Isn't Enough"],"prefix":"10.1145","author":[{"given":"Mucahid","family":"Kutlu","sequence":"first","affiliation":[{"name":"Qatar University, Doha, Qatar"}]},{"given":"Tamer","family":"Elsayed","sequence":"additional","affiliation":[{"name":"Qatar University, Doha, Qatar"}]},{"given":"Maram","family":"Hasanain","sequence":"additional","affiliation":[{"name":"Qatar University, Doha, Qatar"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,10,17]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/956863.956953"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009000"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572017"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080738"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_3_2_1_8_1","volume-title":"Overview of the TREC 2010 Web track. In TREC 2010 .","author":"Clarke Charles L","year":"2010","unstructured":"Charles L Clarke , Nick Craswell , Ian Soboroff , and Gordon Cormack . 2010 a. Overview of the TREC 2010 Web track. In TREC 2010 . Charles L Clarke, Nick Craswell, Ian Soboroff, and Gordon Cormack. 2010a. Overview of the TREC 2010 Web track. In TREC 2010 ."},{"key":"e_1_3_2_1_9_1","volume-title":"Overview of the TREC 2011 Web track. In TREC 2011 .","author":"Clarke Charles L","unstructured":"Charles L Clarke , Nick Craswell , Ian Soboroff , and Ellen M. Voorhees . 2010b . Overview of the TREC 2011 Web track. In TREC 2011 . Charles L Clarke, Nick Craswell, Ian Soboroff, and Ellen M. Voorhees. 2010b. Overview of the TREC 2011 Web track. In TREC 2011 ."},{"key":"e_1_3_2_1_10_1","volume-title":"Overview of the TREC 2012 Web Track. In TREC 2012 .","author":"Clarke Charles L.","unstructured":"Charles L. Clarke , Nick Craswell , and Ellen M. Voorhees . 2012 . Overview of the TREC 2012 Web Track. In TREC 2012 . Charles L. Clarke, Nick Craswell, and Ellen M. Voorhees. 2012. Overview of the TREC 2012 Web Track. In TREC 2012 ."},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the international conference on scientific information","volume":"1","author":"Cleverdon Cyril W","year":"1959","unstructured":"Cyril W Cleverdon . 1959 . The evaluation of systems used in information retrieval . In Proceedings of the international conference on scientific information , Vol. 1 . National Academy of Sciences Washington, DC,, 687--698. Cyril W Cleverdon. 1959. The evaluation of systems used in information retrieval. In Proceedings of the international conference on scientific information, Vol. 1. National Academy of Sciences Washington, DC,, 687--698."},{"key":"e_1_3_2_1_12_1","volume-title":"TREC 2013 Web Track Overview. In TREC 2013 .","author":"Collins-Thompson Kevyn","year":"2013","unstructured":"Kevyn Collins-Thompson , Paul Bennett , Fernando Diaz , Charles L A Clarke , and Ellen M Voorhees . 2013 . TREC 2013 Web Track Overview. In TREC 2013 . Kevyn Collins-Thompson, Paul Bennett, Fernando Diaz, Charles L A Clarke, and Ellen M Voorhees. 2013. TREC 2013 Web Track Overview. In TREC 2013 ."},{"key":"e_1_3_2_1_13_1","volume-title":"TREC 2014 web track overview.","author":"Collins-Thompson Kevyn","year":"2015","unstructured":"Kevyn Collins-Thompson , Craig Macdonald , Paul Bennett , Fernando Diaz , and Ellen M Voorhees . 2015 . TREC 2014 web track overview. Kevyn Collins-Thompson, Craig Macdonald, Paul Bennett, Fernando Diaz, and Ellen M Voorhees. 2015. TREC 2014 web track overview."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277934"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291009"},{"key":"e_1_3_2_1_16_1","unstructured":"Giorgio Maria Di Nunzio and Gianmaria Silvello. 2015. A Graphical View of Distance Between Rankings: The Point and Area Measures.. In IIR .  Giorgio Maria Di Nunzio and Gianmaria Silvello. 2015. A Graphical View of Distance Between Rankings: The Point and Area Measures.. In IIR ."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0895480102412856"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914728"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767793"},{"key":"e_1_3_2_1_20_1","volume-title":"ECIR '14 .","author":"Gao Ning","unstructured":"Ning Gao , William Webber , and Douglas W. Oard . 2014. Reducing reliance on relevance judgments for system comparison by using expectation-maximization . In ECIR '14 . Ning Gao, William Webber, and Douglas W. Oard. 2014. Reducing reliance on relevance judgments for system comparison by using expectation-maximization. In ECIR '14 ."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Donna Harman. 2011. Information retrieval evaluation . Number 19 in Synthesis lectures on information concepts retrieval and services. Morgan & Claypool San Rafael Calif.   Donna Harman. 2011. Information retrieval evaluation . Number 19 in Synthesis lectures on information concepts retrieval and services. Morgan & Claypool San Rafael Calif.","DOI":"10.1007\/978-3-031-02276-0"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-12275-0_16"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"crossref","unstructured":"David Hawking. 2000. Overview of the TREC-9 Web Track. In TREC 9 .  David Hawking. 2000. Overview of the TREC-9 Web Track. In TREC 9 .","DOI":"10.1145\/344250.344254"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-23525-7_26"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177731867"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.2307\/1269344"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772749"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210033"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2017.04.005"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.3844\/jmssp.2008.226.230"},{"key":"e_1_3_2_1_33_1","volume-title":"HCOMP '16","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell , Matthew Lease , Mucahid Kutlu , and Tamer Elsayed . 2016 . Why is that relevant? Collecting annotator rationales for relevance judgments . In HCOMP '16 . 139--148. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? Collecting annotator rationales for relevance judgments. In HCOMP '16. 139--148."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04769-5_7"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2005.03.023"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1098\/rspl.1895.0041"},{"key":"e_1_3_2_1_37_1","volume-title":"Herbert Robbins Selected Papers","author":"Robbins Herbert","unstructured":"Herbert Robbins . 1985. Some aspects of the sequential design of experiments . In Herbert Robbins Selected Papers . Springer , 169--177. Herbert Robbins. 1985. Some aspects of the sequential design of experiments. In Herbert Robbins Selected Papers . Springer, 169--177."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277756"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2641383.2641385"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277935"},{"key":"e_1_3_2_1_41_1","volume-title":"A weighted Kendall's tau statistic. Statistics & probability letters","author":"Shieh Grace S","year":"1998","unstructured":"Grace S Shieh . 1998. A weighted Kendall's tau statistic. Statistics & probability letters , Vol. 39 , 1 ( 1998 ), 17--24. Grace S Shieh. 1998. A weighted Kendall's tau statistic. Statistics & probability letters , Vol. 39, 1 (1998), 17--24."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321528"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148220"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2015.2448541"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741088"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_3_2_1_48_1","volume-title":"Overview of TREC 2001. In TREC 2001 .","author":"Voorhees Ellen M","unstructured":"Ellen M Voorhees and Donna Harman . {n. d.}. Overview of TREC 2001. In TREC 2001 . Ellen M Voorhees and Donna Harman. {n. d.}. Overview of TREC 2001. In TREC 2001 ."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183633"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390435"}],"event":{"name":"CIKM '18: The 27th ACM International Conference on Information and Knowledge Management","location":"Torino Italy","acronym":"CIKM '18","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 27th ACM International Conference on Information and Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3269206.3271751","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3269206.3271751","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:29Z","timestamp":1750207469000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3269206.3271751"}},"subtitle":["New Statistical-Significance-Aware Correlation Measures"],"short-title":[],"issued":{"date-parts":[[2018,10,17]]},"references-count":48,"alternative-id":["10.1145\/3269206.3271751","10.1145\/3269206"],"URL":"https:\/\/doi.org\/10.1145\/3269206.3271751","relation":{},"subject":[],"published":{"date-parts":[[2018,10,17]]},"assertion":[{"value":"2018-10-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}