{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:11:18Z","timestamp":1784178678211,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":42,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,7,25]],"date-time":"2020-07-25T00:00:00Z","timestamp":1595635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"German Research Foundation (DFG)","award":["407518790"],"award-info":[{"award-number":["407518790"]}]},{"name":"Innovationsfonden Denmark","award":["AMAOS (Advanced Machine Learning for Automatic Omni-Channel Supp"],"award-info":[{"award-number":["AMAOS (Advanced Machine Learning for Automatic Omni-Channel Supp"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,7,25]]},"DOI":"10.1145\/3397271.3401036","type":"proceedings-article","created":{"date-parts":[[2020,7,25]],"date-time":"2020-07-25T07:50:08Z","timestamp":1595663408000},"page":"349-358","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["How to Measure the Reproducibility of System-oriented IR Experiments"],"prefix":"10.1145","author":[{"given":"Timo","family":"Breuer","sequence":"first","affiliation":[{"name":"TH K\u00f6ln, Cologne, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"University of Padua, Padua, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Norbert","family":"Fuhr","sequence":"additional","affiliation":[{"name":"Universit\u00e4t Duisburg-Essen, Duisburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Maistro","sequence":"additional","affiliation":[{"name":"University of Copenhagen, Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tetsuya","family":"Sakai","sequence":"additional","affiliation":[{"name":"Waseda University, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philipp","family":"Schaer","sequence":"additional","affiliation":[{"name":"TH K\u00f6ln, Cologne, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ian","family":"Soboroff","sequence":"additional","affiliation":[{"name":"National Institute of Standards and Technology (NIST), Gaithersburg, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,7,25]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"TREC 2017 Common Core Track Overview. In The Twenty-Sixth Text REtrieval Conference Proceedings (TREC","author":"Allan J.","year":"2017","unstructured":"J. Allan , D. K. Harman , E. Kanoulas , D. Li , C. Van Gysel , and E. M. Voorhees . 2018 . TREC 2017 Common Core Track Overview. In The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017 ), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500--324, Washington, USA. J. Allan, D. K. Harman, E. Kanoulas, D. Li, C. Van Gysel, and E. M. Voorhees. 2018. TREC 2017 Common Core Track Overview. In The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500--324, Washington, USA."},{"key":"e_1_3_2_2_2_1","first-page":"2","article-title":"Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR)","volume":"49","author":"Arguello J.","year":"2015","unstructured":"J. Arguello , M. Crane , F. Diaz , J. Lin , and A. Trotman . 2015 . Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR) . SIGIR Forum , Vol. 49 , 2 (December 2015), 107--116. J. Arguello, M. Crane, F. Diaz, J. Lin, and A. Trotman. 2015. Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR). SIGIR Forum, Vol. 49, 2 (December 2015), 107--116.","journal-title":"SIGIR Forum"},{"key":"e_1_3_2_2_3_1","volume-title":"Nature","volume":"533","author":"Baker M.","year":"2016","unstructured":"M. Baker . 2016 . 1,500 Scientists Lift the Lid on Reproducibility . Nature , Vol. 533 (May 2016), 452--454. M. Baker. 2016. 1,500 Scientists Lift the Lid on Reproducibility. Nature, Vol. 533 (May 2016), 452--454."},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"crossref","unstructured":"T. Breuer N. Ferro N. Fuhr M. Maistro T. Sakai P. Schaer and I. Soboroff. 2020. How to Measure the Reproducibility of System-oriented IR Experiments. https:\/\/doi.org\/10.5281\/zenodo.3856042  T. Breuer N. Ferro N. Fuhr M. Maistro T. Sakai P. Schaer and I. Soboroff. 2020. How to Measure the Reproducibility of System-oriented IR Experiments. https:\/\/doi.org\/10.5281\/zenodo.3856042","DOI":"10.1145\/3397271.3401036"},{"key":"e_1_3_2_2_5_1","unstructured":"T. Breuer and P. Schaer. 2019. Replicability and Reproducibility of Automatic Routing Runs. In Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum Lugano Switzerland September 9-12 2019 (CEUR Workshop Proceedings) Linda Cappellato Nicola Ferro David E. Losada and Henning M\u00fc ller (Eds.) Vol. 2380. CEUR-WS.org. http:\/\/ceur-ws.org\/Vol-2380\/paper_84.pdf  T. Breuer and P. Schaer. 2019. Replicability and Reproducibility of Automatic Routing Runs. In Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum Lugano Switzerland September 9-12 2019 (CEUR Workshop Proceedings) Linda Cappellato Nicola Ferro David E. Losada and Henning M\u00fc ller (Eds.) Vol. 2380. CEUR-WS.org. http:\/\/ceur-ws.org\/Vol-2380\/paper_84.pdf"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334"},{"key":"e_1_3_2_2_7_1","volume-title":"Proc. of the Open-Source IR Replicability Challenge (OSIRRC 2019),, R. Clancy, N. Ferro, C. Hauff, T. Sakai, and Z. Z. Wu (Eds.). CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073","author":"Clancy R.","unstructured":"R. Clancy , N. Ferro , C. Hauff , T. Sakai , and Z. Z. Wu . 2019. Overview of the 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) . In Proc. of the Open-Source IR Replicability Challenge (OSIRRC 2019),, R. Clancy, N. Ferro, C. Hauff, T. Sakai, and Z. Z. Wu (Eds.). CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073 , http:\/\/ceur-ws.org\/Vol-2409\/, 1--7. R. Clancy, N. Ferro, C. Hauff, T. Sakai, and Z. Z. Wu. 2019. Overview of the 2019 Open-Source IR Replicability Challenge (OSIRRC 2019). In Proc. of the Open-Source IR Replicability Challenge (OSIRRC 2019),, R. Clancy, N. Ferro, C. Hauff, T. Sakai, and Z. Z. Wu (Eds.). CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http:\/\/ceur-ws.org\/Vol-2409\/, 1--7."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00018"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10260-010-0142-z"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_2_2_11_1","first-page":"3","article-title":"The Future of Scholarly Communications","volume":"27","author":"Roure D. De","year":"2014","unstructured":"D. De Roure . 2014 . The Future of Scholarly Communications . Insights , Vol. 27 , 3 (November 2014), 233--238. D. De Roure. 2014. The Future of Scholarly Communications. Insights, Vol. 27, 3 (November 2014), 233--238.","journal-title":"Insights"},{"key":"e_1_3_2_2_12_1","volume-title":"Proc. 1st ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR","author":"Ferrante M.","year":"2015","unstructured":"M. Ferrante , N. Ferro , and M. Maistro . 2015. Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness . In Proc. 1st ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2015 ),, J. Allan, W. B. Croft, A. P. de Vries, C. Zhai, N. Fuhr, and Y. Zhang (Eds.). ACM Press, New York, USA, 21--30. M. Ferrante, N. Ferro, and M. Maistro. 2015. Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness. In Proc. 1st ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2015),, J. Allan, W. B. Croft, A. P. de Vries, C. Zhai, N. Fuhr, and Y. Zhang (Eds.). ACM Press, New York, USA, 21--30."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106371"},{"key":"e_1_3_2_2_14_1","volume-title":"Proceedings of the Tenth International Conference of the CLEF Association (CLEF","author":"Ferro N.","year":"2019","unstructured":"N. Ferro , N. Fuhr , M. Maistro , T. Sakai , and I. Soboroff . 2019. Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality, Multimodality, and Interaction . Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019 ),, F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro (Eds.). Lecture Notes in Computer Science (LNCS) 11696, Springer, Heidelberg, Germany, 287--300. N. Ferro, N. Fuhr, M. Maistro, T. Sakai, and I. Soboroff. 2019. Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019),, F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro (Eds.). Lecture Notes in Computer Science (LNCS) 11696, Springer, Heidelberg, Germany, 287--300."},{"key":"e_1_3_2_2_15_1","first-page":"3","article-title":"Introduction to the Special Issue on Reproducibility in Information Retrieval: Evaluation Campaigns, Collections, and Analyses","volume":"10","author":"Ferro N.","year":"2018","unstructured":"N. Ferro , N. Fuhr , and A. Rauber . 2018 a. Introduction to the Special Issue on Reproducibility in Information Retrieval: Evaluation Campaigns, Collections, and Analyses . ACM Journal of Data and Information Quality (JDIQ) , Vol. 10 , 3 (October 2018), 9:1--9:4. N. Ferro, N. Fuhr, and A. Rauber. 2018a. Introduction to the Special Issue on Reproducibility in Information Retrieval: Evaluation Campaigns, Collections, and Analyses. ACM Journal of Data and Information Quality (JDIQ), Vol. 10, 3 (October 2018), 9:1--9:4.","journal-title":"ACM Journal of Data and Information Quality (JDIQ)"},{"key":"e_1_3_2_2_16_1","first-page":"4","article-title":"Introduction to the Special Issue on Reproducibility in Information Retrieval: Tools and Infrastructures","volume":"10","author":"Ferro N.","year":"2018","unstructured":"N. Ferro , N. Fuhr , and A. Rauber . 2018 b. Introduction to the Special Issue on Reproducibility in Information Retrieval: Tools and Infrastructures . ACM Journal of Data and Information Quality (JDIQ) , Vol. 10 , 4 (November 2018), 14:1--14:4. N. Ferro, N. Fuhr, and A. Rauber. 2018b. Introduction to the Special Issue on Reproducibility in Information Retrieval: Tools and Infrastructures. ACM Journal of Data and Information Quality (JDIQ), Vol. 10, 4 (November 2018), 14:1--14:4.","journal-title":"ACM Journal of Data and Information Quality (JDIQ)"},{"key":"e_1_3_2_2_17_1","first-page":"1","article-title":"SIGIR Initiative to Implement ACM Artifact Review and Badging","volume":"52","author":"Ferro N.","year":"2018","unstructured":"N. Ferro and D. Kelly . 2018 . SIGIR Initiative to Implement ACM Artifact Review and Badging . SIGIR Foru , Vol. 52 , 1 (June 2018), 4--10. N. Ferro and D. Kelly. 2018. SIGIR Initiative to Implement ACM Artifact Review and Badging. SIGIR Foru, Vol. 52, 1 (June 2018), 4--10.","journal-title":"SIGIR Foru"},{"key":"e_1_3_2_2_18_1","volume-title":"Proceedings of the Nineth International Conference of the CLEF Association (CLEF","author":"Ferro N.","year":"2018","unstructured":"N. Ferro , M. Maistro , T. Sakai , and I. Soboroff . 2018c. Overview of CENTRE@CLEF 2018: a First Tale in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality, Multimodality, and Interaction . Proceedings of the Nineth International Conference of the CLEF Association (CLEF 2018 ),, P. Bellot, C. Trabelsi, J. Mothe, F. Murtagh, J.-Y. Nie, L. Soulier, E. SanJuan, L. Cappellato, and N. Ferro (Eds.). Lecture Notes in Computer Science (LNCS) 11018, Springer, Heidelberg, Germany, 239--246. N. Ferro, M. Maistro, T. Sakai, and I. Soboroff. 2018c. Overview of CENTRE@CLEF 2018: a First Tale in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Nineth International Conference of the CLEF Association (CLEF 2018),, P. Bellot, C. Trabelsi, J. Mothe, F. Murtagh, J.-Y. Nie, L. Soulier, E. SanJuan, L. Cappellato, and N. Ferro (Eds.). Lecture Notes in Computer Science (LNCS) 11018, Springer, Heidelberg, Germany, 239--246."},{"key":"e_1_3_2_2_19_1","volume-title":"The Information Retrieval Series","volume":"41","author":"Ferro N.","year":"2019","unstructured":"N. Ferro and C. Peters ( Eds .). 2019 . Information Retrieval Evaluation in a Changing World -- Lessons Learned from 20 Years of CLEF . The Information Retrieval Series , Vol. 41 . Springer International Publishing, Germany. N. Ferro and C. Peters (Eds.). 2019. Information Retrieval Evaluation in a Changing World -- Lessons Learned from 20 Years of CLEF. The Information Retrieval Series, Vol. 41. Springer International Publishing, Germany."},{"key":"e_1_3_2_2_20_1","first-page":"3","article-title":"Some Common Mistakes In IR Evaluation, And How They Can Be Avoided","volume":"51","author":"Fuhr N.","year":"2017","unstructured":"N. Fuhr . 2017 . Some Common Mistakes In IR Evaluation, And How They Can Be Avoided . SIGIR Forum , Vol. 51 , 3 (December 2017), 32--41. N. Fuhr. 2017. Some Common Mistakes In IR Evaluation, And How They Can Be Avoided. SIGIR Forum, Vol. 51, 3 (December 2017), 32--41.","journal-title":"SIGIR Forum"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"crossref","unstructured":"N. Fuhr. 2019. Reproducibility and Validity in CLEF See citeNzz-FerroPeters2019-editor.  N. Fuhr. 2019. Reproducibility and Validity in CLEF See citeNzz-FerroPeters2019-editor.","DOI":"10.1007\/978-3-030-22948-1_23"},{"key":"e_1_3_2_2_22_1","volume-title":"Nature","volume":"577","author":"Gibney E.","year":"2020","unstructured":"E. Gibney . 2020 . This AI researcher is trying to ward off a reproducibility crisis . Nature , Vol. 577 (January 2020), 14. E. Gibney. 2020. This AI researcher is trying to ward off a reproducibility crisis. Nature, Vol. 577 (January 2020), 14."},{"key":"e_1_3_2_2_23_1","volume-title":"Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017","author":"Maura","year":"2017","unstructured":"Maura R. Grossman and Gordon V. Cormack. 2017. MRG_UWaterloo and WaterlooCormack Participation in the TREC 2017 Common Core Track . In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017 , Gaithersburg, Maryland, USA , November 15-17, 2017 ,, Ellen M. Voorhees and Angela Ellis (Eds.), Vol. Special Publication 500--324. National Institute of Standards and Technology (NIST). https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/MRG_UWaterloo-CC.pdf Maura R. Grossman and Gordon V. Cormack. 2017. MRG_UWaterloo and WaterlooCormack Participation in the TREC 2017 Common Core Track. In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017, Gaithersburg, Maryland, USA, November 15-17, 2017,, Ellen M. Voorhees and Angela Ellis (Eds.), Vol. Special Publication 500--324. National Institute of Standards and Technology (NIST). https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/MRG_UWaterloo-CC.pdf"},{"key":"e_1_3_2_2_24_1","unstructured":"ISO 5725-2:2019. 2019. Accuracy (Trueness and Precision) of Measurement Methods and Results -- Part 2: Basic Method for the Determination of Repeatability and Reproducibility of a Standard Measurement method. Recommendation ISO\/IEC 5725--2:2019.  ISO 5725-2:2019. 2019. Accuracy (Trueness and Precision) of Measurement Methods and Results -- Part 2: Basic Method for the Determination of Repeatability and Reproducibility of a Standard Measurement method. Recommendation ISO\/IEC 5725--2:2019."},{"key":"e_1_3_2_2_25_1","volume-title":"Rank correlation methods","author":"Kendall M. G.","unstructured":"M. G. Kendall . 1948. Rank correlation methods . Griffin , Oxford, England . M. G. Kendall. 1948. Rank correlation methods. Griffin, Oxford, England."},{"key":"e_1_3_2_2_26_1","unstructured":"J. F. Kenney and E. S. Keeping. 1954. Mathematics of Statistics -- Part One 3rd ed.). D. Van Nostrand Company Princeton USA.  J. F. Kenney and E. S. Keeping. 1954. Mathematics of Statistics -- Part One 3rd ed.). D. Van Nostrand Company Princeton USA."},{"key":"e_1_3_2_2_27_1","volume-title":"Proc. 38th European Conference on IR Research (ECIR","author":"Lin J.","year":"2016","unstructured":"J. Lin , M. Crane , A. Trotman , J. Callan , I. Chattopadhyaya , J. Foley , G. Ingersoll , C. Macdonald , and S. Vigna . 2016. Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge. In Advances in Information Retrieval . Proc. 38th European Conference on IR Research (ECIR 2016 ), N. Ferro, F. Crestani, M.-F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff, and G. Silvello (Eds.). Lecture Notes in Computer Science (LNCS) 9626, Springer, Heidelberg, Germany, 357--368. J. Lin, M. Crane, A. Trotman, J. Callan, I. Chattopadhyaya, J. Foley, G. Ingersoll, C. Macdonald, and S. Vigna. 2016. Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge. In Advances in Information Retrieval. Proc. 38th European Conference on IR Research (ECIR 2016), N. Ferro, F. Crestani, M.-F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff, and G. Silvello (Eds.). Lecture Notes in Computer Science (LNCS) 9626, Springer, Heidelberg, Germany, 357--368."},{"key":"e_1_3_2_2_28_1","volume-title":"Engineering, and Medicine","author":"National Academies of Sciences","year":"2016","unstructured":"National Academies of Sciences , Engineering, and Medicine . 2016 . Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. The National Academies Press , Washington, USA. National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. The National Academies Press, Washington, USA."},{"key":"e_1_3_2_2_29_1","volume-title":"Engineering, and Medicine","author":"National Academies of Sciences","year":"2019","unstructured":"National Academies of Sciences , Engineering, and Medicine . 2019 . Reproducibility and Replicability in Science. The National Academies Press , Washington, USA. National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. The National Academies Press, Washington, USA."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"crossref","first-page":"6251","DOI":"10.1126\/science.aac4716","article-title":"Estimating the Reproducibility of Psychological Science","volume":"349","author":"Collaboration Open Science","year":"2015","unstructured":"Open Science Collaboration . 2015 . Estimating the Reproducibility of Psychological Science . Science , Vol. 349 , 6251 (August 2015), 943--952. Open Science Collaboration. 2015. Estimating the Reproducibility of Psychological Science. Science, Vol. 349, 6251 (August 2015), 943--952.","journal-title":"Science"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2017.00076"},{"key":"e_1_3_2_2_32_1","unstructured":"S. Robertson and J. Callan. 2005. Routing and Filtering. In TREC: Experiment and Evaluation in Information Retrieval E. M. Voorhees and D. K. Harman (Eds.). MIT Press Cambridge Massachusetts 99--122.  S. Robertson and J. Callan. 2005. Routing and Filtering. In TREC: Experiment and Evaluation in Information Retrieval E. M. Voorhees and D. K. Harman (Eds.). MIT Press Cambridge Massachusetts 99--122."},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914684"},{"key":"e_1_3_2_2_34_1","series-title":"The Information Retrieval Series","volume-title":"Laboratory Experiments in Information Retrieval","author":"Sakai T.","unstructured":"T. Sakai . 2018. Laboratory Experiments in Information Retrieval . The Information Retrieval Series , Vol. 40 . Springer Singapore . T. Sakai. 2018. Laboratory Experiments in Information Retrieval. The Information Retrieval Series, Vol. 40. Springer Singapore."},{"key":"e_1_3_2_2_35_1","volume-title":"Proc. 14th NTCIR Conference on Evaluation of Information Access Technologies,, E. Ishita, N. Kando, M. P. Kato, and Y. Liu (Eds.). National Institute of Informatics","author":"Sakai T.","unstructured":"T. Sakai , N. Ferro , I. Soboroff , Z. Zeng , P. Xiao , and M. Maistro . 2019. Overview of the NTCIR-14 CENTRE Task . In Proc. 14th NTCIR Conference on Evaluation of Information Access Technologies,, E. Ishita, N. Kando, M. P. Kato, and Y. Liu (Eds.). National Institute of Informatics , Tokyo, Japan, 494--509. T. Sakai, N. Ferro, I. Soboroff, Z. Zeng, P. Xiao, and M. Maistro. 2019. Overview of the NTCIR-14 CENTRE Task. In Proc. 14th NTCIR Conference on Evaluation of Information Access Technologies,, E. Ishita, N. Kando, M. P. Kato, and Y. Liu (Eds.). National Institute of Informatics, Tokyo, Japan, 494--509."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277935"},{"key":"e_1_3_2_2_37_1","volume-title":"Overview of the TREC 2018 CENTRE Track. In The Twenty-Seventh Text REtrieval Conference Proceedings (TREC","author":"Soboroff I.","year":"2018","unstructured":"I. Soboroff , N. Ferro , M. Maistro , and T. Sakai . 2019 . Overview of the TREC 2018 CENTRE Track. In The Twenty-Seventh Text REtrieval Conference Proceedings (TREC 2018 ), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500--331, Washington, USA. I. Soboroff, N. Ferro, M. Maistro, and T. Sakai. 2019. Overview of the TREC 2018 CENTRE Track. In The Twenty-Seventh Text REtrieval Conference Proceedings (TREC 2018), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500--331, Washington, USA."},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/6.1.1"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291017"},{"key":"e_1_3_2_2_40_1","first-page":"28","article-title":"A Similarity Measure for Indefinite Rankings","volume":"4","author":"Webber W.","year":"2010","unstructured":"W. Webber , A. Moffat , and J. Zobel . 2010 . A Similarity Measure for Indefinite Rankings . ACM Transactions on Information Systems (TOIS) , Vol. 4 , 28 (November 2010), 20:1--20:38. W. Webber, A. Moffat, and J. Zobel. 2010. A Similarity Measure for Indefinite Rankings. ACM Transactions on Information Systems (TOIS), Vol. 4, 28 (November 2010), 20:1--20:38.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"e_1_3_2_2_41_1","unstructured":"W. Webber A. Moffat J. Zobel and T. Sakai. 2008. Precision-at-ten Considered Redundant See citeNzz-SIGIR2008 695--696.  W. Webber A. Moffat J. Zobel and T. Sakai. 2008. Precision-at-ten Considered Redundant See citeNzz-SIGIR2008 695--696."},{"key":"e_1_3_2_2_42_1","unstructured":"E. Yilmaz J. A. Aslam and S. E. Robertson. 2008. A New Rank Correlation Coefficient for Information Retrieval See citeNzz-SIGIR2008 587--594.  E. Yilmaz J. A. Aslam and S. E. Robertson. 2008. A New Rank Correlation Coefficient for Information Retrieval See citeNzz-SIGIR2008 587--594."}],"event":{"name":"SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval","location":"Virtual Event China","acronym":"SIGIR '20","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397271.3401036","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3397271.3401036","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:37Z","timestamp":1750195897000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397271.3401036"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,25]]},"references-count":42,"alternative-id":["10.1145\/3397271.3401036","10.1145\/3397271"],"URL":"https:\/\/doi.org\/10.1145\/3397271.3401036","relation":{},"subject":[],"published":{"date-parts":[[2020,7,25]]},"assertion":[{"value":"2020-07-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}