{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T23:34:42Z","timestamp":1774308882759,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":70,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T00:00:00Z","timestamp":1657065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Deutsche Forschungsgemeinschaft (DFG)","award":["407518790"],"award-info":[{"award-number":["407518790"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,6]]},"DOI":"10.1145\/3477495.3531738","type":"proceedings-article","created":{"date-parts":[[2022,7,7]],"date-time":"2022-07-07T15:12:13Z","timestamp":1657206733000},"page":"3078-3089","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["ir_metadata: An Extensible Metadata Schema for IR Experiments"],"prefix":"10.1145","author":[{"given":"Timo","family":"Breuer","sequence":"first","affiliation":[{"name":"TH K\u00f6ln, K\u00f6ln, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J\u00fcri","family":"Keller","sequence":"additional","affiliation":[{"name":"TH K\u00f6ln, K\u00f6ln, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philipp","family":"Schaer","sequence":"additional","affiliation":[{"name":"TH K\u00f6ln, K\u00f6ln, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,7,7]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006","author":"Agosti M.","year":"2006","unstructured":"M. Agosti , G. M. Di Nunzio , and N. Ferro . 2006. Scientific Data of an Evaluation Campaign: Do We Properly Deal with Them?. In Evaluation of Multilingual and Multi-modal Information Retrieval , 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006 , Alicante, Spain, September 20--22 , 2006 , Revised Selected Papers (Lecture Notes in Computer Science, Vol. 4730), , C. Peters, P. D. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, and M. Stempfhuber (Eds.). Springer, 11--20. https:\/\/doi.org\/10.1007\/978--3--540--74999--8_2 10.1007\/978--3--540--74999--8_2 M. Agosti, G. M. Di Nunzio, and N. Ferro. 2006. Scientific Data of an Evaluation Campaign: Do We Properly Deal with Them?. In Evaluation of Multilingual and Multi-modal Information Retrieval, 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain, September 20--22, 2006, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 4730), , C. Peters, P. D. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, and M. Stempfhuber (Eds.). Springer, 11--20. https:\/\/doi.org\/10.1007\/978--3--540--74999--8_2"},{"key":"e_1_3_2_2_2_1","volume-title":"Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017","author":"Allan J.","year":"2017","unstructured":"J. Allan , D. Harman , E. Kanoulas , D. Li , C. Van Gysel , and E. M. Voorhees . 2017. TREC 2017 Common Core Track Overview . In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017 , Gaithersburg, Maryland, USA, November 15--17 , 2017 (NIST Special Publication, Vol. 500--324), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/Overview-CC.pdf J. Allan, D. Harman, E. Kanoulas, D. Li, C. Van Gysel, and E. M. Voorhees. 2017. TREC 2017 Common Core Track Overview. In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017, Gaithersburg, Maryland, USA, November 15--17, 2017 (NIST Special Publication, Vol. 500--324), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/Overview-CC.pdf"},{"key":"e_1_3_2_2_3_1","volume-title":"Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009","author":"Armstrong T. G.","year":"2009","unstructured":"T. G. Armstrong , A. Moffat , W. Webber , and J. Zobel . 2009a. EvaluatIR: an online tool for evaluating and comparing IR systems . In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009 , Boston, MA, USA, July 19--23 , 2009 , , J. Allan, J. A. Aslam, M. Sanderson, C. Zhai, and J. Zobel (Eds.). ACM , 833. https:\/\/doi.org\/10.1145\/1571941.1572153 10.1145\/1571941.1572153 T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. 2009a. EvaluatIR: an online tool for evaluating and comparing IR systems. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19--23, 2009 , , J. Allan, J. A. Aslam, M. Sanderson, C. Zhai, and J. Zobel (Eds.). ACM , 833. https:\/\/doi.org\/10.1145\/1571941.1572153"},{"key":"e_1_3_2_2_4_1","volume-title":"Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009","author":"Armstrong T. G.","year":"2009","unstructured":"T. G. Armstrong , A. Moffat , W. Webber , and J. Zobel . 2009b. Improvements that don't add up: ad-hoc retrieval results since 1998 . In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009 , Hong Kong, China, November 2--6 , 2009 , D. W. Cheung, I. Song, W. W. Chu, X. Hu, and J. Lin (Eds.). ACM, 601--610. https:\/\/doi.org\/10.1145\/1645953.1646031 10.1145\/1645953.1646031 T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. 2009b. Improvements that don't add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, November 2--6, 2009, D. W. Cheung, I. Song, W. W. Chu, X. Hu, and J. Lin (Eds.). ACM, 601--610. https:\/\/doi.org\/10.1145\/1645953.1646031"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_3_2_2_6_1","unstructured":"H. Bar and H. Wang. 2020. Reproducible Science with LaTeX. CoRR Vol. abs\/2010.01482 (2020). showeprint[arXiv]2010.01482 https:\/\/arxiv.org\/abs\/2010.01482  H. Bar and H. Wang. 2020. Reproducible Science with LaTeX. CoRR Vol. abs\/2010.01482 (2020). showeprint[arXiv]2010.01482 https:\/\/arxiv.org\/abs\/2010.01482"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723872.2723882"},{"key":"e_1_3_2_2_8_1","volume-title":"Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020","author":"Breuer T.","year":"2020","unstructured":"T. Breuer , N. Ferro , N. Fuhr , M. Maistro , T. Sakai , P. Schaer , and I. Soboroff . 2020. How to Measure the Reproducibility of System-oriented IR Experiments . In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020 , Virtual Event, China, July 25--30 , 2020 , J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu (Eds.). ACM, 349--358. https:\/\/doi.org\/10.1145\/3397271.3401036 10.1145\/3397271.3401036 T. Breuer, N. Ferro, N. Fuhr, M. Maistro, T. Sakai, P. Schaer, and I. Soboroff. 2020. How to Measure the Reproducibility of System-oriented IR Experiments. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25--30, 2020, J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu (Eds.). ACM, 349--358. https:\/\/doi.org\/10.1145\/3397271.3401036"},{"key":"e_1_3_2_2_9_1","volume-title":"ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II (Lecture Notes in Computer Science","volume":"486","author":"Breuer T.","unstructured":"T. Breuer , N. Ferro , M. Maistro , and P. Schaer . 2021 a. repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments. In Advances in Information Retrieval - 43rd European Conference on IR Research , ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II (Lecture Notes in Computer Science , Vol. 12657), , D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast, and F. Sebastiani (Eds.). Springer, 481-- 486 . https:\/\/doi.org\/10.1007\/978--3-030--72240--1_51 10.1007\/978--3-030--72240--1_51 T. Breuer, N. Ferro, M. Maistro, and P. Schaer. 2021 a. repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 12657), , D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast, and F. Sebastiani (Eds.). Springer, 481--486. https:\/\/doi.org\/10.1007\/978--3-030--72240--1_51"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-85251-1_5"},{"key":"e_1_3_2_2_11_1","unstructured":"T. Breuer and P. Schaer. 2019. Replicability and Reproducibility of Automatic Routing Runs. In Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum Lugano Switzerland September 9--12 2019 (CEUR Workshop Proceedings Vol. 2380) L. Cappellato N. Ferro D. E. Losada and H. M\u00fc ller (Eds.). CEUR-WS.org. http:\/\/ceur-ws.org\/Vol-2380\/paper_84.pdf  T. Breuer and P. Schaer. 2019. Replicability and Reproducibility of Automatic Routing Runs. In Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum Lugano Switzerland September 9--12 2019 (CEUR Workshop Proceedings Vol. 2380) L. Cappellato N. Ferro D. E. Losada and H. M\u00fc ller (Eds.). CEUR-WS.org. http:\/\/ceur-ws.org\/Vol-2380\/paper_84.pdf"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767812"},{"key":"e_1_3_2_2_13_1","volume-title":"5th Workshop on the Theory and Practice of Provenance, TaPP'13","author":"Chirigati F. S.","year":"2013","unstructured":"F. S. Chirigati , D. E. Shasha , and J. Freire . 2013. ReproZip: Using Provenance to Support Computational Reproducibility . In 5th Workshop on the Theory and Practice of Provenance, TaPP'13 , Lombard, IL, USA, April 2--3 , 2013 , , A. Meliou and V. Tannen (Eds.). USENIX Association. https:\/\/www.usenix.org\/conference\/tapp13\/technical-sessions\/presentation\/chirigati F. S. Chirigati, D. E. Shasha, and J. Freire. 2013. ReproZip: Using Provenance to Support Computational Reproducibility. In 5th Workshop on the Theory and Practice of Provenance, TaPP'13, Lombard, IL, USA, April 2--3, 2013 , , A. Meliou and V. Tannen (Eds.). USENIX Association. https:\/\/www.usenix.org\/conference\/tapp13\/technical-sessions\/presentation\/chirigati"},{"key":"e_1_3_2_2_14_1","volume-title":"University of Arizona TR","volume":"14","author":"Collberg C.","year":"2015","unstructured":"C. Collberg , T. Proebsting , and A. M. Warren . 2015. Repeatability and benefaction in computer systems research . University of Arizona TR , Vol. 14 , 4 ( 2015 ). C. Collberg, T. Proebsting, and A. M. Warren. 2015. Repeatability and benefaction in computer systems research. University of Arizona TR , Vol. 14, 4 (2015)."},{"key":"e_1_3_2_2_15_1","volume-title":"S. Soiland-Reyes, and C. A. Goble.","author":"Crusoe M. R.","year":"2021","unstructured":"M. R. Crusoe , S. Abeln , A. Iosup , P. Amstutz , J. Chilton , N. Tijanic , H. M\u00e9 nager , S. Soiland-Reyes, and C. A. Goble. 2021 . Methods Included : Standardizing Computational Reuse and Portability with the Common Workflow Language. CoRR , Vol. abs\/ 2105 .07028 (2021). showeprint[arXiv]2105.07028 https:\/\/arxiv.org\/abs\/2105.07028 M. R. Crusoe, S. Abeln, A. Iosup, P. Amstutz, J. Chilton, N. Tijanic, H. M\u00e9 nager, S. Soiland-Reyes, and C. A. Goble. 2021. Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language. CoRR , Vol. abs\/2105.07028 (2021). showeprint[arXiv]2105.07028 https:\/\/arxiv.org\/abs\/2105.07028"},{"key":"e_1_3_2_2_16_1","volume-title":"9th European Conference, ECDL 2005, Vienna, Austria, September 18--23, 2005, Proceedings (Lecture Notes in Computer Science","volume":"484","author":"Di Nunzio G. M.","unstructured":"G. M. Di Nunzio and N. Ferro . 2005. DIRECT: A System for Evaluating Information Access Components of Digital Libraries. In Research and Advanced Technology for Digital Libraries , 9th European Conference, ECDL 2005, Vienna, Austria, September 18--23, 2005, Proceedings (Lecture Notes in Computer Science , Vol. 3652), , A. Rauber, S. Christodoulakis, and A. M. Tjoa (Eds.). Springer, 483-- 484 . https:\/\/doi.org\/10.1007\/11551362_46 10.1007\/11551362_46 G. M. Di Nunzio and N. Ferro. 2005. DIRECT: A System for Evaluating Information Access Components of Digital Libraries. In Research and Advanced Technology for Digital Libraries, 9th European Conference, ECDL 2005, Vienna, Austria, September 18--23, 2005, Proceedings (Lecture Notes in Computer Science, Vol. 3652), , A. Rauber, S. Christodoulakis, and A. M. Tjoa (Eds.). Springer, 483--484. https:\/\/doi.org\/10.1007\/11551362_46"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824114"},{"key":"e_1_3_2_2_18_1","volume-title":"Analyzing the GitHub Repositories of Research Papers. In JCDL '20: Proceedings of the ACM\/IEEE Joint Conference on Digital Libraries in 2020","author":"M.","year":"2020","unstructured":"M. F\"a rber. 2020 . Analyzing the GitHub Repositories of Research Papers. In JCDL '20: Proceedings of the ACM\/IEEE Joint Conference on Digital Libraries in 2020 , Virtual Event, China, August 1--5 , 2020, R. Huang, D. Wu, G. Marchionini, D. He, S. J. Cunningham, and P. Hansen (Eds.). ACM , 491--492. https:\/\/doi.org\/10.1145\/3383583.3398578 10.1145\/3383583.3398578 M. F\"a rber. 2020. Analyzing the GitHub Repositories of Research Papers. In JCDL '20: Proceedings of the ACM\/IEEE Joint Conference on Digital Libraries in 2020, Virtual Event, China, August 1--5, 2020, R. Huang, D. Wu, G. Marchionini, D. He, S. J. Cunningham, and P. Hansen (Eds.). ACM , 491--492. https:\/\/doi.org\/10.1145\/3383583.3398578"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3116857"},{"key":"e_1_3_2_2_20_1","article-title":"Reproducibility Challenges in Information Retrieval Evaluation","volume":"8","author":"Ferro N.","year":"2017","unstructured":"N. Ferro . 2017 . Reproducibility Challenges in Information Retrieval Evaluation . ACM J. Data Inf. Qual. , Vol. 8 , 2 (2017), 8:1--8:4. https:\/\/doi.org\/10.1145\/3020206 10.1145\/3020206 N. Ferro. 2017. Reproducibility Challenges in Information Retrieval Evaluation. ACM J. Data Inf. Qual. , Vol. 8, 2 (2017), 8:1--8:4. https:\/\/doi.org\/10.1145\/3020206","journal-title":"ACM J. Data Inf. Qual."},{"key":"e_1_3_2_2_21_1","volume-title":"IR: Findings from the Dagstuhl Seminar on \"Reproducibility of Data-Oriented Experiments in e-Science\". SIGIR Forum","author":"Ferro N.","year":"2016","unstructured":"N. Ferro , N. Fuhr , K. J\"a rvelin, N. Kando , M. Lippold , and J. Zobel . 2016 . Increasing Reproducibility in IR: Findings from the Dagstuhl Seminar on \"Reproducibility of Data-Oriented Experiments in e-Science\". SIGIR Forum , Vol. 50 , 1 (2016), 68--82. https:\/\/doi.org\/10.1145\/2964797.2964808 10.1145\/2964797.2964808 N. Ferro, N. Fuhr, K. J\"a rvelin, N. Kando, M. Lippold, and J. Zobel. 2016. Increasing Reproducibility in IR: Findings from the Dagstuhl Seminar on \"Reproducibility of Data-Oriented Experiments in e-Science\". SIGIR Forum , Vol. 50, 1 (2016), 68--82. https:\/\/doi.org\/10.1145\/2964797.2964808"},{"key":"#cr-split#-e_1_3_2_2_22_1.1","doi-asserted-by":"crossref","unstructured":"N. Ferro N. Fuhr M. Maistro T. Sakai and I. Soboroff. 2019. Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality Multimodality and Interaction - 10th International Conference of the CLEF Association CLEF 2019 Lugano Switzerland September 9--12 2019 Proceedings (Lecture Notes in Computer Science Vol. 11696) F. Crestani M. Braschler J. Savoy A. Rauber H. M\u00fc ller D. E. Losada G. Heinatz B\u00fc rki L. Cappellato and N. Ferro (Eds.). Springer 287--300. https:\/\/doi.org\/10.1007\/978--3-030--28577--7_24 10.1007\/978--3-030--28577--7_24","DOI":"10.1007\/978-3-030-28577-7_24"},{"key":"#cr-split#-e_1_3_2_2_22_1.2","doi-asserted-by":"crossref","unstructured":"N. Ferro N. Fuhr M. Maistro T. Sakai and I. Soboroff. 2019. Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality Multimodality and Interaction - 10th International Conference of the CLEF Association CLEF 2019 Lugano Switzerland September 9--12 2019 Proceedings (Lecture Notes in Computer Science Vol. 11696) F. Crestani M. Braschler J. Savoy A. Rauber H. M\u00fc ller D. E. Losada G. Heinatz B\u00fc rki L. Cappellato and N. Ferro (Eds.). Springer 287--300. https:\/\/doi.org\/10.1007\/978--3-030--28577--7_24","DOI":"10.1007\/978-3-030-28577-7_24"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274784.3274786"},{"key":"#cr-split#-e_1_3_2_2_24_1.1","doi-asserted-by":"crossref","unstructured":"N. Ferro M. Maistro T. Sakai and I. Soboroff. 2018. Overview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality Multimodality and Interaction - 9th International Conference of the CLEF Association CLEF 2018 Avignon France September 10--14 2018 Proceedings (Lecture Notes in Computer Science Vol. 11018) P. Bellot C. Trabelsi J. Mothe F. Murtagh J. Nie L. Soulier E. SanJuan L. Cappellato and N. Ferro (Eds.). Springer 239--246. https:\/\/doi.org\/10.1007\/978--3--319--98932--7_23 10.1007\/978--3--319--98932--7_23","DOI":"10.1007\/978-3-319-98932-7_23"},{"key":"#cr-split#-e_1_3_2_2_24_1.2","doi-asserted-by":"crossref","unstructured":"N. Ferro M. Maistro T. Sakai and I. Soboroff. 2018. Overview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm. In Experimental IR Meets Multilinguality Multimodality and Interaction - 9th International Conference of the CLEF Association CLEF 2018 Avignon France September 10--14 2018 Proceedings (Lecture Notes in Computer Science Vol. 11018) P. Bellot C. Trabelsi J. Mothe F. Murtagh J. Nie L. Soulier E. SanJuan L. Cappellato and N. Ferro (Eds.). Springer 239--246. https:\/\/doi.org\/10.1007\/978--3--319--98932--7_23","DOI":"10.1007\/978-3-319-98932-7_23"},{"key":"e_1_3_2_2_25_1","volume-title":"Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016","author":"Ferro N.","year":"2016","unstructured":"N. Ferro and G. Silvello . 2016. A General Linear Mixed Models Approach to Study System Component Effects . In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016 , Pisa, Italy, July 17--21 , 2016 , , R. Perego, F. Sebastiani, J. A. Aslam, I. Ruthven, and J. Zobel (Eds.). ACM, 25--34. https:\/\/doi.org\/10.1145\/2911451.2911530 10.1145\/2911451.2911530 N. Ferro and G. Silvello. 2016. A General Linear Mixed Models Approach to Study System Component Effects. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17--21, 2016 , , R. Perego, F. Sebastiani, J. A. Aslam, I. Ruthven, and J. Zobel (Eds.). ACM, 25--34. https:\/\/doi.org\/10.1145\/2911451.2911530"},{"key":"e_1_3_2_2_26_1","first-page":"108","article-title":"Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041)","volume":"6","author":"Freire J.","year":"2016","unstructured":"J. Freire , N. Fuhr , and A. Rauber . 2016 . Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041) . Dagstuhl Reports , Vol. 6 , 1 (2016), 108 -- 159 . https:\/\/doi.org\/10.4230\/DagRep.6.1.108 10.4230\/DagRep.6.1.108 J. Freire, N. Fuhr, and A. Rauber. 2016. Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041). Dagstuhl Reports , Vol. 6, 1 (2016), 108--159. https:\/\/doi.org\/10.4230\/DagRep.6.1.108","journal-title":"Dagstuhl Reports"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190580.3190586"},{"key":"e_1_3_2_2_28_1","volume-title":"Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF","author":"Fuhr N.","unstructured":"N. Fuhr . 2019. Reproducibility and Validity in CLEF . In Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF , , N. Ferro and C. Peters (Eds.). The Information Retrieval Series, Vol . 41. Springer , 555--564. https:\/\/doi.org\/10.1007\/978--3-030--22948--1_23 10.1007\/978--3-030--22948--1_23 N. Fuhr. 2019. Reproducibility and Validity in CLEF . In Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF , , N. Ferro and C. Peters (Eds.). The Information Retrieval Series, Vol. 41. Springer, 555--564. https:\/\/doi.org\/10.1007\/978--3-030--22948--1_23"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3483382.3483385"},{"key":"e_1_3_2_2_30_1","volume-title":"CHIIR '21: ACM SIGIR Conference on Human Information Interaction and Retrieval","author":"M.","year":"2021","unstructured":"M. G\"a de, M. Koolen , M. M. Hall , T. Bogers , and V. Petras . 2021. A Manifesto on Resource Re-Use in Interactive Information Retrieval . In CHIIR '21: ACM SIGIR Conference on Human Information Interaction and Retrieval , Canberra, ACT, Australia, March 14--19 , 2021 , , F. Scholer, P. Thomas, D. Elsweiler, H. Joho, N. Kando, and C. Smith (Eds.). ACM, 141--149. https:\/\/doi.org\/10.1145\/3406522.3446056 10.1145\/3406522.3446056 M. G\"a de, M. Koolen, M. M. Hall, T. Bogers, and V. Petras. 2021. A Manifesto on Resource Re-Use in Interactive Information Retrieval. In CHIIR '21: ACM SIGIR Conference on Human Information Interaction and Retrieval, Canberra, ACT, Australia, March 14--19, 2021 , , F. Scholer, P. Thomas, D. Elsweiler, H. Joho, N. Kando, and C. Smith (Eds.). ACM, 141--149. https:\/\/doi.org\/10.1145\/3406522.3446056"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"crossref","unstructured":"R. N. J. Graham R. W. Perriss and A. F. Scarsbrook. 2005. DICOM demystified: a review of digital file formats and their use in radiological practice. Clinical radiology Vol. 60 11 (2005) 1133--1140.  R. N. J. Graham R. W. Perriss and A. F. Scarsbrook. 2005. DICOM demystified: a review of digital file formats and their use in radiological practice. Clinical radiology Vol. 60 11 (2005) 1133--1140.","DOI":"10.1016\/j.crad.2005.07.003"},{"key":"e_1_3_2_2_33_1","volume-title":"Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017","author":"Grossman M. R.","year":"2017","unstructured":"M. R. Grossman and G. V. Cormack . 2017. MRG_UWaterloo and WaterlooCormack Participation in the TREC 2017 Common Core Track . In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017 , Gaithersburg, Maryland, USA, November 15--17 , 2017 (NIST Special Publication, Vol. 500--324), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/MRG_UWaterloo-CC.pdf M. R. Grossman and G. V. Cormack. 2017. MRG_UWaterloo and WaterlooCormack Participation in the TREC 2017 Common Core Track. In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017, Gaithersburg, Maryland, USA, November 15--17, 2017 (NIST Special Publication, Vol. 500--324), E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/MRG_UWaterloo-CC.pdf"},{"key":"e_1_3_2_2_34_1","volume-title":"Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018","author":"Grossman M. R.","year":"2018","unstructured":"M. R. Grossman and G. V. Cormack . 2018. MRG_UWaterloo Participation in the TREC 2018 Common Core Track . In Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018 , Gaithersburg, Maryland, USA, November 14--16 , 2018 (NIST Special Publication, Vol. 500--331), , E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec27\/papers\/MRG_UWaterloo-CC.pdf M. R. Grossman and G. V. Cormack. 2018. MRG_UWaterloo Participation in the TREC 2018 Common Core Track. In Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018, Gaithersburg, Maryland, USA, November 14--16, 2018 (NIST Special Publication, Vol. 500--331), , E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec27\/papers\/MRG_UWaterloo-CC.pdf"},{"key":"e_1_3_2_2_35_1","volume-title":"ECIR 2015, Vienna, Austria, March 29 - April 2, 2015. Proceedings. Lecture Notes in Computer Science","volume":"9022","author":"Hanbury A.","year":"2015","unstructured":"A. Hanbury , G. Kazai , A. Rauber , and N. Fuhr ( Eds .). 2015 . Advances in Information Retrieval - 37th European Conference on IR Research , ECIR 2015, Vienna, Austria, March 29 - April 2, 2015. Proceedings. Lecture Notes in Computer Science , Vol. 9022 . https:\/\/doi.org\/10.1007\/978--3--319--16354--3 10.1007\/978--3--319--16354--3 A. Hanbury, G. Kazai, A. Rauber, and N. Fuhr (Eds.). 2015. Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Vienna, Austria, March 29 - April 2, 2015. Proceedings. Lecture Notes in Computer Science, Vol. 9022. https:\/\/doi.org\/10.1007\/978--3--319--16354--3"},{"key":"e_1_3_2_2_36_1","article-title":"Evaluation-as-a-Service for the Computational Sciences","volume":"10","author":"Hopfgartner F.","year":"2018","unstructured":"F. Hopfgartner , A. Hanbury , H. M\u00fc ller, I. Eggel , K. Balog , T. Brodt , G. V. Cormack , J. Lin , J. Kalpathy-Cramer , N. Kando , M. P. Kato , A. Krithara , T. Gollub , M. Potthast , E. Viegas , and S. Mercer . 2018 . Evaluation-as-a-Service for the Computational Sciences : Overview and Outlook. ACM J. Data Inf. Qual. , Vol. 10 , 4 (2018), 15:1--15:32. https:\/\/doi.org\/10.1145\/3239570 10.1145\/3239570 F. Hopfgartner, A. Hanbury, H. M\u00fc ller, I. Eggel, K. Balog, T. Brodt, G. V. Cormack, J. Lin, J. Kalpathy-Cramer, N. Kando, M. P. Kato, A. Krithara, T. Gollub, M. Potthast, E. Viegas, and S. Mercer. 2018. Evaluation-as-a-Service for the Computational Sciences: Overview and Outlook. ACM J. Data Inf. Qual. , Vol. 10, 4 (2018), 15:1--15:32. https:\/\/doi.org\/10.1145\/3239570","journal-title":"Overview and Outlook. ACM J. Data Inf. Qual."},{"key":"e_1_3_2_2_37_1","volume-title":"Reproducibility in Scientific Computing. ACM Comput. Surv.","volume":"51","author":"Ivie P.","year":"2018","unstructured":"P. Ivie and D. Thain . 2018 . Reproducibility in Scientific Computing. ACM Comput. Surv. , Vol. 51 , 3 ( 2018 ), 63:1--63:36. https:\/\/doi.org\/10.1145\/3186266 10.1145\/3186266 P. Ivie and D. Thain. 2018. Reproducibility in Scientific Computing. ACM Comput. Surv. , Vol. 51, 3 (2018), 63:1--63:36. https:\/\/doi.org\/10.1145\/3186266"},{"key":"e_1_3_2_2_38_1","volume-title":"The Popper Convention: Making Reproducible Systems Evaluation Practical. In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPS Workshops 2017","author":"Jimenez I.","year":"2017","unstructured":"I. Jimenez , M. Sevilla , N. Watkins , C. Maltzahn , J. F. Lofstead , K. Mohror , A. C. Arpaci-Dusseau , and R. H. Arpaci-Dusseau . 2017 . The Popper Convention: Making Reproducible Systems Evaluation Practical. In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPS Workshops 2017 , Orlando \/ Buena Vista, FL, USA, May 29 - June 2, 2017 . IEEE Computer Society, 1561--1570. https:\/\/doi.org\/10.1109\/IPDPSW.2017.157 10.1109\/IPDPSW.2017.157 I. Jimenez, M. Sevilla, N. Watkins, C. Maltzahn, J. F. Lofstead, K. Mohror, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. 2017. The Popper Convention: Making Reproducible Systems Evaluation Practical. In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPS Workshops 2017, Orlando \/ Buena Vista, FL, USA, May 29 - June 2, 2017 . IEEE Computer Society, 1561--1570. https:\/\/doi.org\/10.1109\/IPDPSW.2017.157"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661945"},{"key":"e_1_3_2_2_40_1","volume-title":"ECIR 2020, Lisbon, Portugal, April 14--17, 2020, Proceedings, Part II (Lecture Notes in Computer Science","volume":"34","author":"Kamphuis C.","unstructured":"C. Kamphuis , A. P. de Vries , L. Boytsov , and J. Lin . 2020. Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants. In Advances in Information Retrieval - 42nd European Conference on IR Research , ECIR 2020, Lisbon, Portugal, April 14--17, 2020, Proceedings, Part II (Lecture Notes in Computer Science , Vol. 12036), , J. M. Jose, E. Yilmaz, J. Magalh a es, P. Castells, N. Ferro, M. J. Silva, and F. Martins (Eds.). Springer, 28-- 34 . https:\/\/doi.org\/10.1007\/978--3-030--45442--5_4 10.1007\/978--3-030--45442--5_4 C. Kamphuis, A. P. de Vries, L. Boytsov, and J. Lin. 2020. Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants. In Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14--17, 2020, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 12036), , J. M. Jose, E. Yilmaz, J. Magalh a es, P. Castells, N. Ferro, M. J. Silva, and F. Martins (Eds.). Springer, 28--34. https:\/\/doi.org\/10.1007\/978--3-030--45442--5_4"},{"key":"e_1_3_2_2_41_1","volume-title":"The 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '12","author":"Kohavi R.","year":"2012","unstructured":"R. Kohavi , A. Deng , B. Frasca , R. Longbotham , T. Walker , and Y. Xu . 2012. Trustworthy online controlled experiments: five puzzling outcomes explained . In The 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '12 , Beijing, China, August 12--16 , 2012 , , Q. Yang, D. Agarwal, and J. Pei (Eds.). ACM , 786--794. https:\/\/doi.org\/10.1145\/2339530.2339653 10.1145\/2339530.2339653 R. Kohavi, A. Deng, B. Frasca, R. Longbotham, T. Walker, and Y. Xu. 2012. Trustworthy online controlled experiments: five puzzling outcomes explained. In The 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '12, Beijing, China, August 12--16, 2012 , , Q. Yang, D. Agarwal, and J. Pei (Eds.). ACM , 786--794. https:\/\/doi.org\/10.1145\/2339530.2339653"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-016-1004-2"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2021.100322"},{"key":"#cr-split#-e_1_3_2_2_44_1.1","doi-asserted-by":"crossref","unstructured":"J. Lin X. Ma S. Lin J. Yang R. Pradeep and R. Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval Virtual Event Canada July 11--15 2021 F. Diaz C. Shah T. Suel P. Castells R. Jones and T. Sakai (Eds.). ACM 2356--2362. https:\/\/doi.org\/10.1145\/3404835.3463238 10.1145\/3404835.3463238","DOI":"10.1145\/3404835.3463238"},{"key":"#cr-split#-e_1_3_2_2_44_1.2","doi-asserted-by":"crossref","unstructured":"J. Lin X. Ma S. Lin J. Yang R. Pradeep and R. Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval Virtual Event Canada July 11--15 2021 F. Diaz C. Shah T. Suel P. Castells R. Jones and T. Sakai (Eds.). ACM 2356--2362. https:\/\/doi.org\/10.1145\/3404835.3463238","DOI":"10.1145\/3404835.3463238"},{"key":"e_1_3_2_2_45_1","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019","author":"Lin J.","year":"2019","unstructured":"J. Lin and P. Yang . 2019. The Impact of Score Ties on Repeatability in Document Ranking . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019 , Paris, France, July 21--25 , 2019 , , B. Piwowarski, M. Chevalier, E. Gaussier, Y. Maarek, J. Nie, and F. Scholer (Eds.). ACM , 1125--1128. https:\/\/doi.org\/10.1145\/3331184.3331339 10.1145\/3331184.3331339 J. Lin and P. Yang. 2019. The Impact of Score Ties on Repeatability in Document Ranking. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21--25, 2019 , , B. Piwowarski, M. Chevalier, E. Gaussier, Y. Maarek, J. Nie, and F. Scholer (Eds.). ACM , 1125--1128. https:\/\/doi.org\/10.1145\/3331184.3331339"},{"key":"e_1_3_2_2_46_1","volume-title":"ECIR 2020, Lisbon, Portugal, April 14--17, 2020, Proceedings, Part II (Lecture Notes in Computer Science","volume":"49","author":"Lin J.","unstructured":"J. Lin and Q. Zhang . 2020. Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments. In Advances in Information Retrieval - 42nd European Conference on IR Research , ECIR 2020, Lisbon, Portugal, April 14--17, 2020, Proceedings, Part II (Lecture Notes in Computer Science , Vol. 12036), , J. M. Jose, E. Yilmaz, J. Magalh a es, P. Castells, N. Ferro, M. J. Silva, and F. Martins (Eds.). Springer, 43-- 49 . https:\/\/doi.org\/10.1007\/978--3-030--45442--5_6 10.1007\/978--3-030--45442--5_6 J. Lin and Q. Zhang. 2020. Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments. In Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14--17, 2020, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 12036), , J. M. Jose, E. Yilmaz, J. Magalh a es, P. Castells, N. Ferro, M. J. Silva, and F. Martins (Eds.). Springer, 43--49. https:\/\/doi.org\/10.1007\/978--3-030--45442--5_6"},{"key":"#cr-split#-e_1_3_2_2_47_1.1","unstructured":"A. Lipani F. Piroi L. Andersson and A. Hanbury. 2014. An Information Retrieval Ontology for Information Retrieval Nanopublications. In Information Access Evaluation. Multilinguality Multimodality and Interaction - 5th International Conference of the CLEF Initiative CLEF 2014 Sheffield UK September 15--18 2014. Proceedings (Lecture Notes in Computer Science Vol. 8685) E. Kanoulas M. Lupu P. D. Clough M. Sanderson M. M. Hall A. Hanbury and E. G. Toms (Eds.). Springer 44--49. https:\/\/doi.org\/10.1007\/978--3--319--11382--1_5 10.1007\/978--3--319--11382--1_5"},{"key":"#cr-split#-e_1_3_2_2_47_1.2","doi-asserted-by":"crossref","unstructured":"A. Lipani F. Piroi L. Andersson and A. Hanbury. 2014. An Information Retrieval Ontology for Information Retrieval Nanopublications. In Information Access Evaluation. Multilinguality Multimodality and Interaction - 5th International Conference of the CLEF Initiative CLEF 2014 Sheffield UK September 15--18 2014. Proceedings (Lecture Notes in Computer Science Vol. 8685) E. Kanoulas M. Lupu P. D. Clough M. Sanderson M. M. Hall A. Hanbury and E. G. Toms (Eds.). Springer 44--49. https:\/\/doi.org\/10.1007\/978--3--319--11382--1_5","DOI":"10.1007\/978-3-319-11382-1_5"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000016"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463254"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3409256.3409829"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2017.01.002"},{"key":"e_1_3_2_2_52_1","volume-title":"The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '14, Gold Coast, QLD, Australia - July 06 - 11","author":"H.","year":"2014","unstructured":"H. M\u00fc hleisen, T. Samar , J. Lin , and A. P. de Vries . 2014 . Old dogs are great at new tricks: column stores for ir prototyping . In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '14, Gold Coast, QLD, Australia - July 06 - 11 , 2014 , , S. Geva, A. Trotman, P. Bruza, C. L. A. Clarke, and K. J\"a rvelin (Eds.). ACM , 863--866. https:\/\/doi.org\/10.1145\/2600428.2609460 10.1145\/2600428.2609460 H. M\u00fc hleisen, T. Samar, J. Lin, and A. P. de Vries. 2014. Old dogs are great at new tricks: column stores for ir prototyping. In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '14, Gold Coast, QLD, Australia - July 06 - 11, 2014 , , S. Geva, A. Trotman, P. Bruza, C. L. A. Clarke, and K. J\"a rvelin (Eds.). ACM , 863--866. https:\/\/doi.org\/10.1145\/2600428.2609460"},{"key":"e_1_3_2_2_53_1","volume-title":"The Information Retrieval Series","volume":"41","author":"Potthast M.","unstructured":"M. Potthast , T. Gollub , M. Wiegmann , and B. Stein . 2019. TIRA Integrated Research Architecture. In Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF , , N. Ferro and C. Peters (Eds.) . The Information Retrieval Series , Vol. 41 . Springer, 123--160. https:\/\/doi.org\/10.1007\/978--3-030--22948--1_5 10.1007\/978--3-030--22948--1_5 M. Potthast, T. Gollub, M. Wiegmann, and B. Stein. 2019. TIRA Integrated Research Architecture. In Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF , , N. Ferro and C. Peters (Eds.). The Information Retrieval Series, Vol. 41. Springer, 123--160. https:\/\/doi.org\/10.1007\/978--3-030--22948--1_5"},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1186\/1751-0473-8-7"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460210.3493546"},{"key":"e_1_3_2_2_56_1","article-title":"To Clean or Not to Clean: Document Preprocessing and Reproducibility","volume":"10","author":"Roy D.","year":"2018","unstructured":"D. Roy , M. Mitra , and D. Ganguly . 2018 . To Clean or Not to Clean: Document Preprocessing and Reproducibility . ACM J. Data Inf. Qual. , Vol. 10 , 4 (2018), 18:1--18:25. https:\/\/doi.org\/10.1145\/3242180 10.1145\/3242180 D. Roy, M. Mitra, and D. Ganguly. 2018. To Clean or Not to Clean: Document Preprocessing and Reproducibility. ACM J. Data Inf. Qual. , Vol. 10, 4 (2018), 18:1--18:25. https:\/\/doi.org\/10.1145\/3242180","journal-title":"ACM J. Data Inf. Qual."},{"key":"e_1_3_2_2_57_1","volume-title":"Proceedings of the International Conference on Computational Science, ICCS 2011","author":"Van Gorp P.","year":"2011","unstructured":"P. Van Gorp and S. Mazanek . 2011. SHARE: a web portal for creating and sharing executable research papers . In Proceedings of the International Conference on Computational Science, ICCS 2011 , Nanyang Technological University, Singapore, 1- -3 June , 2011 (Procedia Computer Science, Vol. 4), M. Sato, S. Matsuoka, P. M. A. Sloot, G. D. Albada, and J. J. Dongarra (Eds.). Elsevier, 589--597. https:\/\/doi.org\/10.1016\/j.procs.2011.04.062 10.1016\/j.procs.2011.04.062 P. Van Gorp and S. Mazanek. 2011. SHARE: a web portal for creating and sharing executable research papers. In Proceedings of the International Conference on Computational Science, ICCS 2011, Nanyang Technological University, Singapore, 1--3 June, 2011 (Procedia Computer Science, Vol. 4), M. Sato, S. Matsuoka, P. M. A. Sloot, G. D. Albada, and J. J. Dongarra (Eds.). Elsevier, 589--597. https:\/\/doi.org\/10.1016\/j.procs.2011.04.062"},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.2218\/ijdc.v3i1.45"},{"key":"e_1_3_2_2_59_1","volume-title":"Proceedings of the Seventh International Workshop on Evaluating Information Access, EVIA 2016, a Satellite Workshop of the NTCIR-12 Conference, National Center of Sciences","author":"Voorhees E. M.","year":"2016","unstructured":"E. M. Voorhees , S. Rajput , and I. Soboroff . 2016. Promoting Repeatability Through Open Runs . In Proceedings of the Seventh International Workshop on Evaluating Information Access, EVIA 2016, a Satellite Workshop of the NTCIR-12 Conference, National Center of Sciences , Tokyo, Japan, june 7 , 2016 , E. Yilmaz and C. L. A. Clarke (Eds.). National Institute of Informatics (NII) . http:\/\/research.nii.ac.jp\/ntcir\/workshop\/OnlineProceedings12\/pdf\/evia\/04-EVIA2016-VoorheesE.pdf E. M. Voorhees, S. Rajput, and I. Soboroff. 2016. Promoting Repeatability Through Open Runs. In Proceedings of the Seventh International Workshop on Evaluating Information Access, EVIA 2016, a Satellite Workshop of the NTCIR-12 Conference, National Center of Sciences, Tokyo, Japan, june 7, 2016, E. Yilmaz and C. L. A. Clarke (Eds.). National Institute of Informatics (NII) . http:\/\/research.nii.ac.jp\/ntcir\/workshop\/OnlineProceedings12\/pdf\/evia\/04-EVIA2016-VoorheesE.pdf"},{"key":"e_1_3_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_3_2_2_61_1","volume-title":"Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval, ICTIR 2016","author":"Yang P.","year":"2016","unstructured":"P. Yang and H. Fang . 2016. A Reproducibility Study of Information Retrieval Models . In Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval, ICTIR 2016 , Newark, DE, USA, September 12- 6 , 2016 , , B. Carterette, H. Fang, M. Lalmas, and J. Nie (Eds.). ACM, 77--86. https:\/\/doi.org\/10.1145\/2970398.2970415 10.1145\/2970398.2970415 P. Yang and H. Fang. 2016. A Reproducibility Study of Information Retrieval Models. In Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval, ICTIR 2016, Newark, DE, USA, September 12- 6, 2016 , , B. Carterette, H. Fang, M. Lalmas, and J. Nie (Eds.). ACM, 77--86. https:\/\/doi.org\/10.1145\/2970398.2970415"},{"key":"e_1_3_2_2_62_1","article-title":"Anserini: Reproducible Ranking Baselines Using Lucene","volume":"10","author":"Yang P.","year":"2018","unstructured":"P. Yang , H. Fang , and J. Lin . 2018 . Anserini: Reproducible Ranking Baselines Using Lucene . ACM J. Data Inf. Qual. , Vol. 10 , 4 (2018), 16:1--16:20. https:\/\/doi.org\/10.1145\/3239571 10.1145\/3239571 P. Yang, H. Fang, and J. Lin. 2018. Anserini: Reproducible Ranking Baselines Using Lucene. ACM J. Data Inf. Qual. , Vol. 10, 4 (2018), 16:1--16:20. https:\/\/doi.org\/10.1145\/3239571","journal-title":"ACM J. Data Inf. Qual."},{"key":"e_1_3_2_2_63_1","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019","author":"Yang W.","year":"2019","unstructured":"W. Yang , K. Lu , P. Yang , and J. Lin . 2019. Critically Examining the \"Neural Hype\": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019 , Paris, France, July 21--25 , 2019 , , B. Piwowarski, M. Chevalier, E. Gaussier, Y. Maarek, J. Nie, and F. Scholer (Eds.). ACM , 1129--1132. https:\/\/doi.org\/10.1145\/3331184.3331340 10.1145\/3331184.3331340 W. Yang, K. Lu, P. Yang, and J. Lin. 2019. Critically Examining the \"Neural Hype\": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21--25, 2019 , , B. Piwowarski, M. Chevalier, E. Gaussier, Y. Maarek, J. Nie, and F. Scholer (Eds.). ACM , 1129--1132. https:\/\/doi.org\/10.1145\/3331184.3331340"},{"key":"e_1_3_2_2_64_1","volume-title":"Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020","author":"Yilmaz E.","year":"2020","unstructured":"E. Yilmaz , N. Craswell , B. Mitra , and D. Campos . 2020. On the Reliability of Test Collections for Evaluating Systems of Different Types . In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020 , Virtual Event, China, July 25--30 , 2020 , , J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu (Eds.). ACM , 2101--2104. https:\/\/doi.org\/10.1145\/3397271.3401317 10.1145\/3397271.3401317 E. Yilmaz, N. Craswell, B. Mitra, and D. Campos. 2020. On the Reliability of Test Collections for Evaluating Systems of Different Types. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25--30, 2020 , , J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu (Eds.). ACM , 2101--2104. https:\/\/doi.org\/10.1145\/3397271.3401317"},{"key":"e_1_3_2_2_65_1","volume-title":"Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018","author":"Yu R.","year":"2018","unstructured":"R. Yu , Y. Xie , and J. Lin . 2018. H2oloo at TREC 2018: Cross-Collection Relevance Transfer for the Common Core Track . In Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018 , Gaithersburg, Maryland, USA, November 14--16 , 2018 (NIST Special Publication, Vol. 500--331), , E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec27\/papers\/h2oloo-CC.pdf R. Yu, Y. Xie, and J. Lin. 2018. H2oloo at TREC 2018: Cross-Collection Relevance Transfer for the Common Core Track. In Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018, Gaithersburg, Maryland, USA, November 14--16, 2018 (NIST Special Publication, Vol. 500--331), , E. M. Voorhees and A. Ellis (Eds.). National Institute of Standards and Technology (NIST) . https:\/\/trec.nist.gov\/pubs\/trec27\/papers\/h2oloo-CC.pdf"},{"key":"e_1_3_2_2_66_1","volume-title":"ECIR 2019, Cologne, Germany, April 14--18, 2019, Proceedings, Part I (Lecture Notes in Computer Science","author":"Yu R.","unstructured":"R. Yu , Y. Xie , and J. Lin . 2019. Simple Techniques for Cross-Collection Relevance Feedback. In Advances in Information Retrieval - 41st European Conference on IR Research , ECIR 2019, Cologne, Germany, April 14--18, 2019, Proceedings, Part I (Lecture Notes in Computer Science , Vol. 11437), , L . Azzopardi, B. Stein, N. Fuhr, P. Mayr, C. Hauff, and D. Hiemstra (Eds.). Springer, 397--409. https:\/\/doi.org\/10.1007\/978--3-030--15712--8_26 10.1007\/978--3-030--15712--8_26 R. Yu, Y. Xie, and J. Lin. 2019. Simple Techniques for Cross-Collection Relevance Feedback. In Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14--18, 2019, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 11437), , L. Azzopardi, B. Stein, N. Fuhr, P. Mayr, C. Hauff, and D. Hiemstra (Eds.). Springer, 397--409. https:\/\/doi.org\/10.1007\/978--3-030--15712--8_26"}],"event":{"name":"SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval","location":"Madrid Spain","acronym":"SIGIR '22","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477495.3531738","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477495.3531738","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:07Z","timestamp":1750186927000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477495.3531738"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,6]]},"references-count":70,"alternative-id":["10.1145\/3477495.3531738","10.1145\/3477495"],"URL":"https:\/\/doi.org\/10.1145\/3477495.3531738","relation":{},"subject":[],"published":{"date-parts":[[2022,7,6]]},"assertion":[{"value":"2022-07-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}