{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T20:12:27Z","timestamp":1783541547223,"version":"3.55.0"},"reference-count":26,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2016,6,27]],"date-time":"2016-06-27T00:00:00Z","timestamp":1466985600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2016,6,27]]},"abstract":"<jats:p>The Dagstuhl Seminar on \"Reproducibility of Data-Oriented Experiments in e-Science\", held on 24-29 January 2016, focused on the core issues and approaches to reproducibility of experiments from a multidisciplinary point of view, sharing the experience coming from several fields of computer science.<\/jats:p>\n          <jats:p>In this paper, we discuss, summarize, and adapt the main findings of the seminar to the context of IR evaluation -- both system-oriented and user-oriented -- in order to raise awareness in our community and stimulate the fields towards and increased reproducibility of our experiments.<\/jats:p>","DOI":"10.1145\/2964797.2964808","type":"journal-article","created":{"date-parts":[[2016,6,29]],"date-time":"2016-06-29T12:56:52Z","timestamp":1467205012000},"page":"68-82","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":48,"title":["Increasing Reproducibility in IR"],"prefix":"10.1145","volume":"50","author":[{"given":"Nicola","family":"Ferro","sequence":"first","affiliation":[{"name":"University of Padua, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Norbert","family":"Fuhr","sequence":"additional","affiliation":[{"name":"University of Duisburg-Essen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kalervo","family":"J\u00e4rvelin","sequence":"additional","affiliation":[{"name":"University of Tampere, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Noriko","family":"Kando","sequence":"additional","affiliation":[{"name":"National Institute of Informatics, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthias","family":"Lippold","sequence":"additional","affiliation":[{"name":"University of Duisburg-Essen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Justin","family":"Zobel","sequence":"additional","affiliation":[{"name":"University of Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2016,6,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/B978-1-84334-484-1.50006-5","volume-title":"Evaluation of Digital Libraries: An insight into useful applications and methods","author":"Agosti M.","year":"2009","unstructured":"M. Agosti and N. Ferro . Towards an Evaluation Infrastructure for DL Performance Evaluation . In G. Tsakonas and C. Papatheodorou, editors, Evaluation of Digital Libraries: An insight into useful applications and methods , pages 93 -- 120 . Chandos Publishing , Oxford, UK , 2009 . M. Agosti and N. Ferro. Towards an Evaluation Infrastructure for DL Performance Evaluation. In G. Tsakonas and C. Papatheodorou, editors, Evaluation of Digital Libraries: An insight into useful applications and methods, pages 93--120. Chandos Publishing, Oxford, UK, 2009."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2215676.2215681"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2888422.2888439"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572153"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2422256.2422258"},{"key":"e_1_2_1_6_1","first-page":"47","volume-title":"Sp\u00e4rck Jones and Willett {25}","author":"Cleverdon C. W.","unstructured":"C. W. Cleverdon . The Cranfield Tests on Index Languages Devices . In Sp\u00e4rck Jones and Willett {25} , pages 47 -- 60 . C. W. Cleverdon. The Cranfield Tests on Index Languages Devices. In Sp\u00e4rck Jones and Willett {25}, pages 47--60."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1629\/2048-7754.171"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2701583.2701587"},{"key":"e_1_2_1_9_1","volume-title":"Advances in Information Retrieval. Proc. 38th European Conference on IR Research (ECIR 2016)","author":"Ferro N.","year":"2016","unstructured":"N. Ferro , F. Crestani , M.-F. Moens , J. Mothe , F. Silvestri , G. M. Di Nunzio , C. Hauff , and G. Silvello , editors . Advances in Information Retrieval. Proc. 38th European Conference on IR Research (ECIR 2016) , 2016 . Lecture Notes in Computer Science (LNCS) 9626, Springer, Heidelberg, Germany. N. Ferro, F. Crestani, M.-F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff, and G. Silvello, editors. Advances in Information Retrieval. Proc. 38th European Conference on IR Research (ECIR 2016), 2016. Lecture Notes in Computer Science (LNCS) 9626, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_10_1","volume-title":"Reproducibility of data-oriented experiments in e-science. Dagstuhl Reports, 6(1)","author":"Freire J.","year":"2016","unstructured":"J. Freire , N. Fuhr , and A. Rauber . Reproducibility of data-oriented experiments in e-science. Dagstuhl Reports, 6(1) , 2016 . J. Freire, N. Fuhr, and A. Rauber. Reproducibility of data-oriented experiments in e-science. Dagstuhl Reports, 6(1), 2016."},{"key":"e_1_2_1_11_1","volume-title":"Advances in Information Retrieval. Proc. 37th European Conference on IR Research (ECIR 2015)","author":"Fuhr N.","year":"2015","unstructured":"N. Fuhr , A. Rauber , G. Kazai , and A. Hanbury , editors . Advances in Information Retrieval. Proc. 37th European Conference on IR Research (ECIR 2015) , 2015 . Lecture Notes in Computer Science (LNCS) 9022, Springer, Heidelberg, Germany. N. Fuhr, A. Rauber, G. Kazai, and A. Hanbury, editors. Advances in Information Retrieval. Proc. 37th European Conference on IR Research (ECIR 2015), 2015. Lecture Notes in Computer Science (LNCS) 9022, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_12_1","volume-title":"Dialogues Concerning Two New Sciences","author":"Galilei G.","year":"1954","unstructured":"G. Galilei . Dialogues Concerning Two New Sciences . Dover Publications , USA , 1954 . G. Galilei. Dialogues Concerning Two New Sciences. Dover Publications, USA, 1954."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348501"},{"key":"e_1_2_1_14_1","volume-title":"Evaluation-as-a-Service: Overview and Outlook. CoRR, abs\/1512.07454","author":"Hanbury A.","year":"2015","unstructured":"A. Hanbury , H. M\u00fcller , K. Balog , T. Brodt , G. V. Cormack , I. Eggel , T. Gollub , F. Hopfgartner , J. Kalpathy-Cramer , N. Kando , A. Krithara , J. Lin , S. Mercer , and M. Potthast . Evaluation-as-a-Service: Overview and Outlook. CoRR, abs\/1512.07454 , 2015 . A. Hanbury, H. M\u00fcller, K. Balog, T. Brodt, G. V. Cormack, I. Eggel, T. Gollub, F. Hopfgartner, J. Kalpathy-Cramer, N. Kando, A. Krithara, J. Lin, S. Mercer, and M. Potthast. Evaluation-as-a-Service: Overview and Outlook. CoRR, abs\/1512.07454, 2015."},{"key":"e_1_2_1_15_1","volume-title":"TREC. Experiment and Evaluation in Information Retrieval","author":"Harman D. K.","year":"2005","unstructured":"D. K. Harman and E. M. Voorhees , editors . TREC. Experiment and Evaluation in Information Retrieval , 2005 . MIT Press , Cambridge (MA), USA. D. K. Harman and E. M. Voorhees, editors. TREC. Experiment and Evaluation in Information Retrieval, 2005. MIT Press, Cambridge (MA), USA."},{"key":"e_1_2_1_16_1","volume-title":"USA","author":"Hey T.","year":"2009","unstructured":"T. Hey , S. Tansley , and K. Tolle , editors. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research , USA , 2009 . T. Hey, S. Tansley, and K. Tolle, editors. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, USA, 2009."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2795403.2795416"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611430953"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000012"},{"key":"e_1_2_1_20_1","volume-title":"Designing Experiments and Analyzing Data. A Model Comparison Perspective","author":"Maxwell S.","year":"2004","unstructured":"S. Maxwell and H. D. Delaney . Designing Experiments and Analyzing Data. A Model Comparison Perspective . Lawrence Erlbaum Associates , Mahwah (NJ), USA, 2 nd edition, 2004 . S. Maxwell and H. D. Delaney. Designing Experiments and Analyzing Data. A Model Comparison Perspective. Lawrence Erlbaum Associates, Mahwah (NJ), USA, 2nd edition, 2004.","edition":"2"},{"issue":"6251","key":"e_1_2_1_21_1","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1126\/science.aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Collaboration Open Science","year":"2015","unstructured":"Open Science Collaboration . Estimating the reproducibility of psychological science . Science , 349 ( 6251 ): 943 -- 952 , August 2015 . Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251):943--952, August 2015.","journal-title":"Science"},{"issue":"2","key":"e_1_2_1_22_1","first-page":"358","article-title":"How Persuasive Is a Good Fit?","volume":"107","author":"Roberts S.","year":"2000","unstructured":"S. Roberts and H. Pashler . How Persuasive Is a Good Fit? A Comment on Theory Testing. Psychological Review , 107 ( 2 ): 358 -- 367 , April 2000 . S. Roberts and H. Pashler. How Persuasive Is a Good Fit? A Comment on Theory Testing. Psychological Review, 107(2):358--367, April 2000.","journal-title":"A Comment on Theory Testing. Psychological Review"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2641383.2641385"},{"key":"e_1_2_1_24_1","volume-title":"International Journal on Digital Libraries (IJDL)","author":"Silvello G.","year":"2016","unstructured":"G. Silvello , G. Bordea , N. Ferro , P. Buitelaar , and T. Bogers . Semantic Representation and Enrichment of Information Retrieval Experimental Data . International Journal on Digital Libraries (IJDL) , 2016 . G. Silvello, G. Bordea, N. Ferro, P. Buitelaar, and T. Bogers. Semantic Representation and Enrichment of Information Retrieval Experimental Data. International Journal on Digital Libraries (IJDL), 2016."},{"key":"e_1_2_1_25_1","volume-title":"Readings in Information Retrieval","author":"Sp\u00e4rck Jones K.","year":"1997","unstructured":"K. Sp\u00e4rck Jones and P. Willett , editors . Readings in Information Retrieval , 1997 . Morgan Kaufmann Publisher, Inc. , San Francisco, CA, USA . K. Sp\u00e4rck Jones and P. Willett, editors. Readings in Information Retrieval, 1997. Morgan Kaufmann Publisher, Inc., San Francisco, CA, USA."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2422256.2422269"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2964797.2964808","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2964797.2964808","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:40:01Z","timestamp":1750218001000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2964797.2964808"}},"subtitle":["Findings from the Dagstuhl Seminar on \"Reproducibility of Data-Oriented Experiments in e-Science\""],"short-title":[],"issued":{"date-parts":[[2016,6,27]]},"references-count":26,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2016,6,27]]}},"alternative-id":["10.1145\/2964797.2964808"],"URL":"https:\/\/doi.org\/10.1145\/2964797.2964808","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2016,6,27]]},"assertion":[{"value":"2016-06-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}