{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T07:29:53Z","timestamp":1772695793894,"version":"3.50.1"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,6,1]],"date-time":"2023-06-01T00:00:00Z","timestamp":1685577600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:p>This report documents the program and the outcomes of Dagstuhl Seminar 23031 \"Frontiers of Information Access Experimentation for Research and Education\", which brought together 38 participants from 12 countries. The seminar addressed technology-enhanced information access (information retrieval, recommender systems, natural language processing) and specifically focused on developing more responsible experimental practices leading to more valid results, both for research as well as for scientific education.<\/jats:p>\n          <jats:p>\n            The seminar featured a series of long and short talks delivered by participants, who helped in setting a common ground and in letting emerge topics of interest to be explored as the main output of the seminar. This led to the definition of five groups which investigated challenges, opportunities, and next steps in the following areas:\n            <jats:italic>reality check, i.e. conducting real-world studies, human-machine-collaborative relevance judgment frameworks, overcoming methodological challenges in information retrieval and recommender systems through awareness and education, results-blind reviewing, and guidance for authors.<\/jats:italic>\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Date:<\/jats:bold>\n            15--20 January 2023.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Website:<\/jats:bold>\n            https:\/\/www.dagstuhl.de\/23031.\n          <\/jats:p>","DOI":"10.1145\/3636341.3636351","type":"journal-article","created":{"date-parts":[[2023,12,4]],"date-time":"2023-12-04T17:07:47Z","timestamp":1701709667000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Report on the Dagstuhl Seminar on Frontiers of Information Access Experimentation for Research and Education"],"prefix":"10.1145","volume":"57","author":[{"given":"Christine","family":"Bauer","sequence":"first","affiliation":[{"name":"Paris Lodron University Salzburg, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ben","family":"Carterette","sequence":"additional","affiliation":[{"name":"University of Delaware and Spotify, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Norbert","family":"Fuhr","sequence":"additional","affiliation":[{"name":"University of Duisburg-Essen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joeran","family":"Beel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timo","family":"Breuer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Charles L. A.","family":"Clarke","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anita","family":"Crescenzi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giorgio Maria","family":"Di Nunzio","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laura","family":"Dietz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guglielmo","family":"Faggioli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bruce","family":"Ferwerda","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maik","family":"Fr\u00f6be","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthias","family":"Hagen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Allan","family":"Hanbury","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Claudia","family":"Hauff","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dietmar","family":"Jannach","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Noriko","family":"Kando","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Evangelos","family":"Kanoulas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bart P.","family":"Knijnenburg","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Udo","family":"Kruschwitz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Meijie","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria","family":"Maistro","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lien","family":"Michiels","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrea","family":"Papenmeier","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Martin","family":"Potthast","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paolo","family":"Rosso","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alan","family":"Said","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philipp","family":"Schaer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christin","family":"Seifert","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Damiano","family":"Spina","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benno","family":"Stein","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nava","family":"Tintarev","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juli\u00e1n","family":"Urbano","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Henning","family":"Wachsmuth","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Martijn C.","family":"Willemsen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Justin","family":"Zobel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,12,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2642"},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1145\/1645953.1646031","volume-title":"Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM '09","author":"Armstrong Timothy G.","year":"2009","unstructured":"Timothy G. Armstrong , Alistair Moffat , William Webber , and Justin Zobel . Improvements that don't add up: Ad-hoc retrieval results since 1998 . In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM '09 , pages 601 -- 610 , 2009 . 10.1145\/1645953.1646031 Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. Improvements that don't add up: Ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM '09, pages 601--610, 2009. 10.1145\/1645953.1646031"},{"key":"e_1_2_1_3_1","volume-title":"1st Workshop on the Impact of Recommender Systems, co-located with 13th ACM Conference on Recommender Systems (ACM RecSys 2019","author":"Bauer Christine","year":"2019","unstructured":"Christine Bauer and Eva Zangerle . Leveraging multi-method evaluation for multi-stakeholder settings . In 1st Workshop on the Impact of Recommender Systems, co-located with 13th ACM Conference on Recommender Systems (ACM RecSys 2019 ), ImpactRS'19. CEUR-WS.org , 2019 . URL http:\/\/ceur-ws.org\/Vol-2462\/short3.pdf. Christine Bauer and Eva Zangerle. Leveraging multi-method evaluation for multi-stakeholder settings. In 1st Workshop on the Impact of Recommender Systems, co-located with 13th ACM Conference on Recommender Systems (ACM RecSys 2019), ImpactRS'19. CEUR-WS.org, 2019. URL http:\/\/ceur-ws.org\/Vol-2462\/short3.pdf."},{"key":"e_1_2_1_4_1","volume-title":"Dagstuhl Reports","volume":"13","author":"Bauer Christine","year":"2023","unstructured":"Christine Bauer , Ben A. Carterette , Nicola Ferro , and Norbert Fuhr , editors. Report from Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education , Dagstuhl Reports , Volume 13 , Number 1 , 2023 a. Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik, Germany. Christine Bauer, Ben A. Carterette, Nicola Ferro, and Norbert Fuhr, editors. Report from Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education, Dagstuhl Reports, Volume 13, Number 1, 2023a. Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik, Germany."},{"key":"e_1_2_1_5_1","first-page":"51","volume-title":"Bauer et al. [2023a]","author":"Bauer Christine","unstructured":"Christine Bauer , Maik Fr\u00f6be , Dietmar Jannach , Udo Kruschwitz , Paolo Rosso , Damiano Spina , and Nava Tintarev . Overcoming methodological challenges in information retrieval and recommender systems through awareness and education . In Bauer et al. [2023a] , pages 51 -- 67 . Christine Bauer, Maik Fr\u00f6be, Dietmar Jannach, Udo Kruschwitz, Paolo Rosso, Damiano Spina, and Nava Tintarev. Overcoming methodological challenges in information retrieval and recommender systems through awareness and education. In Bauer et al. [2023a], pages 51--67."},{"key":"e_1_2_1_6_1","first-page":"67","volume-title":"Bauer et al. [2023a]","author":"Beel Joeran","unstructured":"Joeran Beel , Timo Breuer , Anita Crescenzi , Norbert Fuhr , and Meijie Li . Results-blind reviewing . In Bauer et al. [2023a] , pages 67 -- 73 . Joeran Beel, Timo Breuer, Anita Crescenzi, Norbert Fuhr, and Meijie Li. Results-blind reviewing. In Bauer et al. [2023a], pages 67--73."},{"key":"e_1_2_1_7_1","first-page":"153","volume-title":"Proceedings of the 22nd International Conference on Theory and Practice of Digital Libraries, TPDL '15","author":"Beel J\u00f6ran","year":"2015","unstructured":"J\u00f6ran Beel and Stefan Langer . A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems . In Proceedings of the 22nd International Conference on Theory and Practice of Digital Libraries, TPDL '15 , pages 153 -- 168 , 2015 . J\u00f6ran Beel and Stefan Langer. A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In Proceedings of the 22nd International Conference on Theory and Practice of Digital Libraries, TPDL '15, pages 153--168, 2015."},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1145\/1277741.1277755","volume-title":"Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '07","author":"B\u00fcttcher Stefan","year":"2007","unstructured":"Stefan B\u00fcttcher , Charles L. A. Clarke , Peter C. K. Yeung , and Ian Soboroff . Reliable information retrieval evaluation with incomplete and biased judgements . In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '07 , pages 63 -- 70 . ACM, 2007 . 10.1145\/1277741.1277755 Stefan B\u00fcttcher, Charles L. A. Clarke, Peter C. K. Yeung, and Ian Soboroff. Reliable information retrieval evaluation with incomplete and biased judgements. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '07, pages 63--70. ACM, 2007. 10.1145\/1277741.1277755"},{"key":"e_1_2_1_9_1","volume-title":"Experimental and quasi-experimental designs for research","author":"Campbell Donald T.","year":"1963","unstructured":"Donald T. Campbell and Julian C. Stanley . Experimental and quasi-experimental designs for research . Houghton Mifflin Company , Boston , 1963 . Donald T. Campbell and Julian C. Stanley. Experimental and quasi-experimental designs for research. Houghton Mifflin Company, Boston, 1963."},{"key":"e_1_2_1_10_1","first-page":"15","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17","author":"Chen Ye","year":"2017","unstructured":"Ye Chen , Ke Zhou , Yiqun Liu , Min Zhang , and Shaoping Ma . Meta-evaluation of online and offline web search evaluation metrics . In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17 , pages 15 -- 24 . ACM, 2017 . 10.1145\/3077136.3080804 Ye Chen, Ke Zhou, Yiqun Liu, Min Zhang, and Shaoping Ma. Meta-evaluation of online and offline web search evaluation metrics. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 15--24. ACM, 2017. 10.1145\/3077136.3080804"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324922000043"},{"key":"e_1_2_1_12_1","first-page":"41","volume-title":"Bauer et al. [2023a]","author":"Clarke Charles L. A.","unstructured":"Charles L. A. Clarke , Gianluca Demartini , Laura Dietz , Guglielmo Faggioli , Matthias Hagen , Claudia Hauff , Noriko Kando , Evangelos Kanoulas , Martin Potthast , Ian Soboroff , Benno Stein , and Henning Wachsmuth . Hmc: A spectrum of human-machine-collaborative relevance judgment frameworks . In Bauer et al. [2023a] , pages 41 -- 50 . Charles L. A. Clarke, Gianluca Demartini, Laura Dietz, Guglielmo Faggioli, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Ian Soboroff, Benno Stein, and Henning Wachsmuth. Hmc: A spectrum of human-machine-collaborative relevance judgment frameworks. In Bauer et al. [2023a], pages 41--50."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3451161"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_2_1_15_1","first-page":"74","volume-title":"Bauer et al. [2023a]","author":"Di Nunzio Giorgio Maria","unstructured":"Giorgio Maria Di Nunzio , Maria Maistro , Christin Seifert , Juli\u00e1n Urbano , and Justin Zobel . Guidance for authors . In Bauer et al. [2023a] , pages 74 -- 79 . Giorgio Maria Di Nunzio, Maria Maistro, Christin Seifert, Juli\u00e1n Urbano, and Justin Zobel. Guidance for authors. In Bauer et al. [2023a], pages 74--79."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468507.3468515"},{"key":"e_1_2_1_17_1","first-page":"295","volume-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21","author":"Draws Tim","year":"2021","unstructured":"Tim Draws , Nava Tintarev , Ujwal Gadiraju , Alessandro Bozzon , and Benjamin Timmermans . This is not what we ordered: Exploring why biased search result rankings affect user attitudes on debated topics . In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21 , pages 295 -- 305 . ACM, 2021 b. 10.1145\/3404835.3462851 Tim Draws, Nava Tintarev, Ujwal Gadiraju, Alessandro Bozzon, and Benjamin Timmermans. This is not what we ordered: Exploring why biased search result rankings affect user attitudes on debated topics. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pages 295--305. ACM, 2021b. 10.1145\/3404835.3462851"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-28244-7_18"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1145\/2043932.2043958","volume-title":"Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys '11","author":"Ekstrand Michael D.","year":"2011","unstructured":"Michael D. Ekstrand , Michael Ludwig , Joseph A. Konstan , and John T. Riedl . Rethinking the recommender research ecosystem: Reproducibility, openness, and LensKit . In Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys '11 , pages 133 -- 140 , 2011 . Michael D. Ekstrand, Michael Ludwig, Joseph A. Konstan, and John T. Riedl. Rethinking the recommender research ecosystem: Reproducibility, openness, and LensKit. In Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys '11, pages 133--140, 2011."},{"key":"e_1_2_1_20_1","volume-title":"37th AAAI Conference on Artificial Intelligence, AAAI 2023","author":"Elstner Theresa","year":"2023","unstructured":"Theresa Elstner , Frank Loebe , Yamen Ajjour , Christopher Akiki , Alexander Bondarenko , Maik Frobe , Lukas Gienapp , Nikolay Kolyada , Janis Mohr , Stephan Sandfuchs , Matti Wiegmann , J\u00f6rg Frochte , Nicola Ferro , Sven Hofmann , Benno Stein , Matthias Hagen , and Martin Potthast . Shared tasks as tutorials: A methodical approach . In 37th AAAI Conference on Artificial Intelligence, AAAI 2023 . AAAI, 2023 . Theresa Elstner, Frank Loebe, Yamen Ajjour, Christopher Akiki, Alexander Bondarenko, Maik Frobe, Lukas Gienapp, Nikolay Kolyada, Janis Mohr, Stephan Sandfuchs, Matti Wiegmann, J\u00f6rg Frochte, Nicola Ferro, Sven Hofmann, Benno Stein, Matthias Hagen, and Martin Potthast. Shared tasks as tutorials: A methodical approach. In 37th AAAI Conference on Artificial Intelligence, AAAI 2023. AAAI, 2023."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1419828112"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction, 1(CSCW):1--22","author":"Epstein Robert","year":"2017","unstructured":"Robert Epstein , Ronald E. Robertson , David Lazer , and Christo Wilson . Suppressing the search engine manipulation effect (SEME) . Proceedings of the ACM on Human-Computer Interaction, 1(CSCW):1--22 , 2017 . 10.1145\/3134677 Robert Epstein, Ronald E. Robertson, David Lazer, and Christo Wilson. Suppressing the search engine manipulation effect (SEME). Proceedings of the ACM on Human-Computer Interaction, 1(CSCW):1--22, 2017. 10.1145\/3134677"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 13th International Conference on the Theory of Information Retrieval, ICTIR '23","author":"Faggioli Guglielmo","year":"2023","unstructured":"Guglielmo Faggioli , Laura Dietz , Charles Clarke , Gianluca Demartini , Matthias Hagen , Claudia Hauff , Noriko Kando , Evangelos Kanoulas , Martin Potthast , Benno Stein , Perspectives on large language models for relevance judgment . In Proceedings of the 13th International Conference on the Theory of Information Retrieval, ICTIR '23 , 2023 . Guglielmo Faggioli, Laura Dietz, Charles Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. Perspectives on large language models for relevance judgment. In Proceedings of the 13th International Conference on the Theory of Information Retrieval, ICTIR '23, 2023."},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1145\/3298689.3347058","volume-title":"Proceedings of the 13th ACM Conference on Recommender Systems, RecSys '19","author":"Dacrema Maurizio Ferrari","year":"2019","unstructured":"Maurizio Ferrari Dacrema , Paolo Cremonesi , and Dietmar Jannach . Are we really making much progress? a worrying analysis of recent neural recommendation approaches . In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys '19 , pages 101 -- 109 . ACM, 2019 . 10.1145\/3298689.3347058 Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress? a worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys '19, pages 101--109. ACM, 2019. 10.1145\/3298689.3347058"},{"key":"e_1_2_1_25_1","first-page":"3","volume-title":"CLEF 19","author":"Ferro Nicola","year":"2019","unstructured":"Nicola Ferro . What Happened in CLEF... For a While? In Experimental IR Meets Multilinguality, Multimodality, and Interaction , CLEF 19 , pages 3 -- 45 . Springer , 2019 . Nicola Ferro. What Happened in CLEF... For a While? In Experimental IR Meets Multilinguality, Multimodality, and Interaction, CLEF 19, pages 3--45. Springer, 2019."},{"key":"e_1_2_1_26_1","first-page":"13","volume-title":"Bauer et al. [2023a]","author":"Ferro Nicola","unstructured":"Nicola Ferro . Coordinate research, evaluation, and education in information access: Towards a more sustainable environment for the community . In Bauer et al. [2023a] , pages 13 -- 16 . Nicola Ferro. Coordinate research, evaluation, and education in information access: Towards a more sustainable environment for the community. In Bauer et al. [2023a], pages 13--16."},{"key":"e_1_2_1_27_1","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1145\/3488560.3498406","volume-title":"Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22","author":"Ferro Nicola","year":"2022","unstructured":"Nicola Ferro and Mark Sanderson . How do you test a test?: A multifaceted examination of significance tests . In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22 , pages 280 -- 288 . ACM, 2022 . 10.1145\/3488560.3498406 Nicola Ferro and Mark Sanderson. How do you test a test?: A multifaceted examination of significance tests. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22, pages 280--288. ACM, 2022. 10.1145\/3488560.3498406"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.4230\/DagMan.7.1.96"},{"key":"e_1_2_1_29_1","first-page":"20","volume-title":"Bauer et al. [2023a]","author":"Ferwerda Bruce","unstructured":"Bruce Ferwerda , Akkan Hanbury , Bart P. Knijnenburg , Birger Larsen , Lien Michiels , Andrea Papenmeier , Alan Said , Philipp Schaer , and Martijn C. Willemsen . Reality check---conducting real world studies . In Bauer et al. [2023a] , pages 20 -- 40 . Bruce Ferwerda, Akkan Hanbury, Bart P. Knijnenburg, Birger Larsen, Lien Michiels, Andrea Papenmeier, Alan Said, Philipp Schaer, and Martijn C. Willemsen. Reality check---conducting real world studies. In Bauer et al. [2023a], pages 20--40."},{"key":"e_1_2_1_30_1","volume-title":"Dagstuhl Reports","volume":"6","author":"Freire Juliana","year":"2016","unstructured":"Juliana Freire , Norbert Fuhr , and Andreas Rauber , editors. Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science , Dagstuhl Reports , Volume 6 , Number 1 , 2016 . Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik, Germany. 10.4230\/DagRep.6.1.108 Juliana Freire, Norbert Fuhr, and Andreas Rauber, editors. Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science, Dagstuhl Reports, Volume 6, Number 1, 2016. Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik, Germany. 10.4230\/DagRep.6.1.108"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190580.3190586"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2843948"},{"key":"e_1_2_1_33_1","first-page":"1644","volume-title":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI '18","author":"Gundersen Odd Erik","year":"2018","unstructured":"Odd Erik Gundersen and Sigbj\u00f8rn Kjensmo . State of the art: Reproducibility in artificial intelligence . In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI '18 , pages 1644 -- 1651 , 2018 . 10.1609\/aaai.v32i1.11503 Odd Erik Gundersen and Sigbj\u00f8rn Kjensmo. State of the art: Reproducibility in artificial intelligence. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI '18, pages 1644--1651, 2018. 10.1609\/aaai.v32i1.11503"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-02276-0"},{"key":"e_1_2_1_35_1","volume-title":"TREC. Experiment and Evaluation in Information Retrieval","author":"Harman Donna K.","year":"2005","unstructured":"Donna K. Harman and Ellen M. Voorhees , editors . TREC. Experiment and Evaluation in Information Retrieval , 2005 . MIT Press . Donna K. Harman and Ellen M. Voorhees, editors. TREC. Experiment and Evaluation in Information Retrieval, 2005. MIT Press."},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1145\/1718487.1718515","volume-title":"Proceedings of the Third ACM International Conference on Web Search and Data Mining, WSDM '10","author":"Hassan Ahmed","year":"2010","unstructured":"Ahmed Hassan , Rosie Jones , and Kristina Lisa Klinkner . Beyond DCG : User behavior as a predictor of a successful search . In Proceedings of the Third ACM International Conference on Web Search and Data Mining, WSDM '10 , pages 221 -- 230 . ACM, 2010 . 10.1145\/1718487.1718515 Ahmed Hassan, Rosie Jones, and Kristina Lisa Klinkner. Beyond DCG: User behavior as a predictor of a successful search. In Proceedings of the Third ACM International Conference on Web Search and Data Mining, WSDM '10, pages 221--230. ACM, 2010. 10.1145\/1718487.1718515"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01397280"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the ACM RecSys 2017 Workshop on Value-Aware and Multi-Stakeholder Recommendation","author":"Jannach Dietmar","year":"2017","unstructured":"Dietmar Jannach and Gediminas Adomavicius . Price and profit awareness in recommender systems . In Proceedings of the ACM RecSys 2017 Workshop on Value-Aware and Multi-Stakeholder Recommendation , 2017 . Dietmar Jannach and Gediminas Adomavicius. Price and profit awareness in recommender systems. In Proceedings of the ACM RecSys 2017 Workshop on Value-Aware and Multi-Stakeholder Recommendation, 2017."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v41i4.5312"},{"key":"e_1_2_1_40_1","volume-title":"fast and slow","author":"Kahneman Daniel","year":"2011","unstructured":"Daniel Kahneman . Thinking , fast and slow . Penguin , 2011 . Daniel Kahneman. Thinking, fast and slow. Penguin, 2011."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000012"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9118-4"},{"key":"e_1_2_1_43_1","first-page":"23","volume-title":"Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation","author":"Joseph","year":"2013","unstructured":"Joseph A. Konstan and Gediminas Adomavicius. Toward identification and adoption of best practices in algorithmic recommender systems research . In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation , pages 23 -- 28 , 2013 . Joseph A. Konstan and Gediminas Adomavicius. Toward identification and adoption of best practices in algorithmic recommender systems research. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, pages 23--28, 2013."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550617697177"},{"key":"e_1_2_1_45_1","first-page":"2283","volume-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21","author":"Lin Jimmy","year":"2021","unstructured":"Jimmy Lin , Daniel Campos , Nick Craswell , Bhaskar Mitra , and Emine Yilmaz . Significant improvements over the state of the art? a case study of the MS MARCO document ranking leaderboard . In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21 , pages 2283 -- 2287 . ACM, 2021 . 10.1145\/3404835.3463034 Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Significant improvements over the state of the art? a case study of the MS MARCO document ranking leaderboard. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pages 2283--2287. ACM, 2021. 10.1145\/3404835.3463034"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1108\/jd-08-2021-0170"},{"key":"e_1_2_1_47_1","first-page":"463","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16","author":"Mao Jiaxin","year":"2016","unstructured":"Jiaxin Mao , Yiqun Liu , Ke Zhou , Jian-Yun Nie , Jingtao Song , Min Zhang , Shaoping Ma , Jiashen Sun , and Hengliang Luo . When does relevance mean usefulness and user satisfaction in Web search? In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16 , pages 463 -- 472 . ACM, 2016 . 10.1145\/2911451.2911507 Jiaxin Mao, Yiqun Liu, Ke Zhou, Jian-Yun Nie, Jingtao Song, Min Zhang, Shaoping Ma, Jiashen Sun, and Hengliang Luo. When does relevance mean usefulness and user satisfaction in Web search? In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, pages 463--472. ACM, 2016. 10.1145\/2911451.2911507"},{"key":"e_1_2_1_48_1","first-page":"209","volume-title":"Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '17","author":"Pogacar Frances A.","year":"2017","unstructured":"Frances A. Pogacar , Amira Ghenai , Mark D. Smucker , and Charles L.A. Clarke . The positive and negative influence of search results on people's decisions about the efficacy of medical treatments . In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '17 , pages 209 -- 216 . ACM, 2017 . 10.1145\/3121050.3121074 Frances A. Pogacar, Amira Ghenai, Mark D. Smucker, and Charles L.A. Clarke. The positive and negative influence of search results on people's decisions about the efficacy of medical treatments. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '17, pages 209--216. ACM, 2017. 10.1145\/3121050.3121074"},{"key":"e_1_2_1_49_1","first-page":"1117","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019","author":"Potthast Martin","year":"2019","unstructured":"Martin Potthast , Lukas Gienapp , Florian Euchner , Nick Heilenk\u00f6tter , Nico Weidmann , Henning Wachsmuth , Benno Stein , and Matthias Hagen . Argument search : Assessing argument relevance . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019 , pages 1117 -- 1120 . ACM, 2019 . 10.1145\/3331184.3331327 Martin Potthast, Lukas Gienapp, Florian Euchner, Nick Heilenk\u00f6tter, Nico Weidmann, Henning Wachsmuth, Benno Stein, and Matthias Hagen. Argument search: Assessing argument relevance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, pages 1117--1120. ACM, 2019. 10.1145\/3331184.3331327"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-1199-4"},{"key":"e_1_2_1_51_1","first-page":"136","volume-title":"Proceedings of the Second International Conference on Design of Experimental Search & Information Retrieval Systems","author":"David","year":"2021","unstructured":"David P. Sander and Laura Dietz. EXAM: how to evaluate retrieve-and-generate systems for users who do not (yet) know what they want . In Proceedings of the Second International Conference on Design of Experimental Search & Information Retrieval Systems , pages 136 -- 146 . CEUR-WS.org, 2021 . URL https:\/\/ceur-ws.org\/Vol-2950\/paper-16.pdf. David P. Sander and Laura Dietz. EXAM: how to evaluate retrieve-and-generate systems for users who do not (yet) know what they want. In Proceedings of the Second International Conference on Design of Experimental Search & Information Retrieval Systems, pages 136--146. CEUR-WS.org, 2021. URL https:\/\/ceur-ws.org\/Vol-2950\/paper-16.pdf."},{"key":"e_1_2_1_52_1","first-page":"555","volume-title":"Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10","author":"Sanderson Mark","year":"2010","unstructured":"Mark Sanderson , Monica Lestari Paramita , Paul Clough , and Evangelos Kanoulas . Do user preferences and evaluation measures line up? In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10 , pages 555 -- 562 . ACM, 2010 . 10.1145\/1835449.1835542 Mark Sanderson, Monica Lestari Paramita, Paul Clough, and Evangelos Kanoulas. Do user preferences and evaluation measures line up? In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 555--562. ACM, 2010. 10.1145\/1835449.1835542"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1301687110"},{"key":"e_1_2_1_54_1","volume-title":"Experimental and quasi-experimental designs for generalized causal inference. Houghton","author":"Shadish William R","year":"2002","unstructured":"William R Shadish , Thomas D Cook , and Donald T. Campbell . Experimental and quasi-experimental designs for generalized causal inference. Houghton , Mifflin and Company , New York , 2002 . William R Shadish, Thomas D Cook, and Donald T. Campbell. Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company, New York, 2002."},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1145\/3383313.3412489","volume-title":"Proceedings of the 14th ACM Conference on Recommender Systems, RecSys '20","author":"Sun Zhu","year":"2020","unstructured":"Zhu Sun , Di Yu , Hui Fang , Jie Yang , Xinghua Qu , Jie Zhang , and Cong Geng . Are we evaluating rigorously? Benchmarking recommendation for reproducible evaluation and fair comparison . In Proceedings of the 14th ACM Conference on Recommender Systems, RecSys '20 , pages 23 -- 32 , 2020 . 10.1145\/3383313.3412489 Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. Are we evaluating rigorously? Benchmarking recommendation for reproducible evaluation and fair comparison. In Proceedings of the 14th ACM Conference on Recommender Systems, RecSys '20, pages 23--32, 2020. 10.1145\/3383313.3412489"},{"key":"e_1_2_1_56_1","volume-title":"Experimental standards for deep learning in natural language processing research","author":"Ulmer Dennis","year":"2022","unstructured":"Dennis Ulmer , Elisa Bassignana , Max M\u00fcller-Eberstein , Daniel Varab , Mike Zhang , Rob van der Goot , Christian Hardmeier , and Barbara Plank . Experimental standards for deep learning in natural language processing research , 2022 . URL https:\/\/aclanthology.org\/2022.findings-emnlp.196. Dennis Ulmer, Elisa Bassignana, Max M\u00fcller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, and Barbara Plank. Experimental standards for deep learning in natural language processing research, 2022. URL https:\/\/aclanthology.org\/2022.findings-emnlp.196."},{"issue":"3","key":"e_1_2_1_57_1","first-page":"813","article-title":"Don van den Bergh, Udo B\u00f6hm, Fabian Dablander, Koen Derks, Tim Draws, Alexander Etz, Nathan J Evans, Quentin F Gronau, Julia M Haaf, et al. The JASP guidelines for conducting and reporting a Bayesian analysis","volume":"28","author":"van Doorn Johnny","year":"2021","unstructured":"Johnny van Doorn , Don van den Bergh, Udo B\u00f6hm, Fabian Dablander, Koen Derks, Tim Draws, Alexander Etz, Nathan J Evans, Quentin F Gronau, Julia M Haaf, et al. The JASP guidelines for conducting and reporting a Bayesian analysis . Psychonomic Bulletin & Review , 28 ( 3 ): 813 -- 826 , 2021 . Johnny van Doorn, Don van den Bergh, Udo B\u00f6hm, Fabian Dablander, Koen Derks, Tim Draws, Alexander Etz, Nathan J Evans, Quentin F Gronau, Julia M Haaf, et al. The JASP guidelines for conducting and reporting a Bayesian analysis. Psychonomic Bulletin & Review, 28(3):813--826, 2021.","journal-title":"Psychonomic Bulletin & Review"},{"key":"e_1_2_1_58_1","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139525305","volume-title":"Interactions with search systems","author":"White Ryen W","year":"2016","unstructured":"Ryen W White . Interactions with search systems . Cambridge University Press , New York, NY, USA , 2016 . Ryen W White. Interactions with search systems. Cambridge University Press, New York, NY, USA, 2016."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1080\/1359432x.2018.1496081"},{"key":"e_1_2_1_60_1","first-page":"1129","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19","author":"Yang Wei","year":"2019","unstructured":"Wei Yang , Kuang Lu , Peilin Yang , and Jimmy Lin . Critically examining the \"neural hype\": Weak baselines and the additivity of effectiveness gains from neural ranking models . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19 , pages 1129 -- 1132 , 2019 . 10.1145\/3331184.3331340 Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. Critically examining the \"neural hype\": Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, pages 1129--1132, 2019. 10.1145\/3331184.3331340"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3556536"},{"key":"e_1_2_1_62_1","first-page":"379","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20","author":"Zhang Fan","year":"2020","unstructured":"Fan Zhang , Jiaxin Mao , Yiqun Liu , Xiaohui Xie , Weizhi Ma , Min Zhang , and Shaoping Ma . Models versus satisfaction: Towards a better understanding of evaluation metrics . In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20 , pages 379 -- 388 . ACM, 2020 . 10.1145\/3397271.3401162 Fan Zhang, Jiaxin Mao, Yiqun Liu, Xiaohui Xie, Weizhi Ma, Min Zhang, and Shaoping Ma. Models versus satisfaction: Towards a better understanding of evaluation metrics. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, pages 379--388. ACM, 2020. 10.1145\/3397271.3401162"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3582524.3582540"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3636341.3636351","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3636341.3636351","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:35:41Z","timestamp":1750178141000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3636341.3636351"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6]]},"references-count":63,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,6]]}},"alternative-id":["10.1145\/3636341.3636351"],"URL":"https:\/\/doi.org\/10.1145\/3636341.3636351","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2023,6]]},"assertion":[{"value":"2023-12-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}