{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T17:52:32Z","timestamp":1790358752656,"version":"4.1.0"},"reference-count":17,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2014,6,26]],"date-time":"2014-06-26T00:00:00Z","timestamp":1403740800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2014,6,26]]},"abstract":"<jats:p>Experiment replication and reproduction are key requirements for empirical research methodology, and an important open issue in the field of Recommender Systems. When an experiment is repeated by a different researcher and exactly the same result is obtained, we can say the experiment has been replicated. When the results are not exactly the same but the conclusions are compatible with the prior ones, we have a reproduction of the experiment. Reproducibility and replication involve recommendation algorithm implementations, experimental protocols, and evaluation metrics. While the problem of reproducibility and replication has been recognized in the Recommender Systems community, the need for a clear solution remains largely unmet, which motivates the main questions addressed in the present workshop.<\/jats:p>","DOI":"10.1145\/2641383.2641389","type":"journal-article","created":{"date-parts":[[2014,7,7]],"date-time":"2014-07-07T11:55:18Z","timestamp":1404734118000},"page":"29-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Report on the workshop on reproducibility and replication in recommender systems evaluation (RepSys)"],"prefix":"10.1145","volume":"48","author":[{"given":"Alejandro","family":"Bellog\u00edn","sequence":"first","affiliation":[{"name":"Centrum Wiskunde &amp; Informatica, The Netherlands and Universidad Aut\u00f3noma de Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pablo","family":"Castells","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alan","family":"Said","sequence":"additional","affiliation":[{"name":"Centrum Wiskunde &amp; Informatica, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Domonkos","family":"Tikk","sequence":"additional","affiliation":[{"name":"Gravity R&amp;D, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2014,6,26]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532511"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532512"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2532508"},{"key":"e_1_2_1_4_1","first-page":"591","volume-title":"ISMIR","author":"Bertin-Mahieux Thierry","year":"2011","unstructured":"Thierry Bertin-Mahieux , Daniel P. W. Ellis , Brian Whitman , and Paul Lamere . The million song dataset . In ISMIR , pages 591 -- 596 , 2011 . Thierry Bertin-Mahieux, Daniel P. W. Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In ISMIR, pages 591--596, 2011."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2508008"},{"key":"e_1_2_1_6_1","first-page":"8","volume-title":"KDD Cup","author":"Dror Gideon","year":"2012","unstructured":"Gideon Dror , Noam Koenigstein , Yehuda Koren , and Markus Weimer . The yahoo! music dataset and kdd-cup '11 . In KDD Cup , pages 8 -- 18 , 2012 . Gideon Dror, Noam Koenigstein, Yehuda Koren, and Markus Weimer. The yahoo! music dataset and kdd-cup '11. In KDD Cup, pages 8--18, 2012."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1810295.1810426"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532513"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532510"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532509"},{"key":"e_1_2_1_11_1","first-page":"267","volume-title":"Collaborative filtering and the missing at random assumption","author":"Marlin Benjamin M.","year":"2007","unstructured":"Benjamin M. Marlin , Richard S. Zemel , Sam T. Roweis , and Malcolm Slaney . Collaborative filtering and the missing at random assumption . In Ronald Parr and Linda C. van der Gaag, editors, UAI, pages 267 -- 275 . AUAI Press , 2007 . Benjamin M. Marlin, Richard S. Zemel, Sam T. Roweis, and Malcolm Slaney. Collaborative filtering and the missing at random assumption. In Ronald Parr and Linda C. van der Gaag, editors, UAI, pages 267--275. AUAI Press, 2007."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/2631387"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3_8"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1314498.1314577"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2422256.2422269"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the CHI 2013 Workshop on the Replication of HCI Research, Paris, France, April 27-28, 2013, volume 976 of CEUR Workshop Proceedings. CEUR-WS.org","author":"Wilson Max L.","year":"2013","unstructured":"Max L. Wilson , Ed H. Chi , David Coyle , and Paul Resnick , editors. Proceedings of the CHI 2013 Workshop on the Replication of HCI Research, Paris, France, April 27-28, 2013, volume 976 of CEUR Workshop Proceedings. CEUR-WS.org , 2013 . Max L. Wilson, Ed H. Chi, David Coyle, and Paul Resnick, editors. Proceedings of the CHI 2013 Workshop on the Replication of HCI Research, Paris, France, April 27-28, 2013, volume 976 of CEUR Workshop Proceedings. CEUR-WS.org, 2013."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1189702.1189712"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2641383.2641389","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2641383.2641389","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:01:07Z","timestamp":1750276867000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2641383.2641389"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,6,26]]},"references-count":17,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2014,6,26]]}},"alternative-id":["10.1145\/2641383.2641389"],"URL":"https:\/\/doi.org\/10.1145\/2641383.2641389","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2014,6,26]]},"assertion":[{"value":"2014-06-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}