{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T10:05:53Z","timestamp":1775815553365,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":109,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,11]],"date-time":"2021-07-11T00:00:00Z","timestamp":1625961600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,11]]},"DOI":"10.1145\/3404835.3463245","type":"proceedings-article","created":{"date-parts":[[2021,7,12]],"date-time":"2021-07-12T03:08:25Z","timestamp":1626059305000},"page":"2405-2414","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":89,"title":["Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation"],"prefix":"10.1145","author":[{"given":"Vito Walter","family":"Anelli","sequence":"first","affiliation":[{"name":"Politecnico di Bari, Bari, Italy"}]},{"given":"Alejandro","family":"Bellogin","sequence":"additional","affiliation":[{"name":"Aut\u00f3noma Madrid, Madrid, Spain"}]},{"given":"Antonio","family":"Ferrara","sequence":"additional","affiliation":[{"name":"Politecnico di Bari, Bari, Italy"}]},{"given":"Daniele","family":"Malitesta","sequence":"additional","affiliation":[{"name":"Politecnico di Bari, Bari, Italy"}]},{"given":"Felice Antonio","family":"Merra","sequence":"additional","affiliation":[{"name":"Politecnico di Bari, Bari, Italy"}]},{"given":"Claudio","family":"Pomo","sequence":"additional","affiliation":[{"name":"Politecnico di Bari, Bari, Italy"}]},{"given":"Francesco Maria","family":"Donini","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Tuscia, Viterbo, Italy"}]},{"given":"Tommaso","family":"Di Noia","sequence":"additional","affiliation":[{"name":"Politecnico di Bari, Bari, Italy"}]}],"member":"320","published-online":{"date-parts":[[2021,7,11]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314309"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109912"},{"key":"e_1_3_2_2_3_1","volume-title":"Proceedings of the Thirty-Second International Florida Artificial Intelligence Research Society Conference","author":"Abdollahpouri Himan","year":"2019","unstructured":"Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2019. Managing Popularity Bias in Recommender Systems with Personalized Re-Ranking. In Proceedings of the Thirty-Second International Florida Artificial Intelligence Research Society Conference, Sarasota, Florida, USA, May 19--22 2019, Roman Bart\u00e1k and Keith W. Brawner (Eds.). AAAI Press, 413--418."},{"key":"e_1_3_2_2_4_1","volume-title":"RecSys 2020 Challenge Workshop: Engagement Prediction on Twitter's Home Timeline. In RecSys 2020: Fourteenth ACM Conference on Recommender Systems","author":"Anelli Vito Walter","year":"2020","unstructured":"Vito Walter Anelli, Amra Delic, Gabriele Sottocornola, Jessie Smith, Nazareno Andrade, Luca Belli, Michael M. Bronstein, Akshay Gupta, Sofia Ira Ktena, Alexandre Lung-Yut-Fong, Frank Portman, Alykhan Tejani, Yuanpu Xie, Xiao Zhu, and Wenzhe Shi. 2020. RecSys 2020 Challenge Workshop: Engagement Prediction on Twitter's Home Timeline. In RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22--26, 2020, Rodrygo L. T. Santos, Leandro Balby Marinho, Elizabeth M. Daly, Li Chen, Kim Falk, Noam Koenigstein, and Edleno Silva de Moura (Eds.). ACM, 623--627."},{"key":"e_1_3_2_2_5_1","unstructured":"V. W. Anelli T. Di Noia E. Di Sciascio A. Ragone and J. Trotta. 2020. Semantic Interpretation of Top-N Recommendations. IEEE Transactions on Knowledge and Data Engineering (2020) 1--1."},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347010"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"crossref","unstructured":"Vito Walter Anelli Tommaso Di Noia Eugenio Di Sciascio Azzurra Ragone and Joseph Trotta. 2019 b. How to Make Latent Factors Interpretable by Feeding Factorization Machines with Knowledge Graphs. In The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference Auckland New Zealand October 26--30 2019 Proceedings Part I (Lecture Notes in Computer Science Vol. 11778) Chiara Ghidini Olaf Hartig Maria Maleshkova Vojtech Sv\u00e1 tek Isabel F. Cruz Aidan Hogan Jie Song Maxime Lefrancc ois and Fabien Gandon (Eds.). Springer 38--56.","DOI":"10.1007\/978-3-030-30793-6_3"},{"key":"e_1_3_2_2_8_1","volume-title":"ECIR 2019, Cologne, Germany, April 14--18, 2019, Proceedings, Part I (Lecture Notes in Computer Science","volume":"868","author":"Anelli Vito Walter","year":"2019","unstructured":"Vito Walter Anelli, Tommaso Di Noia, Eugenio Di Sciascio, Azzurra Ragone, and Joseph Trotta. 2019 c. Local Popularity and Time in top-N Recommendation. In Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14--18, 2019, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 11437), Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (Eds.). Springer, 861--868."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3418435"},{"key":"e_1_3_2_2_10_1","volume-title":"Proceedings of the 2011 ACM Conference on Recommender Systems, RecSys 2011","author":"Alejandro Bellog'i","year":"2011","unstructured":"Alejandro Bellog'i n, Pablo Castells, and Iv\u00e1n Cantador. 2011. Precision-oriented evaluation of recommender systems: an algorithmic comparison. In Proceedings of the 2011 ACM Conference on Recommender Systems, RecSys 2011, Chicago, IL, USA, October 23--27, 2011, Bamshad Mobasher, Robin D. Burke, Dietmar Jannach, and Gediminas Adomavicius (Eds.). ACM, 333--336."},{"key":"e_1_3_2_2_11_1","volume-title":"Improving Accountability in Recommender Systems Research Through Reproducibility. CoRR","author":"Alan Said Alejandro Bellog'i","year":"2021","unstructured":"Alejandro Bellog'i n and Alan Said. 2021. Improving Accountability in Recommender Systems Research Through Reproducibility. CoRR, Vol. abs\/2102.00482 (2021)."},{"key":"e_1_3_2_2_12_1","volume-title":"Proceedings of the 1st Workshop on Temporal Reasoning in Recommender Systems co-located with 11th International Conference on Recommender Systems (RecSys","author":"Alejandro Bellog'i","year":"2017","unstructured":"Alejandro Bellog'i n and Pablo S\u00e1 nchez. 2017. Revisiting Neighbourhood-Based Recommenders For Temporal Scenarios. In Proceedings of the 1st Workshop on Temporal Reasoning in Recommender Systems co-located with 11th International Conference on Recommender Systems (RecSys 2017), Como, Italy, August 27--31, 2017 (CEUR Workshop Proceedings, Vol. 1922), M\u00e1 ria Bielikov\u00e1, Veronika Bogina, Tsvi Kuflik, and Roy Sasson (Eds.). CEUR-WS.org, 40--44."},{"key":"e_1_3_2_2_13_1","volume-title":"Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Bennett James","year":"2007","unstructured":"James Bennett and Stan Lanning. 2007. The netflix prize. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Jose, California, USA, August 12--15, 2007. ACM ."},{"key":"e_1_3_2_2_14_1","volume-title":"Yoshua Bengio, and Bal\u00e1 zs K\u00e9 gl.","author":"Bergstra James","year":"2011","unstructured":"James Bergstra, R\u00e9 mi Bardenet, Yoshua Bengio, and Bal\u00e1 zs K\u00e9 gl. 2011. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12--14 December 2011, Granada, Spain, John Shawe-Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger (Eds.). 2546--2554."},{"key":"e_1_3_2_2_15_1","volume-title":"Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16--21 June 2013 (JMLR Workshop and Conference Proceedings","volume":"123","author":"Bergstra James","unstructured":"James Bergstra, Daniel Yamins, and David D. Cox. 2013. Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16--21 June 2013 (JMLR Workshop and Conference Proceedings, Vol. 28). JMLR.org, 115--123."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-012-9136-x"},{"key":"e_1_3_2_2_17_1","volume-title":"Novelty and Diversity in Recommender Systems","author":"Castells Pablo","unstructured":"Pablo Castells, Neil J. Hurley, and Saul Vargas. 2015. Novelty and Diversity in Recommender Systems. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 881--918."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271743"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080797"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2988450.2988454"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864721"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_2_2_23_1","volume-title":"Hamed Zamani, Alejandro Bellogin, and Tommaso Di Noia.","author":"Deldjoo Yashar","year":"2020","unstructured":"Yashar Deldjoo, Vito Walter Anelli, Hamed Zamani, Alejandro Bellogin, and Tommaso Di Noia. 2020. A flexible framework for evaluating user and item fairness in recommender systems. User Modeling and User-Adapted Interaction (2020), 1--47."},{"key":"e_1_3_2_2_24_1","volume-title":"LensKit for Python: Next-Generation Software for Recommender Systems Experiments. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management","author":"Ekstrand Michael D.","year":"2020","unstructured":"Michael D. Ekstrand. 2020. LensKit for Python: Next-Generation Software for Recommender Systems Experiments. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19--23, 2020, Mathieu d'Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudr\u00e9 -Mauroux (Eds.). ACM, 2999--3006."},{"key":"e_1_3_2_2_25_1","volume-title":"Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019","author":"Ekstrand Michael D.","year":"2019","unstructured":"Michael D. Ekstrand, Robin Burke, and Fernando Diaz. 2019 a. Fairness and discrimination in recommendation and retrieval. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, September 16--20, 2019, Toine Bogers, Alan Said, Peter Brusilovsky, and Domonkos Tikk (Eds.). ACM, 576--577."},{"key":"e_1_3_2_2_26_1","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019","author":"Ekstrand Michael D.","year":"2019","unstructured":"Michael D. Ekstrand, Robin Burke, and Fernando Diaz. 2019 b. Fairness and Discrimination in Retrieval and Recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21--25, 2019, Benjamin Piwowarski, Max Chevalier, \u00c9 ric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 1403--1404."},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043958"},{"key":"e_1_3_2_2_28_1","unstructured":"Ben Frederickson. 2018. Fast python collaborative filtering for implicit datasets."},{"key":"e_1_3_2_2_29_1","unstructured":"Simon Funk. 2006. Netflix update: Try this at home."},{"key":"e_1_3_2_2_30_1","volume-title":"Proceedings of KDD Cup 2011 competition, San Diego, CA, USA, 2011 (JMLR Proceedings","volume":"247","author":"Gantner Zeno","year":"2012","unstructured":"Zeno Gantner, Lucas Drumond, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2012. Personalized Ranking for Non-Uniformly Sampled Items. In Proceedings of KDD Cup 2011 competition, San Diego, CA, USA, 2011 (JMLR Proceedings, Vol. 18), Gideon Dror, Yehuda Koren, and Markus Weimer (Eds.). JMLR.org, 231--247."},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043989"},{"key":"e_1_3_2_2_32_1","volume-title":"Evaluating Recommender Systems","author":"Gunawardana Asela","unstructured":"Asela Gunawardana and Guy Shani. 2015. Evaluating Recommender Systems. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 265--308."},{"key":"e_1_3_2_2_33_1","volume-title":"Late-breaking Results and Workshop Proceedings of the 23rd Conference on User Modeling, Adaptation, and Personalization (UMAP 2015), Dublin, Ireland, June 29 - July 3, 2015 (CEUR Workshop Proceedings","author":"Guo Guibing","year":"2015","unstructured":"Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. 2015. LibRec: A Java Library for Recommender Systems. In Posters, Demos, Late-breaking Results and Workshop Proceedings of the 23rd Conference on User Modeling, Adaptation, and Personalization (UMAP 2015), Dublin, Ireland, June 29 - July 3, 2015 (CEUR Workshop Proceedings, Vol. 1388), Alexandra I. Cristea, Judith Masthoff, Alan Said, and Nava Tintarev (Eds.). CEUR-WS.org."},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/239"},{"key":"e_1_3_2_2_35_1","volume-title":"DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference. In 47th ACM\/IEEE Annual International Symposium on Computer Architecture, ISCA 2020","author":"Gupta Udit","year":"2020","unstructured":"Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu. 2020. DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference. In 47th ACM\/IEEE Annual International Symposium on Computer Architecture, ISCA 2020, Valencia, Spain, May 30 - June 3, 2020. IEEE, 982--995."},{"key":"e_1_3_2_2_36_1","volume-title":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12--17","author":"He Ruining","year":"2016","unstructured":"Ruining He and Julian J. McAuley. 2016. VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12--17, 2016, Phoenix, Arizona, USA, Dale Schuurmans and Michael P. Wellman (Eds.). AAAI Press, 144--150."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080777"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401063"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/308"},{"key":"e_1_3_2_2_40_1","volume-title":"Adversarial Personalized Ranking for Recommendation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018","author":"He Xiangnan","year":"2018","unstructured":"Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. 2018b. Adversarial Personalized Ranking for Recommendation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08--12, 2018, Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 355--364."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2018.2831682"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052639"},{"key":"e_1_3_2_2_44_1","volume-title":"22nd ACM International Conference on Information and Knowledge Management, CIKM'13","author":"Huang Po-Sen","year":"2013","unstructured":"Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In 22nd ACM International Conference on Information and Knowledge Management, CIKM'13, San Francisco, CA, USA, October 27 - November 1, 2013, Qi He, Arun Iyengar, Wolfgang Nejdl, Jian Pei, and Rajeev Rastogi (Eds.). ACM, 2333--2338."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.02174"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1944339.1944341"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864736"},{"key":"e_1_3_2_2_48_1","first-page":"1","article-title":"Logistic matrix factorization for implicit feedback data","volume":"27","author":"Johnson Christopher C","year":"2014","unstructured":"Christopher C Johnson. 2014. Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems, Vol. 27, 78 (2014), 1--9.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959134"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487589"},{"key":"e_1_3_2_2_51_1","volume-title":"Visually-Aware Fashion Recommendation and Design with Generative Image Models. In 2017 IEEE International Conference on Data Mining, ICDM 2017","author":"Kang Wang-Cheng","year":"2017","unstructured":"Wang-Cheng Kang, Chen Fang, Zhaowen Wang, and Julian J. McAuley. 2017. Visually-Aware Fashion Recommendation and Design with Generative Image Models. In 2017 IEEE International Conference on Data Mining, ICDM 2017, New Orleans, LA, USA, November 18--21, 2017, Vijay Raghavan, Srinivas Aluru, George Karypis, Lucio Miele, and Xindong Wu (Eds.). IEEE Computer Society, 207--216."},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959165"},{"key":"e_1_3_2_2_53_1","volume-title":"Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys 2013","author":"Joseph","year":"2013","unstructured":"Joseph A. Konstan and Gediminas Adomavicius. 2013. Toward identification and adoption of best practices in algorithmic recommender systems research. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys 2013, Hong Kong, China, October 12, 2013, Alejandro Bellog'i n, Pablo Castells, Alan Said, and Domonkos Tikk (Eds.). ACM, 23--28."},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401944"},{"key":"e_1_3_2_2_55_1","volume-title":"Bell","author":"Koren Yehuda","year":"2015","unstructured":"Yehuda Koren and Robert M. Bell. 2015. Advances in Collaborative Filtering. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 77--118."},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2009.263"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639726"},{"key":"e_1_3_2_2_58_1","volume-title":"On Sampled Metrics for Item Recommendation. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","author":"Krichene Walid","year":"2020","unstructured":"Walid Krichene and Steffen Rendle. 2020. On Sampled Metrics for Item Recommendation. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23--27, 2020, Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (Eds.). ACM, 1748--1757."},{"key":"e_1_3_2_2_59_1","volume-title":"Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender Systems co-located with 9th ACM Conference on Recommender Systems (RecSys","author":"Kula Maciej","year":"2015","unstructured":"Maciej Kula. 2015. Metadata Embeddings for User and Item Cold-start Recommendations. In Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender Systems co-located with 9th ACM Conference on Recommender Systems (RecSys 2015), Vienna, Austria, September 16--20, 2015. (CEUR Workshop Proceedings, Vol. 1448), Toine Bogers and Marijn Koolen (Eds.). CEUR-WS.org, 14--21."},{"key":"e_1_3_2_2_60_1","unstructured":"Maciej Kula. 2017. Spotlight. https:\/\/github.com\/maciejkula\/spotlight ."},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972757.43"},{"key":"e_1_3_2_2_62_1","volume-title":"On Sampling Top-K Recommendation Evaluation. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","author":"Li Dong","year":"2020","unstructured":"Dong Li, Ruoming Jin, Jing Gao, and Zhi Liu. 2020. On Sampling Top-K Recommendation Evaluation. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23--27, 2020, Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (Eds.). ACM, 2114--2124."},{"key":"e_1_3_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_2_2_64_1","first-page":"76","article-title":"Amazon.com Recommendations","volume":"7","author":"Linden Greg","year":"2003","unstructured":"Greg Linden, Brent Smith, and Jeremy York. 2003. Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Comput., Vol. 7, 1 (2003), 76--80.","journal-title":"Item-to-Item Collaborative Filtering. IEEE Internet Comput."},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080658"},{"key":"e_1_3_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347041"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09277-1"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2014.2308433"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458205"},{"key":"e_1_3_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935877"},{"key":"e_1_3_2_2_71_1","volume-title":"Extended Abstracts Proceedings of the 2006 Conference on Human Factors in Computing Systems, CHI 2006, Montr\u00e9 al, Qu\u00e9 bec, Canada, April 22--27","author":"McNee Sean M.","year":"2006","unstructured":"Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being accurate is not enough: how accuracy metrics have hurt recommender systems. In Extended Abstracts Proceedings of the 2006 Conference on Human Factors in Computing Systems, CHI 2006, Montr\u00e9 al, Qu\u00e9 bec, Canada, April 22--27, 2006, Gary M. Olson and Robin Jeffries (Eds.). ACM, 1097--1101."},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3418479"},{"key":"e_1_3_2_2_73_1","volume-title":"SLIM: Sparse Linear Methods for Top-N Recommender Systems. In 11th IEEE International Conference on Data Mining, ICDM 2011","author":"Ning Xia","year":"2011","unstructured":"Xia Ning and George Karypis. 2011. SLIM: Sparse Linear Methods for Top-N Recommender Systems. In 11th IEEE International Conference on Data Mining, ICDM 2011, Vancouver, BC, Canada, December 11--14, 2011, Diane J. Cook, Jian Pei, Wei Wang, Osmar R. Za\"i ane, and Xindong Wu (Eds.). IEEE Computer Society, 497--506."},{"key":"e_1_3_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159728"},{"key":"e_1_3_2_2_75_1","volume-title":"I-SEMANTICS '12","author":"Noia Tommaso Di","year":"2012","unstructured":"Tommaso Di Noia, Roberto Mirizzi, Vito Claudio Ostuni, Davide Romito, and Markus Zanker. 2012. Linked open data to support content-based recommender systems. In I-SEMANTICS 2012 - 8th International Conference on Semantic Systems, I-SEMANTICS '12, Graz, Austria, September 5--7, 2012, Valentina Presutti and Helena Sofia Pinto (Eds.). ACM, 1--8."},{"key":"e_1_3_2_2_76_1","volume-title":"Factorization Machines. In ICDM 2010, The 10th IEEE International Conference on Data Mining","author":"Rendle Steffen","year":"2010","unstructured":"Steffen Rendle. 2010. Factorization Machines. In ICDM 2010, The 10th IEEE International Conference on Data Mining, Sydney, Australia, 14--17 December 2010, Geoffrey I. Webb, Bing Liu, Chengqi Zhang, Dimitrios Gunopulos, and Xindong Wu (Eds.). IEEE Computer Society, 995--1000."},{"key":"e_1_3_2_2_77_1","volume-title":"UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18--21","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18--21, 2009, Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452--461."},{"key":"e_1_3_2_2_78_1","volume-title":"Anderson","author":"Rendle Steffen","year":"2020","unstructured":"Steffen Rendle, Walid Krichene, Li Zhang, and John R. Anderson. 2020. Neural Collaborative Filtering vs. Matrix Factorization Revisited. In RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22--26, 2020, Rodrygo L. T. Santos, Leandro Balby Marinho, Elizabeth M. Daly, Li Chen, Kim Falk, Noam Koenigstein, and Edleno Silva de Moura (Eds.). ACM, 240--248."},{"key":"e_1_3_2_2_79_1","volume-title":"On the Difficulty of Evaluating Baselines: A Study on Recommender Systems. CoRR","author":"Rendle Steffen","year":"2019","unstructured":"Steffen Rendle, Li Zhang, and Yehuda Koren. 2019. On the Difficulty of Evaluating Baselines: A Study on Recommender Systems. CoRR, Vol. abs\/1905.01395 (2019)."},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/192844.192905"},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645746"},{"key":"e_1_3_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645712"},{"key":"e_1_3_2_2_83_1","article-title":"Cornac: A Comparative Framework for Multimodal Recommender Systems","volume":"21","author":"Salah Aghiles","year":"2020","unstructured":"Aghiles Salah, Quoc-Tuan Truong, and Hady W. Lauw. 2020. Cornac: A Comparative Framework for Multimodal Recommender Systems. J. Mach. Learn. Res., Vol. 21 (2020), 95:1--95:5.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_2_84_1","volume-title":"Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems","author":"Salakhutdinov Ruslan","year":"2007","unstructured":"Ruslan Salakhutdinov and Andriy Mnih. 2007. Probabilistic Matrix Factorization. In Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3--6, 2007, John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis (Eds.). Curran Associates, Inc., 1257--1264."},{"key":"e_1_3_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372071"},{"key":"e_1_3_2_2_86_1","volume-title":"CEUR Workshop Proceedings","volume":"811","author":"Schr\u00f6der G.","year":"2011","unstructured":"G. Schr\u00f6der, M. Thiele, and W. Lehner. 2011. Setting goals and choosing metrics for recommender system evaluations. CEUR Workshop Proceedings, Vol. 811 (2011), 78--85."},{"key":"e_1_3_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742726"},{"key":"e_1_3_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412489"},{"key":"e_1_3_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2019.2893638"},{"key":"e_1_3_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159656"},{"key":"e_1_3_2_2_91_1","volume-title":"Proceedings of the Workshop on Recommendation in Multi-stakeholder Environments co-located with the 13th ACM Conference on Recommender Systems (RecSys","author":"Tsintzou Virginia","year":"2019","unstructured":"Virginia Tsintzou, Evaggelia Pitoura, and Panayiotis Tsaparas. 2019. Bias Disparity in Recommendation Systems. In Proceedings of the Workshop on Recommendation in Multi-stakeholder Environments co-located with the 13th ACM Conference on Recommender Systems (RecSys 2019), Copenhagen, Denmark, September 20, 2019 (CEUR Workshop Proceedings, Vol. 2440), Robin Burke, Himan Abdollahpouri, Edward C. Malthouse, K. P. Thai, and Yongfeng Zhang (Eds.). CEUR-WS.org."},{"key":"e_1_3_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240347"},{"key":"e_1_3_2_2_93_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2610382"},{"key":"e_1_3_2_2_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043955"},{"key":"e_1_3_2_2_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080786"},{"key":"e_1_3_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331267"},{"key":"e_1_3_2_2_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835837"},{"key":"e_1_3_2_2_98_1","doi-asserted-by":"publisher","DOI":"10.5555\/3172077.3172324"},{"key":"e_1_3_2_2_99_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/447"},{"key":"e_1_3_2_2_100_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159681"},{"key":"e_1_3_2_2_101_1","doi-asserted-by":"publisher","DOI":"10.14778\/2311906.2311916"},{"key":"e_1_3_2_2_102_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2019.00087"},{"key":"e_1_3_2_2_103_1","volume-title":"SIGIR 2003: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Zhai ChengXiang","year":"2003","unstructured":"ChengXiang Zhai, William W. Cohen, and John D. Lafferty. 2003. Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In SIGIR 2003: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 28 - August 1, 2003, Toronto, Canada, Charles L. A. Clarke, Gordon V. Cormack, Jamie Callan, David Hawking, and Alan F. Smeaton (Eds.). ACM, 10--17."},{"key":"e_1_3_2_2_104_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972764.58"},{"key":"e_1_3_2_2_105_1","volume-title":"RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms. CoRR","author":"Zhao Wayne Xin","year":"2020","unstructured":"Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Kaiyuan Li, Yushuo Chen, Yujie Lu, Hui Wang, Changxin Tian, Xingyu Pan, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-Rong Wen. 2020. RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms. CoRR, Vol. abs\/2011.01731 (2020)."},{"key":"e_1_3_2_2_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219823"},{"key":"e_1_3_2_2_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441820"},{"key":"e_1_3_2_2_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271795"},{"key":"e_1_3_2_2_109_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401177"}],"event":{"name":"SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval","location":"Virtual Event Canada","acronym":"SIGIR '21","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404835.3463245","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3404835.3463245","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:44Z","timestamp":1750191464000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404835.3463245"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,11]]},"references-count":109,"alternative-id":["10.1145\/3404835.3463245","10.1145\/3404835"],"URL":"https:\/\/doi.org\/10.1145\/3404835.3463245","relation":{},"subject":[],"published":{"date-parts":[[2021,7,11]]},"assertion":[{"value":"2021-07-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}