{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T02:57:11Z","timestamp":1779332231628,"version":"3.51.4"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T00:00:00Z","timestamp":1709769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>The extent to which popularity bias is propagated by media recommender systems is a current topic within the community, as is the uneven propagation among users with varying interests for niche items. Recent work focused on exactly this topic, with movies being the domain of interest. Later on, two different research teams reproduced the methodology in the domains of music and books, respectively. The results across the different domains diverge. In this paper, we reproduce the three studies and identify four aspects that are relevant in investigating the differences in results: data, algorithms, division of users in groups and evaluation strategy. We run a set of experiments in which we measure general popularity bias propagation and unfair treatment of certain users with various combinations of these aspects. We conclude that all aspects account to some degree for the divergence in results, and should be carefully considered in future studies. Further, we find that the divergence in findings can be in large part attributed to the choice of evaluation strategy.<\/jats:p>","DOI":"10.1145\/3637066","type":"journal-article","created":{"date-parts":[[2023,12,11]],"date-time":"2023-12-11T11:24:19Z","timestamp":1702293859000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Reproducing Popularity Bias in Recommendation: The Effect of Evaluation Strategies"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8888-2869","authenticated-orcid":false,"given":"Savvina","family":"Daniil","sequence":"first","affiliation":[{"name":"Centrum Wiskunde &amp; Informatica, Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0187-9873","authenticated-orcid":false,"given":"Mirjam","family":"Cuper","sequence":"additional","affiliation":[{"name":"National Library of the Netherlands, The Hague, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5385-7695","authenticated-orcid":false,"given":"Cynthia C. S.","family":"Liem","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7748-4715","authenticated-orcid":false,"given":"Jacco","family":"van Ossenbruggen","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6865-0021","authenticated-orcid":false,"given":"Laura","family":"Hollink","sequence":"additional","affiliation":[{"name":"Centrum Wiskunde &amp; Informatica, Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,3,7]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314309"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.5555\/AAI28092057"},{"key":"e_1_3_3_4_2","volume-title":"RecSys Workshop on Recommendation in Multistakeholder Environments (RMSE \u201919)","author":"Abdollahpouri Himan","year":"2019","unstructured":"Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2019. The unfairness of popularity bias in recommendation. In RecSys Workshop on Recommendation in Multistakeholder Environments (RMSE \u201919). https:\/\/ceur-ws.org\/Vol-2440\/paper4.pdfRecSys Workshop on Recommendation in Multistakeholder Environments (RMSE); Conference date: 20-09-2019."},{"key":"e_1_3_3_5_2","volume-title":"The Long Tail: Why the Future of Business is Selling Less of More","author":"Anderson Chris","year":"2006","unstructured":"Chris Anderson. 2006. The Long Tail: Why the Future of Business is Selling Less of More. Hachette UK."},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-016-9174-x"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-017-9312-z"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3412841.3442123"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/1722149.1722154"},{"issue":"1","key":"e_1_3_3_10_2","article-title":"How algorithmic popularity bias hinders or promotes quality","volume":"8","author":"Ciampaglia Giovanni Luca","year":"2018","unstructured":"Giovanni Luca Ciampaglia, Azadeh Nematzadeh, Filippo Menczer, and Alessandro Flammini. 2018. How algorithmic popularity bias hinders or promotes quality. Scientific Reports 8, 1 (2018), 15951.","journal-title":"Scientific Reports"},{"key":"e_1_3_3_11_2","first-page":"1","article-title":"Pearson correlation coefficient","author":"Cohen Israel","year":"2009","unstructured":"Israel Cohen, Yiteng Huang, Jingdong Chen, and Jacob Benesty. 2009. Pearson correlation coefficient. Noise Reduction in Speech Processing (2009), 1\u20134.","journal-title":"Noise Reduction in Speech Processing"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v42i3.18145"},{"key":"e_1_3_3_13_2","first-page":"172","volume-title":"Conference on Fairness, Accountability and Transparency","author":"Ekstrand Michael D.","year":"2018","unstructured":"Michael D. Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D. Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness. In Conference on Fairness, Accountability and Transparency. PMLR, 172\u2013186."},{"issue":"5","key":"e_1_3_3_14_2","article-title":"Investigating the impact of recommender systems on user-based and item-based popularity bias","volume":"58","author":"Elahi Mehdi","year":"2021","unstructured":"Mehdi Elahi, Danial Khosh Kholgh, Mohammad Sina Kiarostami, Sorush Saghari, Shiva Parsa Rad, and Marko Tkal\u010di\u010d. 2021. Investigating the impact of recommender systems on user-based and item-based popularity bias. Information Processing & Management 58, 5 (2021), 102655.","journal-title":"Information Processing & Management"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_15"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v39i3.2816"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11503"},{"key":"e_1_3_3_19_2","doi-asserted-by":"crossref","unstructured":"Benjamin Haibe-Kains George Alexandru Adam Ahmed Hosny Farnoosh Khodakarami Massive Analysis Quality Control (MAQC) Society Board of Directors Levi Waldron Bo Wang Chris McIntosh Anna Goldenberg Anshul Kundaje Casey S. Greene Tamara Broderick Michael M. Hoffman Jeffrey T. Leek Keegan Korthauer Wolfgang Huber Alvis Brazma Joelle Pineau Robert Tibshirani Trevor Hastie John P. A. Ioannidis John Quackenbush and Hugo J. W. L. Aerts. 2020. Transparency and reproducibility in artificial intelligence. Nature 586 7829 (2020) E14\u2013E16.","DOI":"10.1038\/s41586-020-2766-y"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2827872"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1140\/epjb\/e2018-80374-8"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.02174"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.359.6377.725"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_3_25_2","article-title":"Correcting popularity bias by enhancing recommendation neutrality.","volume":"805","author":"Kamishima Toshihiro","year":"2014","unstructured":"Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2014. Correcting popularity bias by enhancing recommendation neutrality. RecSys Posters 805 (2014).","journal-title":"RecSys Posters"},{"key":"e_1_3_3_26_2","first-page":"1","volume-title":"Advances in Bias and Fairness in Information Retrieval: Third International Workshop, BIAS 2022, Stavanger, Norway, April 10, 2022, Revised Selected Papers","author":"Kowald Dominik","year":"2022","unstructured":"Dominik Kowald and Emanuel Lacic. 2022. Popularity bias in collaborative filtering-based multimedia recommender systems. In Advances in Bias and Fairness in Information Retrieval: Third International Workshop, BIAS 2022, Stavanger, Norway, April 10, 2022, Revised Selected Papers. Springer, 1\u201311."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45442-5_5"},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","unstructured":"Mohammadmehdi Naghiaei Hossein A. Rahmani and Mahdi Dehghan. 2022. The Unfairness of Popularity Bias in Book Recommendation. (2022). arxiv:cs.IR\/2202.13446","DOI":"10.1007\/978-3-031-09316-6_7"},{"key":"e_1_3_3_29_2","article-title":"Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program","volume":"22","author":"Pineau Joelle","year":"2021","unstructured":"Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivi\u00e8re, Alina Beygelzimer, Florence d\u2019Alch\u00e9 Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22 (2021).","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645746"},{"issue":"95","key":"e_1_3_3_31_2","first-page":"1","article-title":"Cornac: A comparative framework for multimodal recommender systems","volume":"21","author":"Salah Aghiles","year":"2020","unstructured":"Aghiles Salah, Quoc-Tuan Truong, and Hady W. Lauw. 2020. Cornac: A comparative framework for multimodal recommender systems. Journal of Machine Learning Research 21, 95 (2020), 1\u20135. http:\/\/jmlr.org\/papers\/v21\/19-805.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-72079-9_9"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911996.2912004"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482006"},{"key":"e_1_3_3_35_2","first-page":"1","article-title":"Algorithms are not neutral","author":"Stinson Catherine","year":"2022","unstructured":"Catherine Stinson. 2022. Algorithms are not neutral. AI and Ethics (2022), 1\u20138.","journal-title":"AI and Ethics"},{"issue":"5","key":"e_1_3_3_36_2","article-title":"Investigating and counteracting popularity bias in group recommendations","volume":"58","author":"Yalcin Emre","year":"2021","unstructured":"Emre Yalcin and Alper Bilge. 2021. Investigating and counteracting popularity bias in group recommendations. Information Processing & Management 58, 5 (2021), 102608.","journal-title":"Information Processing & Management"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.061"},{"key":"e_1_3_3_38_2","article-title":"Popularity bias is not always evil: Disentangling benign and harmful bias for recommendation","volume":"2109","author":"Zhao Zihao","year":"2021","unstructured":"Zihao Zhao, Jiawei Chen, Sheng Zhou, Xiangnan He, Xuezhi Cao, Fuzheng Zhang, and Wei Wu. 2021. Popularity bias is not always evil: Disentangling benign and harmful bias for recommendation. CoRR arXiv preprint arXiv:2109.07946 abs\/2109.07946 (2021). https:\/\/arxiv.org\/abs\/2109.07946","journal-title":"CoRR arXiv preprint arXiv:2109.07946"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467376"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/1060745.1060754"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637066","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637066","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:32Z","timestamp":1750287032000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637066"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,7]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3637066"],"URL":"https:\/\/doi.org\/10.1145\/3637066","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,7]]},"assertion":[{"value":"2022-11-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-02","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}