{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T17:51:57Z","timestamp":1790358717400,"version":"4.1.0"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>\n                    Countless new machine learning models are published every year and are reported to significantly advance the state-of-the-art in\n                    <jats:italic toggle=\"yes\">top-n<\/jats:italic>\n                    recommendation. However, earlier reproducibility studies indicate that progress in this area may be quite limited, due to widespread methodological issues, e.g., comparisons with untuned baseline models, creating an\n                    <jats:italic toggle=\"yes\">illusion of progress<\/jats:italic>\n                    . In this work, we examine whether these problems persist in today\u2019s research by attempting to reproduce nine SIGIR 2023 and 2024 recommendation algorithms based on Denoising Diffusion Probabilistic Models, a recent but rapidly expanding research area. Only 25% of reported results are fully reproducible and, since the original papers relied on\n                    <jats:italic toggle=\"yes\">weak baselines<\/jats:italic>\n                    , they do not establish the superiority of diffusion models over state-of-the-art methods. In our controlled evaluations, well-tuned simpler baselines consistently exceed the diffusion-based models\u2019 effectiveness reported in the original papers. Furthermore, we identify key mismatches between the characteristics of diffusion models and those of the traditional\n                    <jats:italic toggle=\"yes\">top-n<\/jats:italic>\n                    recommendation task, raising doubts about their suitability for recommendation. Moreover, in the analyzed papers, the generative capabilities of these models are constrained to a minimum. Overall, our results call for greater scientific rigor and a disruptive change in the research and publication culture in this area.\n                  <\/jats:p>","DOI":"10.1145\/3795792","type":"journal-article","created":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T21:18:53Z","timestamp":1770412733000},"page":"1-69","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-2810-5800","authenticated-orcid":false,"given":"Michael","family":"Benigni","sequence":"first","affiliation":[{"name":"ReMAP Lab, Politecnico di Milano","place":["Milano, Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7103-2788","authenticated-orcid":false,"given":"Maurizio","family":"Ferrari Dacrema","sequence":"additional","affiliation":[{"name":"ReMAP Lab, Politecnico di Milano","place":["Milano, Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4698-8507","authenticated-orcid":false,"given":"Dietmar","family":"Jannach","sequence":"additional","affiliation":[{"name":"University of Klagenfurt","place":["Klagenfurt, Austria"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,3,16]]},"reference":[{"key":"e_1_3_4_2_2","unstructured":"Ahmed M. Alaa Boris van Breugel Evgeny S. Saveliev and Mihaela van der Schaar. 2022. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In Proceedings of the International Conference on Machine Learning Kamalika Chaudhuri Stefanie Jegelka Le Song Csaba Szepesv\u00e1ri Gang Niu and Sivan Sabato (Eds.). PMLR 290\u2013306. Retrieved from https:\/\/proceedings.mlr.press\/v162\/alaa22a.html"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3503252.3531292"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609489"},{"key":"e_1_3_4_5_2","doi-asserted-by":"crossref","unstructured":"Timothy G. Armstrong Alistair Moffat William Webber and Justin Zobel. 2009. Improvements that don\u2019t add up: Ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management. 601\u2013610.","DOI":"10.1145\/1645953.1646031"},{"key":"e_1_3_4_6_2","volume-title":"Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education","author":"Bauer Christine","year":"2023","unstructured":"Christine Bauer, Maik Fr\u00f6be, Dietmar Jannach, Udo Kruschwitz, Paolo Rosso, Damiano Spina, and Nava Tintarev. 2023. Overcoming methodological challenges in information retrieval and recommender systems through awareness and education. In Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education 13, 1 (2023), 116\u2013133."},{"key":"e_1_3_4_7_2","first-page":"7","volume-title":"Proceedings of the KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Bell Robert M.","year":"2007","unstructured":"Robert M. Bell and Yehuda Koren. 2007. Improved neighborhood-based collaborative filtering. In Proceedings of the KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 7\u201314."},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2508006"},{"key":"e_1_3_4_9_2","unstructured":"Yoshua Bengio. 2020. Time to rethink the publication process in machine learning. Retrieved from https:\/\/yoshuabengio.org\/2020\/02\/26\/time-to-rethink-the-publication-process-in-machine-learning\/"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412259"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3596519"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v42i3.18145"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864721"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3475943"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3637066"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043958"},{"key":"e_1_3_4_17_2","first-page":"pages 403\u2013432","article-title":"On the theory of stochastic processes, with particular reference to applications","author":"Feller W.","year":"1945","unstructured":"W. Feller. 1945. On the theory of stochastic processes, with particular reference to applications. In Proceedings of the 1st Berkeley Symposium on Mathematical Statistics and Probability.pages 403\u2013432. Retrieved from http:\/\/digicoll.lib.berkeley.edu\/record\/112810Referenced by: MathSciNet [MR0027980].","journal-title":"Proceedings of the 1st Berkeley Symposium on Mathematical Statistics and Probability."},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3772275"},{"issue":"2","key":"e_1_3_4_19_2","article-title":"A troubling analysis of reproducibility and progress in recommender systems research","volume":"39","author":"Dacrema Maurizio Ferrari","year":"2021","unstructured":"Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. 2021. A troubling analysis of reproducibility and progress in recommender systems research. ACM Transactions on Information Systems 39, 2 (2021), 20:1\u201320:49.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_4_20_2","volume-title":"Proceedings of the 2019 ACM Conference on Recommender Systems. Copenhagen","author":"Dacrema Maurizio Ferrari","year":"2019","unstructured":"Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? a worrying analysis of recent neural recommendation approaches. In Proceedings of the 2019 ACM Conference on Recommender Systems. Copenhagen."},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.24963\/IJCAI.2020\/650"},{"key":"e_1_3_4_22_2","volume-title":"Proceedings of the 2020 International Conference on Information and Knowledge Management","author":"Dacrema Maurizio Ferrari","year":"2020","unstructured":"Maurizio Ferrari Dacrema, Federico Parroni, Paolo Cremonesi, and Dietmar Jannach. 2020. Critically examining the claimed value of convolutions over user-item embedding maps for recommender systems. In Proceedings of the 2020 International Conference on Information and Knowledge Management."},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2204.07610"},{"key":"e_1_3_4_24_2","volume-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 13th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence","author":"Gundersen Odd Erik","year":"2018","unstructured":"Odd Erik Gundersen and Sigbj\u00f8rn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 13th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence . Article 201, 8 pages."},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401063"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609487"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3608839"},{"key":"e_1_3_4_30_2","volume-title":"Proceedings of the 4th International Conference on Learning Representations","author":"Hidasi Bal\u00e1zs","year":"2016","unstructured":"Bal\u00e1zs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based Recommendations with Recurrent Neural Networks. In Proceedings of the 4th International Conference on Learning Representations."},{"key":"e_1_3_4_31_2","first-page":"12","volume-title":"Proceedings of the 34th International Conference on Neural Information Processing Systems.","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems. (Vancouver, BC, Canada), 12 pages."},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657742"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.22"},{"key":"e_1_3_4_34_2","volume-title":"Proceedings of the 11th ACM Conference on Recommender Systems","author":"Jannach Dietmar","year":"2017","unstructured":"Dietmar Jannach and Malte Ludewig. 2017. When recurrent neural networks meet the neighborhood for session-based recommendation. In Proceedings of the 11th ACM Conference on Recommender Systems. Como, Italy."},{"key":"e_1_3_4_35_2","volume-title":"Proceedings of the ACM RecSys Challenge Workshop","author":"Jannach Dietmar","year":"2020","unstructured":"Dietmar Jannach, Gabriel De Souza P. Moreira, and Even Oldridge. 2020. Why are deep learning models not consistently winning recommender systems competitions yet?. In Proceedings of the ACM RecSys Challenge Workshop. Online."},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2018.00035"},{"key":"e_1_3_4_37_2","first-page":"21696","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021","author":"Kingma Diederik P.","year":"2021","unstructured":"Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. 2021. On density estimation with diffusion models. In Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, Marc\u2019Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 21696\u201321707. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2021\/hash\/b578f2a52a0229873fefc2a4b06377fa-Abstract.html"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532513"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401944"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3535335"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2021.05.048"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645759"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609492"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09277-1"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467350"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.IPM.2023.103332"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0194889"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482291"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640457.3688073"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.134"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/2955101"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99736-6_45"},{"issue":"164","key":"e_1_3_4_54_2","first-page":"1","article-title":"Improving reproducibility in machine learning research (A report from the NeurIPS 2019 reproducibility program)","volume":"22","author":"Pineau Joelle","year":"2021","unstructured":"Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d\u2019Alche Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research (A report from the NeurIPS 2019 reproducibility program). Journal of Machine Learning Research 22, 164 (2021), 1\u201320. Retrieved from http:\/\/jmlr.org\/papers\/v22\/20-303.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_4_55_2","first-page":"452","volume-title":"UAI 2009, Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI 2009, Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452\u2013461. Retrieved from https:\/\/www.auai.org\/uai2009\/papers\/UAI2009_0139_48141db02b9f0b02bc7158819ebfa2c7.pdf"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412488"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3548486"},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/192844.192905"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372071"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","unstructured":"Harald Semmelrock Tony Ross\u2010Hellauer Simone Kopeinik Dieter Theiler Armin Haberl Stefan Thalmann and Dominik Kowald. 2025. Reproducibility in machine\u2010learning\u2010based research: Overview barriers and drivers. AI Mag. 46 2 (April 2025) 19. DOI:10.1002\/aaai.70002","DOI":"10.1002\/aaai.70002"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3705328.3748157"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730307"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609488"},{"key":"e_1_3_4_64_2","volume-title":"Proceedings of the 46th European Conference on Information Retrieval.","author":"Shehzad Faisal","year":"2024","unstructured":"Faisal Shehzad and Dietmar Jannach. 2024. Performance comparison of session-based recommendation algorithms based on GNNs. In Proceedings of the 46th European Conference on Information Retrieval."},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482264"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045358"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313710"},{"key":"e_1_3_4_68_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023","author":"Stein George","year":"2023","unstructured":"George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L. Caterini, J. Eric T. Taylor, and Gabriel Loaiza-Ganem. 2023. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/0bc795afae289ed465a65a3b4b1f4eb7-Abstract-Conference.html"},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357895"},{"key":"e_1_3_4_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412489"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-10989-8_47"},{"key":"e_1_3_4_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313417"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441800"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591663"},{"key":"e_1_3_4_75_2","unstructured":"Ting-Ruen Wei and Yi Fang. 2025. Diffusion Models in Recommendation Systems: A Survey. arxiv:2501.10548. Retrieved from https:\/\/arxiv.org\/abs\/2501.10548"},{"key":"e_1_3_4_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462862"},{"key":"e_1_3_4_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3535101"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657719"},{"key":"e_1_3_4_79_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023","author":"Yang Zhengyi","year":"2023","unstructured":"Zhengyi Yang, Jiancan Wu, Zhicai Wang, Xiang Wang, Yancheng Yuan, and Xiangnan He. 2023. Generate what you prefer: Reshaping sequential recommendation via guided diffusion. In Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/4c5e2bcbf21bdf40d75fddad0bd43dc9-Abstract-Conference.html"},{"key":"e_1_3_4_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657825"},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531723"},{"key":"e_1_3_4_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657759"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3795792","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T07:28:52Z","timestamp":1773646132000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3795792"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,16]]},"references-count":81,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3795792"],"URL":"https:\/\/doi.org\/10.1145\/3795792","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,16]]},"assertion":[{"value":"2025-05-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}