{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,22]],"date-time":"2026-11-22T01:55:58Z","timestamp":1795312558218,"version":"4.10.3"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2026,1,31]]},"abstract":"<jats:p>Graph-based techniques relying on neural networks and embeddings have gained attention as a way to develop Recommender Systems (RS) with several papers on the topic presented at SIGIR 2022 and 2023. Given the importance of ensuring that published research is methodologically sound and reproducible, in this paper, we analyze 10 graph-based RS papers, most of which were published at SIGIR 2022, and assess their impact on subsequent work published in SIGIR 2023. Our analysis reveals several critical points that require attention: (i) the prevalence of bad practices, such as erroneous data splits or information leakage between training and testing data, which call into question the validity of the results; (ii) frequent inconsistencies between the provided artifacts (source code and data) and their descriptions in the paper, causing uncertainty about what is actually being evaluated; and (iii) the preference for new or complex baselines that are weaker compared to simpler ones, creating the impression of continuous improvement even when, particularly for the Amazon-Book dataset, the state-of-the-art has significantly worsened. Due to these issues, we are unable to confirm the claims made in most of the papers that we examined and attempted to reproduce.<\/jats:p>","DOI":"10.1145\/3772275","type":"journal-article","created":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T16:07:21Z","timestamp":1761149241000},"page":"1-43","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing"],"prefix":"10.1145","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7103-2788","authenticated-orcid":false,"given":"Maurizio","family":"Ferrari Dacrema","sequence":"first","affiliation":[{"name":"ReMAP Lab, Politecnico di Milano, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2810-5800","authenticated-orcid":false,"given":"Michael","family":"Benigni","sequence":"additional","affiliation":[{"name":"ReMAP Lab, Politecnico di Milano, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9219-6239","authenticated-orcid":false,"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"Universit\u00e1 degli Studi di Padova, Padova, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,12,15]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477141"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3475944"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609489"},{"key":"e_1_3_3_6_2","first-page":"601","volume-title":"Proceedings of the 18th International Conference on Information and Knowledge Management (CIKM \u201909)","author":"Armstrong T. G.","year":"2009","unstructured":"T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. 2009. Improvements that don\u2019t add up: Ad-hoc retrieval results since 1998. In Proceedings of the 18th International Conference on Information and Knowledge Management (CIKM \u201909). ACM Press, New York, 601\u2013610."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3483595"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.4230\/DAGREP.13.1.68"},{"key":"e_1_3_3_10_2","first-page":"7","volume-title":"KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201907)","author":"Bell Robert M.","year":"2007","unstructured":"Robert M. Bell and Yehuda Koren. 2007. Improved neighborhood-based collaborative filtering. In KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201907), 7\u201314."},{"key":"e_1_3_3_11_2","unstructured":"Michael Benigni Maurizio Ferrari Dacrema and Dietmar Jannach. 2025. Diffusion recommender models and the illusion of progress: A concerning study of reproducibility and a conceptual mismatch. arXiv:2505.09364. Retrieved from https:\/\/arxiv.org\/abs\/2505.09364"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401036"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401036"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591645"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463249"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1629\/2048-7754.171"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531987"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531985"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3434185"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411901"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3020206"},{"key":"e_1_3_3_25_2","volume-title":"Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science","author":"Freire J.","year":"2016","unstructured":"J. Freire, N. Fuhr, and A. Rauber (Eds.). 2016. Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science. Schloss Dagstuhl\u2013Leibniz-Zentrum f\u00fcr Informatik, Germany."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532059"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-019-03895-5"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591647"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401063"},{"key":"e_1_3_3_30_2","unstructured":"Jos\u00e9 Miguel Hern\u00e1ndez-Lobato Matthew W. Hoffman Zoubin Ghahramani Welling Max and Cortes Corinna. 2014. Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014. 918\u2013926. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2014\/hash\/069d3bb002acd8d7dd095917f9efe4cb-Abstract.html"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609487"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.22"},{"key":"e_1_3_3_33_2","volume-title":"International vocabulary of metrology\u2014Basic and general concepts and associated terms (VIM)","author":"Joint Committee for Guides in Metrology (JCGM)","year":"2008","unstructured":"Joint Committee for Guides in Metrology (JCGM). 2008. International vocabulary of metrology\u2014Basic and general concepts and associated terms (VIM) (3rd ed.). JCGM 200:2012."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2882782"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1137\/S1064827500366124"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403226"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591723"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_3_39_2","unstructured":"J. Lin. 2022. Building a culture of reproducibility in academic research. arXiv:2212.13534. Retrieved from https:\/\/arxiv.org\/abs\/2212.13534"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463034"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45442-5_6"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591779"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532070"},{"key":"e_1_3_3_44_2","doi-asserted-by":"crossref","unstructured":"A. Lucic M. Bleeker M. de Rijke K. Sinha S. Jullien and R. Stojnic. 2022. Towards reproducible machine learning research in information retrieval. In The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201922) 3459\u20133461.","DOI":"10.1145\/3477495.3532686"},{"key":"e_1_3_3_45_2","first-page":"133","volume-title":"Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201921)","author":"Lv Q.","year":"2021","unstructured":"Q. Lv, M. Ding, Q. Liu, Y. Chen, W. Feng, S. He, C. Zhou, J. Jiang, Y. Dong, and J. Tang. 2021. Are we really making much progress? Revisiting, benchmarking and refining heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201921). ACM Press, New York, 133\u2013142."},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463254"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2023.103332"},{"key":"e_1_3_3_48_2","volume-title":"Reproducibility and Replicability in Science","author":"National Academies of Sciences, Engineering, and Medicine","year":"2019","unstructured":"National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. The National Academies Press, Washington, USA."},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.134"},{"key":"e_1_3_3_50_2","volume-title":"NISO RP-31-2021\u2014Reproducibility Badging and Definitions","author":"NISO","year":"2021","unstructured":"NISO. 2021. NISO RP-31-2021\u2014Reproducibility Badging and Definitions. National Information Standards Organization (NISO)."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aac4716"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/2955101"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532014"},{"issue":"164","key":"e_1_3_3_54_2","first-page":"1","article-title":"Improving reproducibility in machine learning research. A report from the NeurIPS 2019 reproducibility program","volume":"22","author":"Pineau J.","year":"2021","unstructured":"J. Pineau, P. Vincent-Lamarre, K Sinha, V. Lariviere, A. Beygelzimer, F. d\u2019Alche Buc, E. Fox, and H. Larochelle. 2021. Improving reproducibility in machine learning research. A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22, 164 (May. 2021), 1\u201320.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2017.00076"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591665"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591666"},{"key":"e_1_3_3_58_2","first-page":"452","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI \u201909)","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI \u201909). AUAI Press, 452\u2013461. Retrieved from https:\/\/www.auai.org\/uai2009\/papers\/UAI2009_0139_48141db02b9f0b02bc7158819ebfa2c7.pdf"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412488"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/192844.192905"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3424672"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372071"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730307"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609488"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482264"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v42i3.18140"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3474273"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1708290115"},{"key":"e_1_3_3_69_2","volume-title":"Differential Equations and Linear Algebra","author":"Strang Gilbert","year":"2014","unstructured":"Gilbert Strang. 2014. Differential Equations and Linear Algebra. Wellesley-Cambridge Press Wellesley."},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531889"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591730"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330989"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331267"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591649"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3535101"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532000"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532058"},{"key":"e_1_3_3_78_2","first-page":"1129","volume-title":"Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201919)","author":"Yang W.","year":"2019","unstructured":"W. Yang, K. Lu, P. Yang, and J. Lin. 2019. Critically examining the \u201cneural hype\u201d: Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201919). ACM Press, New York, 1129\u20131132."},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532009"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591691"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531937"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591632"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591707"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532025"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3772275","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,16]],"date-time":"2025-12-16T04:09:00Z","timestamp":1765858140000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3772275"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,15]]},"references-count":83,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1,31]]}},"alternative-id":["10.1145\/3772275"],"URL":"https:\/\/doi.org\/10.1145\/3772275","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,15]]},"assertion":[{"value":"2024-07-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}