{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T17:13:39Z","timestamp":1776705219184,"version":"3.51.2"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>\n                    Offline evaluation is the predominant method for scientific research in recommender systems, enabling the comparison of alternative recommendation approaches using pre-collected datasets and computational metrics without involving human participants. However, recent reproducibility studies reveal that many offline evaluations in the literature lack scientific rigor or adopt research practices that cast doubt on the validity of their findings. A particularly common and ultimately catastrophic flaw is the comparison of newly proposed machine learning models against untuned or poorly tuned baseline models. Combined with limited reproducibility, such practices raise serious concerns about the true progress achieved by increasingly complex recommendation algorithms. In this editorial, we argue for stronger methodological standards and summarize essential guidance and best practices for conducting rigorous offline evaluations of recommender systems. Accordingly,\n                    <jats:italic toggle=\"yes\">ACM Transactions on Recommender Systems<\/jats:italic>\n                    will place increased emphasis on methodological rigor in all future submissions, with particular priority given to work that provides comprehensive reproducibility materials and clearly documents the tuning procedures used for baseline models.\n                  <\/jats:p>","DOI":"10.1145\/3800587","type":"journal-article","created":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T16:23:00Z","timestamp":1776702180000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Improving Methodological Standards in Recommender Systems Offline Evaluation"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4698-8507","authenticated-orcid":false,"given":"Dietmar","family":"Jannach","sequence":"first","affiliation":[{"name":"University of Klagenfurt","place":["Klagenfurt, Austria"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5842-838X","authenticated-orcid":false,"given":"Li","family":"Chen","sequence":"additional","affiliation":[{"name":"Hong Kong Baptist University","place":["Hong Kong, Hong Kong"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,4,20]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-019-09256-1"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609489"},{"key":"e_1_3_3_4_2","first-page":"601","volume-title":"Proceedings of the ACM Conference on Information and Knowledge Management (CIKM\u201909)","author":"Armstrong Timothy G.","year":"2009","unstructured":"Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements that don\u2019t add up: Ad-hoc retrieval results since 1998. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM\u201909). 601\u2013610."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-016-9174-x"},{"key":"e_1_3_3_6_2","first-page":"26","volume-title":"Proceedings of ACM RecSys 2019 Late-Breaking Results co-located with the 13th ACM Conference on Recommender Systems","volume":"2431","author":"Beel J\u00f6ran","year":"2019","unstructured":"J\u00f6ran Beel and Victor Brunel. 2019. Data pruning in recommender systems research: Best-practice or malpractice?. In Proceedings of ACM RecSys 2019 Late-Breaking Results co-located with the 13th ACM Conference on Recommender Systems, Vol. 2431. CEUR-WS.org, 26\u201330. Retrieved from https:\/\/ceur-ws.org\/Vol-2431\/paper6.pdf"},{"key":"e_1_3_3_7_2","volume-title":"Report from Dagstuhl Seminar 24211 \u2013 Evaluation Perspectives of Recommender Systems: Driving Research and Education","author":"Beel Joeran","year":"2024","unstructured":"Joeran Beel, Dietmar Jannach, Alan Said, Guy Shani, Tobias Vente, and Leon Wegmeth. 2024. Best-practices for offline evaluations of recommender systems. In Report from Dagstuhl Seminar 24211 \u2013 Evaluation Perspectives of Recommender Systems: Driving Research and Education, Christoph Bauer, Alan Said, and Eva Zangerle (Eds.). Dagstuhl. Seminar Report 24211."},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2508006"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412259"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3564284"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043958"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/343418"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159687"},{"issue":"4","key":"e_1_3_3_15_2","first-page":"13:1\u201313:19","article-title":"The Netflix recommender system: Algorithms, business value, and innovation","volume":"6","author":"Gomez-Uribe Carlos A.","year":"2015","unstructured":"Carlos A. Gomez-Uribe and Neil Hunt. 2015. The Netflix recommender system: Algorithms, business value, and innovation. Transactions on Management Information Systems 6, 4 (2015), 13:1\u201313:19.","journal-title":"Transactions on Management Information Systems"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.16905"},{"key":"e_1_3_3_17_2","volume-title":"Proceedings of the AAAI\u201918\/IAAI\u201918\/EAAI\u201918","author":"Gundersen Odd Erik","year":"2018","unstructured":"Odd Erik Gundersen and Sigbj\u00f8rn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. In Proceedings of the AAAI\u201918\/IAAI\u201918\/EAAI\u201918."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609487"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3608839"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2891406"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3746648"},{"key":"e_1_3_3_22_2","volume-title":"Recommender Systems Handbook","author":"Jannach Dietmar","year":"2021","unstructured":"Dietmar Jannach and Markus Zanker. 2021. Impact and value of recommender systems. In Recommender Systems Handbook, Francesco Ricci, Bracha Shapira, and Lior Rokach (Eds.). Springer US."},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-022-09351-w"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-022-09351-w"},{"issue":"1","key":"e_1_3_3_25_2","first-page":"2:1\u20132:42","article-title":"Diversity, serendipity, novelty, and coverage: A survey and empirical analysis of beyond-accuracy objectives in recommender systems","volume":"7","author":"Kaminskas Marius","year":"2016","unstructured":"Marius Kaminskas and Derek Bridge. 2016. Diversity, serendipity, novelty, and coverage: A survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems 7, 1 (2016), 2:1\u20132:42.","journal-title":"ACM Transactions on Interactive Intelligent Systems"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9118-4"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532513"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3535335"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3678004"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1230"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09277-1"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0194889"},{"key":"e_1_3_3_33_2","first-page":"1097","volume-title":"Extended Abstracts of the 2006 Conference on Human Factors in Computing Systems (CHI\u201906)","author":"McNee Sean M.","year":"2006","unstructured":"Sean M. McNee, John Riedl, and Jospeh A. Konstan. 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In Extended Abstracts of the 2006 Conference on Human Factors in Computing Systems (CHI\u201906). 1097\u20131101."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640457.3688073"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9115-7"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412488"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3548486"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/192844.192905"},{"key":"e_1_3_3_39_2","unstructured":"David Rohde Stephen Bonner Travis Dunlop Flavian Vasile and Alexandros Karatzoglou. 2018. Recogym: A reinforcement learning environment for the problem of product recommendation in online advertising. arXiv:1808.00720. Retrieved from https:\/\/arxiv.org\/abs\/1808.00720"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645746"},{"key":"e_1_3_3_41_2","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1145\/3287560.3287598","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*\u201919)","author":"Selbst Andrew D.","year":"2019","unstructured":"Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*\u201919). 59\u201368."},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1002\/aaai.70002"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3_8"},{"key":"e_1_3_3_44_2","volume-title":"Proceedings of the 19th ACM Conference on Recommender Systems","author":"Shehzad Faisal","year":"2025","unstructured":"Faisal Shehzad, Timo Breuer, Maria Maistro, and Dietmar Jannach. 2025. \u201cWe share our code online\u201d: Why this is not enough to ensure reproducibility and progress in recommender systems research. In Proceedings of the 19th ACM Conference on Recommender Systems."},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3609488"},{"key":"e_1_3_3_46_2","volume-title":"Proceedings of the 46th European Conference on Information Retrieval (ECIR)","author":"Shehzad Faisal","year":"2024","unstructured":"Faisal Shehzad and Dietmar Jannach. 2024. Performance comparison of session-based recommendation algorithms based on GNNs. In Proceedings of the 46th European Conference on Information Retrieval (ECIR)."},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412489"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3478848"},{"key":"e_1_3_3_49_2","first-page":"529","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201912)","author":"Wagstaff Kiri","year":"2012","unstructured":"Kiri Wagstaff. 2012. Machine learning that matters. In Proceedings of the International Conference on Machine Learning (ICML\u201912). 529\u2013536."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3556536"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2019.0876"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2024.3392335"},{"key":"e_1_3_3_53_2","unstructured":"Kuan Zou and Aixin Sun. 2025. A survey of real-world recommender systems: Challenges constraints and industrial perspectives. arxiv:2509.06002 [cs.IR]. Retrieved from https:\/\/arxiv.org\/abs\/2509.06002"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3800587","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T16:23:05Z","timestamp":1776702185000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3800587"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,20]]},"references-count":52,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3800587"],"URL":"https:\/\/doi.org\/10.1145\/3800587","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,20]]},"assertion":[{"value":"2026-02-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-20","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}