{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T02:41:15Z","timestamp":1783737675777,"version":"3.55.0"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,1,6]],"date-time":"2021-01-06T00:00:00Z","timestamp":1609891200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2021,4,30]]},"abstract":"<jats:p>\n            The design of algorithms that generate personalized ranked item lists is a central topic of research in the field of recommender systems. In the past few years, in particular, approaches based on deep learning (neural) techniques have become dominant in the literature. For all of them, substantial progress over the state-of-the-art is claimed. However, indications exist of certain problems in today\u2019s research practice, e.g., with respect to the choice and optimization of the baselines used for comparison, raising questions about the published claims. To obtain a better understanding of the actual progress, we have compared recent results in the area of neural recommendation approaches based on collaborative filtering against a consistent set of existing simple baselines. The worrying outcome of the analysis of these recent works\u2014all were published at prestigious scientific conferences between 2015 and 2018\u2014is that 11 of the 12 reproducible neural approaches can be outperformed by conceptually simple methods, e.g., based on the nearest-neighbor heuristic or linear models. None of the computationally complex neural methods was actually consistently better than already existing learning-based techniques, e.g., using matrix factorization or linear models. In our analysis, we discuss common issues in today\u2019s research practice, which, despite the many papers that are published on the topic, have apparently led the field to a certain level of stagnation.\n            <jats:sup>1<\/jats:sup>\n          <\/jats:p>","DOI":"10.1145\/3434185","type":"journal-article","created":{"date-parts":[[2021,1,6]],"date-time":"2021-01-06T17:52:36Z","timestamp":1609955556000},"page":"1-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":155,"title":["A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research"],"prefix":"10.1145","volume":"39","author":[{"given":"Maurizio","family":"Ferrari Dacrema","sequence":"first","affiliation":[{"name":"Politecnico di Milano, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simone","family":"Boglio","sequence":"additional","affiliation":[{"name":"Politecnico di Milano, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paolo","family":"Cremonesi","sequence":"additional","affiliation":[{"name":"Politecnico di Milano, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dietmar","family":"Jannach","sequence":"additional","affiliation":[{"name":"University of Klagenfurt, Klagenfurt, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,1,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507189"},{"key":"e_1_2_1_2_1","first-page":"1","article-title":"Artist-driven layering and user\u2019s behaviour impact on recommendations in a playlist continuation scenario. In Proceedings of the ACM Recommender Systems Challenge 2018","volume":"4","author":"Antenucci Sebastiano","year":"2018","unstructured":"Sebastiano Antenucci , Simone Boglio , Emanuele Chioso , Ervin Dervishaj , Shuwen Kang , Tommaso Scarlatti , and Maurizio Ferrari Dacrema . 2018 . Artist-driven layering and user\u2019s behaviour impact on recommendations in a playlist continuation scenario. In Proceedings of the ACM Recommender Systems Challenge 2018 . ACM , 4 : 1 -- 4 :6. DOI:https:\/\/doi.org\/10.1145\/3267471.3267475 10.1145\/3267471.3267475 Sebastiano Antenucci, Simone Boglio, Emanuele Chioso, Ervin Dervishaj, Shuwen Kang, Tommaso Scarlatti, and Maurizio Ferrari Dacrema. 2018. Artist-driven layering and user\u2019s behaviour impact on recommendations in a playlist continuation scenario. In Proceedings of the ACM Recommender Systems Challenge 2018. ACM, 4:1--4:6. DOI:https:\/\/doi.org\/10.1145\/3267471.3267475","journal-title":"ACM"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646031"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24592-8_12"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201907)","author":"Robert","unstructured":"Robert M. Bell and Yehuda Koren. 2007. Improved neighborhood-based collaborative filtering . In Proceedings of the KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201907) . 7--14. Robert M. Bell and Yehuda Koren. 2007. Improved neighborhood-based collaborative filtering. In Proceedings of the KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201907). 7--14."},{"key":"e_1_2_1_6_1","unstructured":"James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13 (Feb.2012) 281--305.  James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13 (Feb.2012) 281--305."},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"Shlomo Berkovsky Ivan Cantador and Domonkos Tikk (Eds.). 2019. Collaborative Recommendations\u2014Algorithms Practical Challenges and Applications. World Scientific.  Shlomo Berkovsky Ivan Cantador and Domonkos Tikk (Eds.). 2019. Collaborative Recommendations\u2014Algorithms Practical Challenges and Applications. World Scientific.","DOI":"10.1142\/11131"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918)","author":"Bharadhwaj Homanga","unstructured":"Homanga Bharadhwaj , Homin Park , and Brian Y. Lim . 2018. RecGAN: Recurrent generative adversarial networks for recommendation systems . In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918) . 372--376. DOI:https:\/\/doi.org\/10.1145\/3240323.3240383 10.1145\/3240323.3240383 Homanga Bharadhwaj, Homin Park, and Brian Y. Lim. 2018. RecGAN: Recurrent generative adversarial networks for recommendation systems. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918). 372--376. DOI:https:\/\/doi.org\/10.1145\/3240323.3240383"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 15th International Conference on Machine Learning (ICML\u201998)","author":"Billsus Daniel","unstructured":"Daniel Billsus and Michael J. Pazzani . 1998. Learning collaborative information filters . In Proceedings of the 15th International Conference on Machine Learning (ICML\u201998) . 46--54. Daniel Billsus and Michael J. Pazzani. 1998. Learning collaborative information filters. In Proceedings of the 15th International Conference on Machine Learning (ICML\u201998). 46--54."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI\u201998)","author":"Breese John S.","year":"1998","unstructured":"John S. Breese , David Heckerman , and Carl Kadie . 1998 . Empirical analysis of predictive algorithms for collaborative filtering . In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI\u201998) . 43--52. John S. Breese, David Heckerman, and Carl Kadie. 1998. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI\u201998). 43--52."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09371-3"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080797"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/462"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2812803"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2579244"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2209310.2209314"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864721"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org, 4706--4710","author":"Dacrema Maurizio Ferrari","year":"2020","unstructured":"Maurizio Ferrari Dacrema , Paolo Cremonesi , and Dietmar Jannach . 2020 . Methodological issues in recommender systems research (extended abstract) . In Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org, 4706--4710 . DOI:https:\/\/doi.org\/10.24963\/ijcai.2020\/650 10.24963\/ijcai.2020 Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2020. Methodological issues in recommender systems research (extended abstract). In Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org, 4706--4710. DOI:https:\/\/doi.org\/10.24963\/ijcai.2020\/650"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.2307\/1932409"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3209991"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741667"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM\u201920)","author":"Dacrema Maurizio Ferrari","year":"2020","unstructured":"Maurizio Ferrari Dacrema , Federico Parroni , Paolo Cremonesi , and Dietmar Jannach . 2020 . Critically examining the claimed value of convolutions over user-item embedding maps for recommender systems . In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM\u201920) . DOI:https:\/\/doi.org\/10.1145\/3340531.3411901 10.1145\/3340531.3411901 Maurizio Ferrari Dacrema, Federico Parroni, Paolo Cremonesi, and Dietmar Jannach. 2020. Critically examining the claimed value of convolutions over user-item embedding maps for recommender systems. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM\u201920). DOI:https:\/\/doi.org\/10.1145\/3340531.3411901"},{"key":"e_1_2_1_25_1","first-page":"108","article-title":"Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041)","volume":"6","author":"Freire Juliana","year":"2016","unstructured":"Juliana Freire , Norbert Fuhr , and Andreas Rauber . 2016 . Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041) . Dagstuhl Rep. 6 , 1 (2016), 108 -- 159 . Juliana Freire, Norbert Fuhr, and Andreas Rauber. 2016. Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041). Dagstuhl Rep. 6, 1 (2016), 108--159.","journal-title":"Dagstuhl Rep."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788579"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645745"},{"key":"e_1_2_1_28_1","volume-title":"Recommender Systems Handbook","author":"Gunawardana Asela","unstructured":"Asela Gunawardana and Guy Shani . 2015. Evaluating recommender systems . In Recommender Systems Handbook . Springer , 265--308. Asela Gunawardana and Guy Shani. 2015. Evaluating recommender systems. In Recommender Systems Handbook. Springer, 265--308."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/3304889.3304969"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_2_1_31_1","volume-title":"Advances in Neural Information Processing Systems 27","author":"Hern\u00e1ndez-Lobato Jos\u00e9 Miguel","unstructured":"Jos\u00e9 Miguel Hern\u00e1ndez-Lobato , Matthew W. Hoffman , and Zoubin Ghahramani . 2014. Predictive entropy search for efficient global optimization of black-box functions . In Advances in Neural Information Processing Systems 27 , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc. , 918--926. Retrieved from http:\/\/papers.nips.cc\/paper\/5324-predictive-entropy-search-for-efficient-global-optimization-of-black-box-functions.pdf. Jos\u00e9 Miguel Hern\u00e1ndez-Lobato, Matthew W. Hoffman, and Zoubin Ghahramani. 2014. Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 918--926. Retrieved from http:\/\/papers.nips.cc\/paper\/5324-predictive-entropy-search-for-efficient-global-optimization-of-black-box-functions.pdf."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201918)","author":"Hu Binbin","unstructured":"Binbin Hu , Chuan Shi , Wayne Xin Zhao , and Philip S. Yu . 2018. Leveraging meta-path based context for top-n recommendation with a neural co-attention model . In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201918) . 1531--1540. Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S. Yu. 2018. Leveraging meta-path based context for top-n recommendation with a neural co-attention model. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201918). 1531--1540."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.22"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959165"},{"key":"e_1_2_1_35_1","unstructured":"Doudou LaLoudouana and Mambobo Bonouliqui Tarare.2002. Data set selection. Retrieved from http:\/\/www.jmlg.org\/papers\/laloudouana03.pdf.  Doudou LaLoudouana and Mambobo Bonouliqui Tarare.2002. Data set selection. Retrieved from http:\/\/www.jmlg.org\/papers\/laloudouana03.pdf."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the RecSys Large Scale Recommender Systems Workshop.","author":"Levy Mark","year":"2013","unstructured":"Mark Levy and Kris Jack . 2013 . Efficient top-n recommendation by linear regression . In Proceedings of the RecSys Large Scale Recommender Systems Workshop. Mark Levy and Kris Jack. 2013. Efficient top-n recommendation by linear regression. In Proceedings of the RecSys Large Scale Recommender Systems Workshop."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098077"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308774.3308781"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458553.3458563"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2003.1167344"},{"key":"e_1_2_1_42_1","volume-title":"Lipton and Jacob Steinhardt","author":"Zachary","year":"2019","unstructured":"Zachary C. Lipton and Jacob Steinhardt . 2019 . Troubling trends in machine learning scholarship. Queue 17, 1, Article Pages 80 (Feb. 2019), 33 pages. DOI:https:\/\/doi.org\/10.1145\/3317287.3328534 10.1145\/3317287.3328534 Zachary C. Lipton and Jacob Steinhardt. 2019. Troubling trends in machine learning scholarship. Queue 17, 1, Article Pages 80 (Feb. 2019), 33 pages. DOI:https:\/\/doi.org\/10.1145\/3317287.3328534"},{"key":"e_1_2_1_43_1","volume-title":"Marco De Gemmis, and Giovanni Semeraro","author":"Lops Pasquale","year":"2011","unstructured":"Pasquale Lops , Marco De Gemmis, and Giovanni Semeraro . 2011 . Content-based recommender systems: State of the art and trends. In Recommender Systems Handbook. Springer , 73--105. Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. 2011. Content-based recommender systems: State of the art and trends. In Recommender Systems Handbook. Springer, 73--105."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-018-9209-6"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347041"},{"key":"e_1_2_1_46_1","volume-title":"Statistical and machine learning forecasting methods: Concerns and ways forward. PloS One 13, 3","author":"Makridakis Spyros","year":"2018","unstructured":"Spyros Makridakis , Evangelos Spiliotis , and Vassilios Assimakopoulos . 2018. Statistical and machine learning forecasting methods: Concerns and ways forward. PloS One 13, 3 ( 2018 ). Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2018. Statistical and machine learning forecasting methods: Concerns and ways forward. PloS One 13, 3 (2018)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800184"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210042"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the CHI Extended Abstracts on Human Factors in Computing Systems (CHI EA\u201906)","author":"McNee Sean M.","unstructured":"Sean M. McNee , John Riedl , and Joseph A. Konstan . 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems . In Proceedings of the CHI Extended Abstracts on Human Factors in Computing Systems (CHI EA\u201906) . 1097--1101. DOI:https:\/\/doi.org\/10.1145\/1125451.1125659 10.1145\/1125451.1125659 Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In Proceedings of the CHI Extended Abstracts on Human Factors in Computing Systems (CHI EA\u201906). 1097--1101. DOI:https:\/\/doi.org\/10.1145\/1125451.1125659"},{"key":"e_1_2_1_50_1","volume-title":"Web Mining: From Web to Semantic Web, Bettina Berendt, Andreas Hotho, Dunja Mladeni\u010d, Maarten van Someren","author":"Mobasher Bamshad","unstructured":"Bamshad Mobasher , Xin Jin , and Yanzan Zhou . 2004. Semantically enhanced collaborative filtering on the web . In Web Mining: From Web to Semantic Web, Bettina Berendt, Andreas Hotho, Dunja Mladeni\u010d, Maarten van Someren , Myra Spiliopoulou, and Gerd Stumme (Eds.). Springer Berlin , 57--76. Bamshad Mobasher, Xin Jin, and Yanzan Zhou. 2004. Semantically enhanced collaborative filtering on the web. In Web Mining: From Web to Semantic Web, Bettina Berendt, Andreas Hotho, Dunja Mladeni\u010d, Maarten van Someren, Myra Spiliopoulou, and Gerd Stumme (Eds.). Springer Berlin, 57--76."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/219"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.134"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the KDD Cup and Workshop. 39--42","author":"Paterek Arkadiusz","year":"2007","unstructured":"Arkadiusz Paterek . 2007 . Improving regularized singular value decomposition for collaborative filtering . In Proceedings of the KDD Cup and Workshop. 39--42 . Arkadiusz Paterek. 2007. Improving regularized singular value decomposition for collaborative filtering. In Proceedings of the KDD Cup and Workshop. 39--42."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2955101"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.81"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412488"},{"key":"e_1_2_1_57_1","unstructured":"Steffen Rendle Li Zhang and Yehuda Koren. 2019. On the difficulty of evaluating baselines: A study on recommender systems. Retrieved from http:\/\/arxiv.org\/abs\/1905.01395.  Steffen Rendle Li Zhang and Yehuda Koren. 2019. On the difficulty of evaluating baselines: A study on recommender systems. Retrieved from http:\/\/arxiv.org\/abs\/1905.01395."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/192844.192905"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959176"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240397"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372071"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the 5th International Conference on Computer and Information Science (ICIS\u201902)","volume":"1","author":"Sarwar Badrul","year":"2002","unstructured":"Badrul Sarwar , George Karypis , Joseph Konstan , and John Riedl . 2002 . Incremental singular value decomposition algorithms for highly scalable recommender systems . In Proceedings of the 5th International Conference on Computer and Information Science (ICIS\u201902) , Vol. 1 . Citeseer, 27--8. Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2002. Incremental singular value decomposition algorithms for highly scalable recommender systems. In Proceedings of the 5th International Conference on Computer and Information Science (ICIS\u201902), Vol. 1. Citeseer, 27--8."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313710"},{"key":"e_1_2_1_64_1","volume-title":"No. 4773-10","author":"Stodden Victoria","year":"2010","unstructured":"Victoria Stodden . 2010. The scientific method in practice: Reproducibility in the computational sciences. MIT Sloan Research Paper , No. 4773-10 ( 2010 ). Victoria Stodden. 2010. The scientific method in practice: Reproducibility in the computational sciences. MIT Sloan Research Paper, No. 4773-10 (2010)."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240361"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186154"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220086"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109900"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.84.4.327"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2009.932122"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959160"},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the 29th International Conference on Machine Learning (ICML\u201912)","author":"Wagstaff Kiri","year":"2012","unstructured":"Kiri Wagstaff . 2012 . Machine learning that matters . In Proceedings of the 29th International Conference on Machine Learning (ICML\u201912) . 529--536. Kiri Wagstaff. 2012. Machine learning that matters. In Proceedings of the 29th International Conference on Machine Learning (ICML\u201912). 529--536."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783273"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9060-1"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835837"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/446"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/447"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331340"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/509"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/510"},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918)","author":"Zheng Lei","unstructured":"Lei Zheng , Chun-Ta Lu , Fei Jiang , Jiawei Zhang , and Philip S. Yu . 2018. Spectral collaborative filtering . In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918) . 311--319. Lei Zheng, Chun-Ta Lu, Fei Jiang, Jiawei Zhang, and Philip S. Yu. 2018. Spectral collaborative filtering. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys\u201918). 311--319."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3434185","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3434185","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:34Z","timestamp":1750195474000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3434185"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,6]]},"references-count":81,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,4,30]]}},"alternative-id":["10.1145\/3434185"],"URL":"https:\/\/doi.org\/10.1145\/3434185","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,6]]},"assertion":[{"value":"2019-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-01-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}