{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:42:32Z","timestamp":1787017352639,"version":"build-2736575974"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,9,13]],"date-time":"2021-09-13T00:00:00Z","timestamp":1631491200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,9,13]]},"DOI":"10.1145\/3460231.3478848","type":"proceedings-article","created":{"date-parts":[[2021,9,13]],"date-time":"2021-09-13T17:45:02Z","timestamp":1631555102000},"page":"708-713","source":"Crossref","is-referenced-by-count":54,"title":["Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?"],"prefix":"10.1145","author":[{"given":"Yan-Martin","family":"Tamm","sequence":"first","affiliation":[{"name":"Sber AI Lab, Russian Federation"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rinchin","family":"Damdinov","sequence":"additional","affiliation":[{"name":"Sber AI Lab, Russian Federation"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexey","family":"Vasilev","sequence":"additional","affiliation":[{"name":"Sber AI Lab, Russian Federation"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,9,13]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"crossref","unstructured":"Vito\u00a0Walter Anelli Alejandro Bellog\u00edn Antonio Ferrara Daniele Malitesta Felice\u00a0Antonio Merra Claudio Pomo Francesco\u00a0Maria Donini and Tommaso Di\u00a0Noia. 2021. Elliot: a comprehensive and rigorous framework for reproducible recommender systems evaluation. arXiv preprint arXiv:2103.02590(2021).  Vito\u00a0Walter Anelli Alejandro Bellog\u00edn Antonio Ferrara Daniele Malitesta Felice\u00a0Antonio Merra Claudio Pomo Francesco\u00a0Maria Donini and Tommaso Di\u00a0Noia. 2021. Elliot: a comprehensive and rigorous framework for reproducible recommender systems evaluation. arXiv preprint arXiv:2103.02590(2021).","DOI":"10.1145\/3404835.3463245"},{"key":"e_1_3_2_2_2_1","volume-title":"The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7","author":"Bradley P","year":"1997","unstructured":"Andrew\u00a0 P Bradley . 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7 ( 1997 ), 1145\u20131159. Andrew\u00a0P Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7 (1997), 1145\u20131159."},{"key":"e_1_3_2_2_3_1","volume-title":"On Target Item Sampling in Offline Recommender System Evaluation. In Fourteenth ACM Conference on Recommender Systems. 259\u2013268","author":"Ca\u00f1amares Roc\u00edo","year":"2020","unstructured":"Roc\u00edo Ca\u00f1amares and Pablo Castells . 2020 . On Target Item Sampling in Offline Recommender System Evaluation. In Fourteenth ACM Conference on Recommender Systems. 259\u2013268 . Roc\u00edo Ca\u00f1amares and Pablo Castells. 2020. On Target Item Sampling in Offline Recommender System Evaluation. In Fourteenth ACM Conference on Recommender Systems. 259\u2013268."},{"key":"e_1_3_2_2_4_1","unstructured":"Roc\u00edo Ca\u00f1amares Pablo Castells and Alistair Moffat. 2020. Offline evaluation options for recommender systems. Information Retrieval Journal(2020) 1\u201324.  Roc\u00edo Ca\u00f1amares Pablo Castells and Alistair Moffat. 2020. Offline evaluation options for recommender systems. Information Retrieval Journal(2020) 1\u201324."},{"key":"e_1_3_2_2_5_1","volume-title":"Proceedings of the 35th Annual ACM Symposium on Applied Computing. 1435\u20131442","author":"Carraro Diego","year":"2020","unstructured":"Diego Carraro and Derek Bridge . 2020 . Debiased offline evaluation of recommender systems: A weighted-sampling approach . In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 1435\u20131442 . Diego Carraro and Derek Bridge. 2020. Debiased offline evaluation of recommender systems: A weighted-sampling approach. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 1435\u20131442."},{"key":"e_1_3_2_2_6_1","volume-title":"DELF: A Dual-Embedding based Deep Latent Factor Model for Recommendation.. In IJCAI, Vol.\u00a018. 3329\u20133335.","author":"Cheng Weiyu","year":"2018","unstructured":"Weiyu Cheng , Yanyan Shen , Yanmin Zhu , and Linpeng Huang . 2018 . DELF: A Dual-Embedding based Deep Latent Factor Model for Recommendation.. In IJCAI, Vol.\u00a018. 3329\u20133335. Weiyu Cheng, Yanyan Shen, Yanmin Zhu, and Linpeng Huang. 2018. DELF: A Dual-Embedding based Deep Latent Factor Model for Recommendation.. In IJCAI, Vol.\u00a018. 3329\u20133335."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3434185"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_2_2_9_1","unstructured":"Darel13712. [n.d.]. rs metrics. https:\/\/github.com\/Darel13712\/rs_metrics  Darel13712. [n.d.]. rs metrics. https:\/\/github.com\/Darel13712\/rs_metrics"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963776"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3209991"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159687"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806504"},{"key":"e_1_3_2_2_14_1","unstructured":"Xiangnan He Xiaoyu Du Xiang Wang Feng Tian Jinhui Tang and Tat-Seng Chua. 2018. Outer product-based neural collaborative filtering. arXiv preprint arXiv:1808.03912(2018).  Xiangnan He Xiaoyu Du Xiang Wang Feng Tian Jinhui Tang and Tat-Seng Chua. 2018. Outer product-based neural collaborative filtering. arXiv preprint arXiv:1808.03912(2018)."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219965"},{"key":"e_1_3_2_2_17_1","unstructured":"Yitong Ji Aixin Sun Jie Zhang and Chenliang Li. 2021. A Critical Study on Data Leakage in Recommender System Offline Evaluation. arxiv:2010.11060\u00a0[cs.IR]  Yitong Ji Aixin Sun Jie Zhang and Chenliang Li. 2021. A Critical Study on Data Leakage in Recommender System Offline Evaluation. arxiv:2010.11060\u00a0[cs.IR]"},{"key":"e_1_3_2_2_18_1","unstructured":"Masahiro Kato Kenshi Abe Kaito Ariu and Shota Yasui. 2020. A Practical Guide of Off-Policy Evaluation for Bandit Problems. arXiv preprint arXiv:2010.12470(2020).  Masahiro Kato Kenshi Abe Kaito Ariu and Shota Yasui. 2020. A Practical Guide of Off-Policy Evaluation for Bandit Problems. arXiv preprint arXiv:2010.12470(2020)."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403226"},{"key":"e_1_3_2_2_20_1","unstructured":"Sber\u00a0AI Lab. [n.d.]. RePlay. https:\/\/github.com\/sberbank-ai-lab\/RePlay  Sber\u00a0AI Lab. [n.d.]. RePlay. https:\/\/github.com\/sberbank-ai-lab\/RePlay"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403262"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098077"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_2_2_24_1","volume-title":"Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. In Fourteenth ACM Conference on Recommender Systems. 681\u2013686","author":"Meng Zaiqiao","year":"2020","unstructured":"Zaiqiao Meng , Richard McCreadie , Craig Macdonald , and Iadh Ounis . 2020 . Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. In Fourteenth ACM Conference on Recommender Systems. 681\u2013686 . Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2020. Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. In Fourteenth ACM Conference on Recommender Systems. 681\u2013686."},{"key":"e_1_3_2_2_25_1","volume-title":"Evaluate and Tune Automated Recommender Systems. In Fourteenth ACM Conference on Recommender Systems. 588\u2013590","author":"Meng Zaiqiao","year":"2020","unstructured":"Zaiqiao Meng , Richard McCreadie , Craig Macdonald , Iadh Ounis , Siwei Liu , Yaxiong Wu , Xi Wang , Shangsong Liang , Yucheng Liang , Guangtao Zeng , 2020 . BETA-Rec: Build , Evaluate and Tune Automated Recommender Systems. In Fourteenth ACM Conference on Recommender Systems. 588\u2013590 . Zaiqiao Meng, Richard McCreadie, Craig Macdonald, Iadh Ounis, Siwei Liu, Yaxiong Wu, Xi Wang, Shangsong Liang, Yucheng Liang, Guangtao Zeng, 2020. BETA-Rec: Build, Evaluate and Tune Automated Recommender Systems. In Fourteenth ACM Conference on Recommender Systems. 588\u2013590."},{"key":"e_1_3_2_2_26_1","unstructured":"Microsoft. [n.d.]. microsoft\/recommenders. https:\/\/github.com\/microsoft\/recommenders  Microsoft. [n.d.]. microsoft\/recommenders. https:\/\/github.com\/microsoft\/recommenders"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.134"},{"key":"e_1_3_2_2_28_1","unstructured":"Steffen Rendle Li Zhang and Yehuda Koren. 2019. On the difficulty of evaluating baselines: A study on recommender systems. arXiv preprint arXiv:1905.01395(2019).  Steffen Rendle Li Zhang and Yehuda Koren. 2019. On the difficulty of evaluating baselines: A study on recommender systems. arXiv preprint arXiv:1905.01395(2019)."},{"key":"e_1_3_2_2_29_1","volume-title":"UCERSTI2 workshop at the 5th ACM conference on recommender systems","author":"Schr\u00f6der Gunnar","unstructured":"Gunnar Schr\u00f6der , Maik Thiele , and Wolfgang Lehner . 2011. Setting goals and choosing metrics for recommender system evaluations . In UCERSTI2 workshop at the 5th ACM conference on recommender systems , Chicago, USA , Vol .\u00a023. 53. Gunnar Schr\u00f6der, Maik Thiele, and Wolfgang Lehner. 2011. Setting goals and choosing metrics for recommender system evaluations. In UCERSTI2 workshop at the 5th ACM conference on recommender systems, Chicago, USA, Vol.\u00a023. 53."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313710"},{"key":"e_1_3_2_2_31_1","volume-title":"Proceedings of the 14th ACM Conference on Recommender Systems.","author":"Sun Zhu","year":"2020","unstructured":"Zhu Sun , Di Yu , Hui Fang , Jie Yang , Xinghua Qu , Jie Zhang , and Cong Geng . 2020 . Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison . In Proceedings of the 14th ACM Conference on Recommender Systems. Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020. Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison. In Proceedings of the 14th ACM Conference on Recommender Systems."},{"key":"e_1_3_2_2_32_1","unstructured":"Vasilev Tamm Damdinov. [n.d.]. Supplemental repository for this paper. https:\/\/github.com\/Darel13712\/compare_metrics  Vasilev Tamm Damdinov. [n.d.]. Supplemental repository for this paper. https:\/\/github.com\/Darel13712\/compare_metrics"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186154"},{"key":"e_1_3_2_2_34_1","volume-title":"Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. 392\u2013396","author":"Tian Mucun","year":"2020","unstructured":"Mucun Tian and Michael\u00a0 D Ekstrand . 2020 . Estimating Error and Bias in Offline Evaluation Results . In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. 392\u2013396 . Mucun Tian and Michael\u00a0D Ekstrand. 2020. Estimating Error and Bias in Offline Evaluation Results. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. 392\u2013396."},{"key":"e_1_3_2_2_35_1","unstructured":"Anne-Marie Tousch. 2019. How robust is MovieLens? A dataset analysis for recommender systems. arXiv preprint arXiv:1909.12799(2019).  Anne-Marie Tousch. 2019. How robust is MovieLens? A dataset analysis for recommender systems. arXiv preprint arXiv:1909.12799(2019)."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783273"},{"key":"e_1_3_2_2_37_1","unstructured":"Wubinzzu. [n.d.]. wubinzzu\/NeuRec. https:\/\/github.com\/wubinzzu\/NeuRec  Wubinzzu. [n.d.]. wubinzzu\/NeuRec. https:\/\/github.com\/wubinzzu\/NeuRec"},{"key":"e_1_3_2_2_38_1","volume-title":"IJCAI, Vol.\u00a017.","author":"Xue Hong-Jian","unstructured":"Hong-Jian Xue , Xinyu Dai , Jianbing Zhang , Shujian Huang , and Jiajun Chen . 2017. Deep Matrix Factorization Models for Recommender Systems .. In IJCAI, Vol.\u00a017. Melbourne, Australia , 3203\u20133209. Hong-Jian Xue, Xinyu Dai, Jianbing Zhang, Shujian Huang, and Jiajun Chen. 2017. Deep Matrix Factorization Models for Recommender Systems.. In IJCAI, Vol.\u00a017. Melbourne, Australia, 3203\u20133209."},{"key":"e_1_3_2_2_39_1","volume-title":"Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM.","author":"Yang Longqi","year":"2018","unstructured":"Longqi Yang , Eugene Bagdasaryan , Joshua Gruenstein , Cheng-Kang Hsieh , and Deborah Estrin . 2018 . OpenRec: A Modular Framework for Extensible and Adaptable Recommendation Algorithms . In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM. Longqi Yang, Eugene Bagdasaryan, Joshua Gruenstein, Cheng-Kang Hsieh, and Deborah Estrin. 2018. OpenRec: A Modular Framework for Extensible and Adaptable Recommendation Algorithms. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM."},{"key":"e_1_3_2_2_40_1","unstructured":"yoongi0428. [n.d.]. RecSys PyTorch. https:\/\/github.com\/yoongi0428\/RecSys_PyTorch  yoongi0428. [n.d.]. RecSys PyTorch. https:\/\/github.com\/yoongi0428\/RecSys_PyTorch"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556195.2556259"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/509"},{"key":"e_1_3_2_2_43_1","unstructured":"Wayne\u00a0Xin Zhao Shanlei Mu Yupeng Hou Zihan Lin Kaiyuan Li Yushuo Chen Yujie Lu Hui Wang Changxin Tian Xingyu Pan 2020. RecBole: Towards a Unified Comprehensive and Efficient Framework for Recommendation Algorithms. arXiv preprint arXiv:2011.01731(2020).  Wayne\u00a0Xin Zhao Shanlei Mu Yupeng Hou Zihan Lin Kaiyuan Li Yushuo Chen Yujie Lu Hui Wang Changxin Tian Xingyu Pan 2020. RecBole: Towards a Unified Comprehensive and Efficient Framework for Recommendation Algorithms. arXiv preprint arXiv:2011.01731(2020)."},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240343"}],"event":{"name":"RecSys '21: Fifteenth ACM Conference on Recommender Systems","location":"Amsterdam Netherlands","acronym":"RecSys '21","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGAI ACM Special Interest Group on Artificial Intelligence","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGIR ACM Special Interest Group on Information Retrieval","SIGCHI ACM Special Interest Group on Computer-Human Interaction","SIGecom Special Interest Group on Economics and Computation"]},"container-title":["Fifteenth ACM Conference on Recommender Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460231.3478848","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3460231.3478848","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:48:30Z","timestamp":1750178910000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460231.3478848"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,13]]},"references-count":44,"alternative-id":["10.1145\/3460231.3478848","10.1145\/3460231"],"URL":"https:\/\/doi.org\/10.1145\/3460231.3478848","relation":{},"subject":[],"published":{"date-parts":[[2021,9,13]]}}}