{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T23:43:42Z","timestamp":1774309422052,"version":"3.50.1"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T00:00:00Z","timestamp":1709769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"The Lebanese University Research Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>Evaluation of recommendation systems continues evolving, especially in recent years. There have been several attempts to standardize the assessment processes and propose replacement metrics better oriented toward measuring effective personalization. However, standard evaluation tools merely possess the capacity to provide a general overview of a system\u2019s performance; they lack consistency and effectiveness in their use, as evidenced by most recent studies on the topic. Furthermore, traditional evaluation techniques fail to detect potentially harmful data on small subsets. Moreover, they generally lack explainable features to interpret how such minor variations could affect the system\u2019s performance. This proposal focuses on data clustering for recommender evaluation and applies a cluster assessment technique to locate such performance issues. Our new approach, named<jats:italic>group validation<\/jats:italic>, aids in spotting critical performance variability in compact subsets of the system\u2019s data and unravels hidden weaknesses in predictions where such unfavorable variations generally go unnoticed with typical assessment methods. Group validation for recommenders is a modular evaluation layer that complements regular evaluation and includes a new unique perspective to the evaluation process. Additionally, it allows several applications to the recommender ecosystem, such as model evolution tests, fraud\/attack detection, and the capacity for hosting a hybrid model setup.<\/jats:p>","DOI":"10.1145\/3640820","type":"journal-article","created":{"date-parts":[[2024,1,19]],"date-time":"2024-01-19T12:11:09Z","timestamp":1705666269000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Group Validation in Recommender Systems: Framework for Multi-layer Performance Evaluation"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9497-0515","authenticated-orcid":false,"given":"Wissam","family":"Al Jurdi","sequence":"first","affiliation":[{"name":"University Bourgogne Franche-Comt\u00e9, FEMTO-ST Institute, CNRS, Montb\u00e9liard, France | LaRRIS, Faculty of Sciences, Lebanese University, Fanar, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3482-9154","authenticated-orcid":false,"given":"Jacques Bou","family":"Abdo","sequence":"additional","affiliation":[{"name":"University of Cincinnati - University of Nebraska at Kearney, Kearney, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9798-8390","authenticated-orcid":false,"given":"Jacques","family":"Demerjian","sequence":"additional","affiliation":[{"name":"LaRRIS, Faculty of Sciences, Lebanese University, Fanar, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0485-097X","authenticated-orcid":false,"given":"Abdallah","family":"Makhoul","sequence":"additional","affiliation":[{"name":"University Bourgogne Franche-Comt\u00e9, FEMTO-ST Institute, CNRS, Montb\u00e9liard, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,3,7]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Alejandro Bellog\u00edn and Alan Said. 2021. Improving accountability in recommender systems research through reproducibility. User Model User-Adap Inter 31 (2021) 941\u2013977.","DOI":"10.1007\/s11257-021-09302-x"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"Craig Macdonald. 2021. The Simpson\u2019s Paradox in the offline evaluation of recommendation systems. ACM Transactions on Information Systems 40 1 (2021).","DOI":"10.1145\/3458509"},{"key":"e_1_3_2_4_2","article-title":"RGRecSys: A toolkit for robustness evaluation of recommender systems","author":"Ovaisi Zohreh","year":"2022","unstructured":"Zohreh Ovaisi, Shelby Heinecke, Jia Li, Yongfeng Zhang, Elena Zheleva, and Caiming Xiong. 2022. RGRecSys: A toolkit for robustness evaluation of recommender systems. arXiv preprint arXiv:2201.04399 (2022).","journal-title":"arXiv preprint arXiv:2201.04399"},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","first-page":"772","DOI":"10.1109\/MSN53354.2021.00121","volume-title":"17th International Conference on Mobility, Sensing and Networking (MSN\u201921)","author":"Han Di","year":"2021","unstructured":"Di Han, Yifan Huang, Xiaotian Jing, and Junmin Liu. 2021. AND: Effective coupling of accuracy, novelty and diversity in the recommender system. In 17th International Conference on Mobility, Sensing and Networking (MSN\u201921). IEEE, 772\u2013777."},{"key":"e_1_3_2_6_2","first-page":"91","volume-title":"ACM SIGIR Forum","year":"2018","unstructured":"N. Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lind\u00e9n, Bernardo Magnini, Jian-Yun Nie, R. Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin M. Verspoor, Martijn C. Willemsen, and Justin Zobel. 2018. The Dagstuhl Perspectives Workshop on performance modeling and prediction. In ACM SIGIR Forum, Vol. 52. ACM New York, NY, 91\u2013101."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240347"},{"key":"e_1_3_2_8_2","first-page":"50","volume-title":"the Web Conference","author":"Argyriou Andreas","year":"2020","unstructured":"Andreas Argyriou, Miguel Gonz\u00e1lez-Fierro, and Le Zhang. 2020. Microsoft recommenders: Best practices for production-ready recommendation systems. In the Web Conference. 50\u201351."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412489"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463245"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482016"},{"key":"e_1_3_2_12_2","first-page":"141","volume-title":"15th ACM International Conference on Web Search and Data Mining","author":"Chin Jin Yao","year":"2022","unstructured":"Jin Yao Chin, Yile Chen, and Gao Cong. 2022. The datasets dilemma: How much do we really know about recommendation datasets? In 15th ACM International Conference on Web Search and Data Mining. 141\u2013149."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3478848"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_16"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3474234"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/1860702.1860709"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076116"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447780"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488200"},{"key":"e_1_3_2_21_2","volume-title":"SysML Conference","author":"Chung Yeounoh","year":"2018","unstructured":"Yeounoh Chung, Tim Kraska, Steven Euijong Whang, and Neoklis Polyzotis. 2018. Slice finder: Automated data slicing for model interpretability. In SysML Conference."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2019.2916074"},{"key":"e_1_3_2_23_2","first-page":"1550","volume-title":"IEEE 35th International Conference on Data Engineering (ICDE\u201919)","author":"Chung Yeounoh","year":"2019","unstructured":"Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. 2019. Slice finder: Automated data slicing for model validation. In IEEE 35th International Conference on Data Engineering (ICDE\u201919). IEEE, 1550\u20131553."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1967.1053964"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"Daniel B. Neill. 2015. Subset Scanning for Event and Pattern Detection. Springer International Publishing Cham 1\u201310.","DOI":"10.1007\/978-3-319-23519-6_1547-1"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-020-10011-7"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9868.2011.01014.x"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2827872"},{"key":"e_1_3_2_30_2","article-title":"Welch\u2019s t-test \u2014 Wikipedia, The Free Encyclopedia","author":"contributors Wikipedia","year":"2022","unstructured":"Wikipedia contributors. 2022. Welch\u2019s t-test \u2014 Wikipedia, The Free Encyclopedia. Retrieved from: https:\/\/en.wikipedia.org\/w\/index.php?title=Welch%27s_t-test&oldid=1091289937","journal-title":"Retrieved from:"},{"key":"e_1_3_2_31_2","article-title":"Effect Size \u2014 Wikipedia, The Free Encyclopedia","author":"contributors Wikipedia","year":"2022","unstructured":"Wikipedia contributors. 2022. Effect Size \u2014 Wikipedia, The Free Encyclopedia. Retrieved from: https:\/\/en.wikipedia.org\/w\/index.php?title=Effect_size&oldid=1102056368","journal-title":"Retrieved from:"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.4300\/JGME-D-12-00156.1"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203771587"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9118-4"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-018-3891-5"},{"key":"e_1_3_2_36_2","first-page":"263","volume-title":"9th International Conference on Cloud Computing, Data Science & Engineering (Confluence\u201919)","author":"Ahuja Rishabh","year":"2019","unstructured":"Rishabh Ahuja, Arun Solanki, and Anand Nayyar. 2019. Movie recommender system using K-Means clustering and K-Nearest Neighbor. In 9th International Conference on Cloud Computing, Data Science & Engineering (Confluence\u201919). IEEE, 263\u2013268."},{"key":"e_1_3_2_37_2","first-page":"1331","volume-title":"33rd Annual ACM Symposium on Applied Computing","author":"Zhao Qian","year":"2018","unstructured":"Qian Zhao, F. Maxwell Harper, Gediminas Adomavicius, and Joseph A. Konstan. 2018. Explicit or implicit feedback? Engagement or satisfaction? A field experiment on machine-learning-based recommender systems. In 33rd Annual ACM Symposium on Applied Computing. 1331\u20131340."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-29659-3"},{"key":"e_1_3_2_39_2","article-title":"Review of clustering-based recommender systems","author":"Beregovskaya Irina","year":"2021","unstructured":"Irina Beregovskaya and Mikhail Koroteev. 2021. Review of clustering-based recommender systems. arXiv preprint arXiv:2109.12839 (2021).","journal-title":"arXiv preprint arXiv:2109.12839"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772862"},{"key":"e_1_3_2_41_2","volume-title":"k-means++: The Advantages of Careful Seeding","author":"Arthur David","year":"2006","unstructured":"David Arthur and Sergei Vassilvitskii. 2006. k-means++: The Advantages of Careful Seeding. Technical Report. Stanford."},{"key":"e_1_3_2_42_2","first-page":"1","volume-title":"International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS\u201917)","author":"Mansur Farhin","year":"2017","unstructured":"Farhin Mansur, Vibha Patel, and Mihir Patel. 2017. A review on recommender systems. In International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS\u201917). IEEE, 1\u20136."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390446"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2610382"},{"key":"e_1_3_2_45_2","volume-title":"Novelty and Diversity Evaluation and Enhancement in Recommender Systems","author":"Vargas Sa\u00fal","year":"2015","unstructured":"Sa\u00fal Vargas. 2015. Novelty and Diversity Evaluation and Enhancement in Recommender Systems. Ph.D. Dissertation. PhD thesis, Universidad Aut\u00f3noma de Madrid, Spain."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313469"},{"key":"e_1_3_2_47_2","first-page":"739","volume-title":"International Conference on Agents and Artificial Intelligence (ICAART\u201919)","author":"Badran Miriam El Khoury","year":"2019","unstructured":"Miriam El Khoury Badran, Jacques Bou Abdo, Wissam Al Jurdi, and Jacques Demerjian. 2019. Adaptive serendipity for recommender systems: Let it find you. In International Conference on Agents and Artificial Intelligence (ICAART\u201919). 739\u2013745."},{"key":"e_1_3_2_48_2","first-page":"118","volume-title":"International Conference on Knowledge Science, Engineering and Management","author":"Jurdi Wissam Al","year":"2019","unstructured":"Wissam Al Jurdi, Chady Abou Jaoude, Miriam El Khoury Badran, Jacques Bou Abdo, Jacques Demerjian, and Abdallah Makhoul. 2019. SCCF parameter and similarity measure optimization and evaluation. In International Conference on Knowledge Science, Engineering and Management. Springer, 118\u2013127."},{"key":"e_1_3_2_49_2","first-page":"625","volume-title":"41st International ACM SIGIR Conference on Research & Development in Information Retrieval","author":"Amig\u00f3 Enrique","year":"2018","unstructured":"Enrique Amig\u00f3, Damiano Spina, and Jorge Carrillo-de Albornoz. 2018. An axiomatic analysis of diversity evaluation metrics: Introducing the rank-biased utility metric. In 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 625\u2013634."},{"key":"e_1_3_2_50_2","unstructured":"Wissam Al Jurdi. 2022. Group Validation in Recommender Systems: Framework for Multi-layer Performance Evaluation. Retrieved from: https:\/\/github.com\/wissamjur\/group-validation"},{"key":"e_1_3_2_51_2","article-title":"BPR: Bayesian personalized ranking from implicit feedback","author":"Rendle Steffen","year":"2012","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012).","journal-title":"arXiv preprint arXiv:1205.2618"},{"key":"e_1_3_2_52_2","unstructured":"Simon Funk. 2006. Netflix Update: Try This at Home. Retrieved from: https:\/\/sifter.org\/simon\/journal\/20061211.html"},{"key":"e_1_3_2_53_2","article-title":"Probabilistic matrix factorization","volume":"20","author":"Mnih Andriy","year":"2007","unstructured":"Andriy Mnih and Russ R. Salakhutdinov. 2007. Probabilistic matrix factorization. Adv. Neural Inf. Process. Syst. 20 (2007).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.02174"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10796-017-9782-y"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401167"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2014.12.011"},{"key":"e_1_3_2_58_2","unstructured":"Wissam Al Jurdi Miriam El Khoury Badran Chady Abou Jaoude Jacques Bou Abdo Jacques Demerjian and Abdallah Makhoul. 2018. Serendipity-aware noise detection system for recommender systems. In Proceedings of the Information and Knowledge Engineering 1 (2019) 107\u2013114."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.5555\/2838885"},{"key":"e_1_3_2_60_2","first-page":"1","volume-title":"IEEE\/ACS 19th International Conference on Computer Systems and Applications (AICCSA\u201922)","author":"Jurdi Wissam Al","year":"2022","unstructured":"Wissam Al Jurdi, Jacques Bou Abdo, Jacques Demerjian, and Abdallah Makhoul. 2022. Strategic attacks on recommender systems: An obfuscation scenario. In IEEE\/ACS 19th International Conference on Computer Systems and Applications (AICCSA\u201922). IEEE, 1\u20138."},{"issue":"1","key":"e_1_3_2_61_2","article-title":"A systematic review and research perspective on recommender systems","volume":"9","author":"Roy Deepjyoti","year":"2022","unstructured":"Deepjyoti Roy and Mala Dutta. 2022. A systematic review and research perspective on recommender systems. J. Big Data 9, 1 (2022), 59.","journal-title":"J. Big Data"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2020.113303"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640820","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3640820","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:41Z","timestamp":1750287041000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640820"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,7]]},"references-count":61,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3640820"],"URL":"https:\/\/doi.org\/10.1145\/3640820","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,7]]},"assertion":[{"value":"2022-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-22","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}