{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,10,10]],"date-time":"2023-10-10T12:28:49Z","timestamp":1696940929411},"reference-count":43,"publisher":"IGI Global","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,4]]},"abstract":"<jats:p>Replicating the results of the recommender system's evaluation is one of the main concerns in the area. This paper discusses this issue from different angles: 1) It investigates the uniformity of recommenders' evaluation designs presented in practice and their consistency with the theoretical side. 2) It highlights some of the issues and challenges that face recommenders' evaluators. 3) It provides stepwise guidelines for offline evaluation settings. A quantitative study of articles published in the last decade is studied. The search process is a manual search for a conference and a random search of journals. The results show a lack of uniformity and consistency in presenting the evaluation methods. Most of the articles miss at least one evaluation aspect (i.e., some aspects are not presented in the article). These discrepancies and the wide variety of evaluation settings lead to non-replicable experiments. To mitigate this issue, the paper proposes the recommender evaluation guidelines (REval), which presents a roadmap for recommender systems' evaluators.<\/jats:p>","DOI":"10.4018\/ijiit.2021040102","type":"journal-article","created":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T12:58:13Z","timestamp":1618837093000},"page":"25-45","source":"Crossref","is-referenced-by-count":1,"title":["Evaluating Recommender Systems"],"prefix":"10.4018","volume":"17","author":[{"given":"Alaa","family":"Alslaity","sequence":"first","affiliation":[{"name":"University of Ottawa, Canada"}]},{"given":"Thomas","family":"Tran","sequence":"additional","affiliation":[{"name":"University of Ottawa, Canada"}]}],"member":"2432","reference":[{"key":"ijiit.2021040102-0","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.99"},{"key":"ijiit.2021040102-1","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800192"},{"key":"ijiit.2021040102-2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-89656-4_32"},{"key":"ijiit.2021040102-3","doi-asserted-by":"publisher","DOI":"10.5815\/ijitcs.2019.12.01"},{"key":"ijiit.2021040102-4","doi-asserted-by":"crossref","unstructured":"Amatriain, X., Castells, P., de Vries, A., & Posse, C. (2012, September). Workshop on recommendation utility evaluation: beyond RMSE--RUE 2012. In Proceedings of the sixth ACM conference on Recommender systems (pp. 351-352). Academic Press.","DOI":"10.1145\/2365952.2366042"},{"key":"ijiit.2021040102-5","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45135-5_10"},{"key":"ijiit.2021040102-6","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639729"},{"key":"ijiit.2021040102-7","doi-asserted-by":"crossref","unstructured":"Beel, J., Langer, S., Genzmehr, M., Gipp, B., Breitinger, C., & N\u00fcrnberger, A. (2013, October). Research paper recommender system evaluation: a quantitative literature survey. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation (pp. 15-22). Academic Press.","DOI":"10.1145\/2532508.2532512"},{"key":"ijiit.2021040102-8","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043996"},{"key":"ijiit.2021040102-9","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800173"},{"key":"ijiit.2021040102-10","unstructured":"Champiri, D., Adeleh, A., & Salim, S. (2019). Meta-Analysis of Evaluation Methods and Metrics Used in Context-Aware Scholarly Recommender Systems. Knowledge and Information Systems, 1-32."},{"key":"ijiit.2021040102-11","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800180"},{"key":"ijiit.2021040102-12","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959190"},{"key":"ijiit.2021040102-13","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645737"},{"key":"ijiit.2021040102-14","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2015.2438867"},{"key":"ijiit.2021040102-15","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959170"},{"key":"ijiit.2021040102-16","first-page":"67","article-title":"Personalized, interactive tag recommendation for flickr.","author":"N.Garg","year":"2008","journal-title":"Proceedings of the 2nd ACM conference on Recommender systems"},{"key":"ijiit.2021040102-17","first-page":"2935","article-title":"A survey of accuracy evaluation metrics of recommendation tasks.","volume":"10","author":"A.Gunawardana","year":"2009","journal-title":"Journal of Machine Learning Research"},{"key":"ijiit.2021040102-18","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800179"},{"key":"ijiit.2021040102-19","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"ijiit.2021040102-20","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511763113"},{"key":"ijiit.2021040102-21","unstructured":"Kille, B., & Sahin, A. (2012). Modeling Difficulty in Recommender Systems. RUE@ RecSys."},{"key":"ijiit.2021040102-22","unstructured":"Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK: Keele University."},{"key":"ijiit.2021040102-23","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532513"},{"key":"ijiit.2021040102-24","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645738"},{"key":"ijiit.2021040102-25","doi-asserted-by":"publisher","DOI":"10.1145\/1297231.1297240"},{"key":"ijiit.2021040102-26","unstructured":"Meyer, F., Fessant, F., Cl\u00e9rot, F., & Gaussier, E. (2012). Toward a new protocol to evaluate recommender systems. arXiv preprint arXiv:1209.1983."},{"key":"ijiit.2021040102-27","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959157"},{"key":"ijiit.2021040102-28","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2007.07.047"},{"key":"ijiit.2021040102-29","article-title":"Trust-aware recommender systems.","author":"M.Paolo","year":"2004","journal-title":"Proceedings of the 2007 ACM conference on Recommender systems"},{"key":"ijiit.2021040102-30","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639720"},{"key":"ijiit.2021040102-31","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3"},{"key":"ijiit.2021040102-32","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645746"},{"key":"ijiit.2021040102-33","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2792841"},{"key":"ijiit.2021040102-34","unstructured":"Said, A., Tikk, D., Stumpf, K., Shi, Y., Larson, M. A., & Cremonesi, P. (2012, September). Recommender Systems Evaluation: A 3D Benchmark. RUE@ RecSys, 21-23."},{"key":"ijiit.2021040102-35","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800190"},{"key":"ijiit.2021040102-36","doi-asserted-by":"publisher","DOI":"10.1145\/1454008.1454015"},{"key":"ijiit.2021040102-37","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3_8"},{"key":"ijiit.2021040102-38","doi-asserted-by":"crossref","unstructured":"Smyth, B., & McClave, P. (2001). Similarity vs. diversity. Case-Based Reasoning Research and Development, 347-361.","DOI":"10.1007\/3-540-44593-5_25"},{"key":"ijiit.2021040102-39","doi-asserted-by":"crossref","unstructured":"Willemsen, M., Bollen, D., & Ekstrand, M. (2011). UCERSTI 2: second workshop on user-centric evaluation of recommender systems and their interfaces. Proceedings of the ACM Conference on Recommender Systems, RecSys 2011, 395\u2013396.","DOI":"10.1145\/2043932.2044020"},{"key":"ijiit.2021040102-40","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959174"},{"key":"ijiit.2021040102-41","doi-asserted-by":"publisher","DOI":"10.1145\/1454008.1454031"},{"key":"ijiit.2021040102-42","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639727"}],"container-title":["International Journal of Intelligent Information Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.igi-global.com\/viewtitle.aspx?TitleId=277071","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,6]],"date-time":"2022-05-06T11:53:30Z","timestamp":1651838010000},"score":1,"resource":{"primary":{"URL":"http:\/\/services.igi-global.com\/resolvedoi\/resolve.aspx?doi=10.4018\/ijiit.2021040102"}},"subtitle":["A Systemized Quantitative Survey"],"short-title":[],"issued":{"date-parts":[[2021,4]]},"references-count":43,"journal-issue":{"issue":"2"},"URL":"https:\/\/doi.org\/10.4018\/ijiit.2021040102","relation":{},"ISSN":["1548-3657","1548-3665"],"issn-type":[{"value":"1548-3657","type":"print"},{"value":"1548-3665","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4]]}}}