{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T02:36:23Z","timestamp":1784601383344,"version":"3.55.0"},"reference-count":231,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T00:00:00Z","timestamp":1671753600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Austrian Science Fund","award":["P33526"],"award-info":[{"award-number":["P33526"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,8,31]]},"abstract":"<jats:p>The comprehensive evaluation of the performance of a recommender system is a complex endeavor: many facets need to be considered in configuring an adequate and effective evaluation setting. Such facets include, for instance, defining the specific goals of the evaluation, choosing an evaluation method, underlying data, and suitable evaluation metrics. In this article, we consolidate and systematically organize this dispersed knowledge on recommender systems evaluation. We introduce the Framework for Evaluating Recommender systems (FEVR), which we derive from the discourse on recommender systems evaluation. In FEVR, we categorize the evaluation space of recommender systems evaluation. We postulate that the comprehensive evaluation of a recommender system frequently requires considering multiple facets and perspectives in the evaluation. The FEVR framework provides a structured foundation to adopt adequate evaluation configurations that encompass this required multi-facetedness and provides the basis to advance in the field. We outline and discuss the challenges of a comprehensive evaluation of recommender systems and provide an outlook on what we need to embrace and do to move forward as a research community.<\/jats:p>","DOI":"10.1145\/3556536","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T12:04:48Z","timestamp":1660737888000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":245,"title":["Evaluating Recommender Systems: Survey and Framework"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3195-8273","authenticated-orcid":false,"given":"Eva","family":"Zangerle","sequence":"first","affiliation":[{"name":"Universit\u00e4t Innsbruck, Innsbruck, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5724-1137","authenticated-orcid":false,"given":"Christine","family":"Bauer","sequence":"additional","affiliation":[{"name":"Utrecht University, Utrecht, Princetonplein, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,23]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-019-09256-1"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2559952"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_6"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/tkde.2005.99"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2151163.2151166"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331202"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.84.5.888"},{"issue":"3","key":"e_1_3_2_9_2","first-page":"200","article-title":"Software quality evaluation: User\u2019s view","volume":"5","author":"AL-Badareen Anas Bassam","year":"2011","unstructured":"Anas Bassam AL-Badareen, Mohd Hasan Selamat, Jamilah Din, Marzanah A. Jabar, and Sherzod Turaev. 2011. Software quality evaluation: User\u2019s view. Int. J. Appl. Math. Info. 5, 3 (2011), 200\u2013207.","journal-title":"Int. J. Appl. Math. Info."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.2753\/jec1086-4415170202"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800192"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.2147\/jmdh.s104807"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639744"},{"key":"e_1_3_2_14_2","volume-title":"The Long Tail: How Endless Choice is Creating Unlimited Demand","author":"Anderson Chris","year":"2007","unstructured":"Chris Anderson. 2007. The Long Tail: How Endless Choice is Creating Unlimited Demand. Random House."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463245"},{"key":"e_1_3_2_16_2","unstructured":"Juan Ar\u00e9valo Juan Ram\u00f3n Duque and Marco Creatura. 2018. A missing information loss function for implicit feedback datasets. Retrieved from https:\/\/arXiv:1805.00121."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45135-5_10"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/0022-2496(75)90001-2"},{"key":"e_1_3_2_19_2","volume-title":"Proceedings of the 1st Workshop on the Impact of Recommender Systems co-located with 13th ACM Conference on Recommender Systems (ImpactRS\u201919)","volume":"2462","author":"Bauer Christine","year":"2019","unstructured":"Christine Bauer and Eva Zangerle. 2019. Leveraging multi-method evaluation for multi-stakeholder settings. In Proceedings of the 1st Workshop on the Impact of Recommender Systems co-located with 13th ACM Conference on Recommender Systems (ImpactRS\u201919), Vol. 2462. CEUR-WS.org, Article 3. Retrieved from http:\/\/ceur-ws.org\/Vol-2462\/short3.pdf."},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/978-3-030-15719-7_27","volume-title":"Advances in Information Retrieval","author":"Beel Joeran","year":"2019","unstructured":"Joeran Beel, Andrew Collins, Oliver Kopp, Linus W. Dietz, and Petr Knoth. 2019. Online evaluations for everyone: Mr. DLib\u2019s living lab for scholarly recommendations. In Advances in Information Retrieval. Springer, 213\u2013219."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532511"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00799-015-0156-0"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24592-8_12"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532512"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-021-09302-x"},{"key":"e_1_3_2_26_2","volume-title":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201918)","author":"Berg Rianne van den","year":"2018","unstructured":"Rianne van den Berg, Thomas N. Kipf, and Max Welling. 2018. Graph convolutional matrix completion. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201918). Retrieved from https:\/\/www.kdd.org\/kdd2018\/files\/deep-learning-day\/DLDay18_paper_32.pdf."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330745"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3483382.3483397"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124930"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365964"},{"key":"e_1_3_2_32_2","first-page":"351","volume-title":"Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR\u201909)","author":"Bosteels Klaas","year":"2009","unstructured":"Klaas Bosteels, Elias Pampalk, and Etienne E. Kerre. 2009. Evaluating and analysing dynamic playlist generation heuristics using radio logs and fuzzy set theory. In Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR\u201909). International Society for Music Information Retrieval, 351\u2013356."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.5555\/2074094.2074100"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1021240730564"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864753"},{"key":"e_1_3_2_36_2","volume-title":"Proceedings of the 1st Workshop on Surprise, Opposition, and Obstruction in Adaptive and Personalized Systems (SOAP\u201916)","volume":"1618","author":"Burke Robin","year":"2016","unstructured":"Robin Burke, Himan Abdollahpouri, Bamshad Mobasher, and Trinadh Gupta. 2016. Towards multi-stakeholder utility evaluation of recommender systems. In Proceedings of the 1st Workshop on Surprise, Opposition, and Obstruction in Adaptive and Personalized Systems (SOAP\u201916), Vol. 1618. CEUR-WS.org, Article 2. Retrieved from http:\/\/ceur-ws.org\/Vol-1618\/SOAP_paper2.pdf."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09371-3"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0040950"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_16"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213586.3226202"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/1454008.1454038"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13287-2"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/0197-2456(81)90056-8"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1177\/0013916509350849"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2016.06.017"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.23940\/ijpe.17.08.p7.12461256"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/hicss.2009.117"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40881-015-0005-3"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939746"},{"key":"e_1_3_2_50_2","volume-title":"Factors Determining the Performance of Indexing Systems, (Volume 1: Design)","author":"Cleverdon Cyril W.","year":"1966","unstructured":"Cyril W. Cleverdon, Jack Mills, and E. Michael Keen. 1966. Factors Determining the Performance of Indexing Systems, (Volume 1: Design). Technical Report, College of Aeronautics. Retrieved from http:\/\/hdl.handle.net\/1826\/861."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642713"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-23765-2_11"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/axmedis.2008.13"},{"key":"e_1_3_2_54_2","volume-title":"Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (2nd ed.)","author":"Creswell John W.","year":"2003","unstructured":"John W. Creswell. 2003. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (2nd ed.). SAGE."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557139"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864770"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-7637-6_4"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-018-1324-5"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.pmcj.2016.09.020"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13740-016-0060-9"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411962"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.2307\/1251362"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2022.913608"},{"key":"e_1_3_2_64_2","first-page":"67","volume-title":"Proceedings of the Workshop on Human Decision Making in Recommender Systems (Decisions@RecSys\u201911) and User-Centric Evaluation of Recommender Systems and Their Interfaces-2 (UCERSTI\u201911)","volume":"811","author":"Dooms Simon","year":"2011","unstructured":"Simon Dooms, Toon De Pessemier, and Luc Martens. 2011. A user-centric evaluation of recommender algorithms for an event recommendation system. In Proceedings of the Workshop on Human Decision Making in Recommender Systems (Decisions@RecSys\u201911) and User-Centric Evaluation of Recommender Systems and Their Interfaces-2 (UCERSTI\u201911), Vol. 811. CEUR-WS.org, 67\u201373. Retrieved from https:\/\/hdl.handle.net\/1854\/LU-2040145."},{"key":"e_1_3_2_65_2","volume-title":"Proceedings of the Workshop on Crowdsourcing and Human Computation for Recommender Systems (CrowdRec\u201913)","author":"Dooms Simon","year":"2013","unstructured":"Simon Dooms, Toon De Pessemier, and Luc Martens. 2013. MovieTweetings: A movie rating dataset collected from Twitter. In Proceedings of the Workshop on Crowdsourcing and Human Computation for Recommender Systems (CrowdRec\u201913). http:\/\/hdl.handle.net\/1854\/LU-4284240."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_67_2","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*\u201918)","author":"Dwork Cynthia","year":"2018","unstructured":"Cynthia Dwork and Christina Ilvento. 2018. Group fairness under composition. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*\u201918). ACM."},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1561\/1100000009"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412778"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3346964"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645737"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09284-2"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043958"},{"key":"e_1_3_2_74_2","first-page":"639","volume-title":"Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference (FLAIRS\u201917)","author":"Ekstrand Michael D.","year":"2017","unstructured":"Michael D. Ekstrand and Vaibhav Mahant. 2017. Sturgeon and the cool kids: Problems with random decoys for top-N recommender evaluation. In Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference (FLAIRS\u201917). AAAI, 639\u2013644."},{"key":"e_1_3_2_75_2","first-page":"172","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency","volume":"81","author":"Ekstrand Michael D.","year":"2018","unstructured":"Michael D. Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D. Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All the cool kids, how do they fit in? Popularity and demographic biases in recommender evaluation and effectiveness. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Vol. 81. PMLR, 172\u2013186. Retrieved from https:\/\/proceedings.mlr.press\/v81\/ekstrand18b.html."},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/2039320.2039330"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-75067-5_3"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3434185"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3406522.3446033"},{"key":"e_1_3_2_81_2","volume-title":"An Introduction to Qualitative Research","author":"Flick Uwe","year":"2014","unstructured":"Uwe Flick. 2014. An Introduction to Qualitative Research. SAGE."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-4778-7_8"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2010.01.002"},{"key":"e_1_3_2_84_2","unstructured":"Simon Funk. 2006. Try this at home. Retrieved from http:\/\/sifter.org\/simon\/journal\/2006."},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043989"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3241622"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645745"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864761"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159687"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/2843948"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.aaf5027"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295x.93.2.216"},{"key":"e_1_3_2_93_2","first-page":"2935","article-title":"A survey of accuracy evaluation metrics of recommendation tasks","volume":"10","author":"Gunawardana Asela","year":"2009","unstructured":"Asela Gunawardana and Guy Shani. 2009. A survey of accuracy evaluation metrics of recommendation tasks. J. Mach. Learn. Res. 10(Dec.2009), 2935\u20132962.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_15"},{"key":"e_1_3_2_95_2","volume-title":"Proceedings of the 23rd Conference on User Modeling, Adaptation and Personalization (UMAP\u201915)","volume":"1388","author":"Guo Guibing","year":"2015","unstructured":"Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. 2015. LibRec: A Java library for recommender systems. In Proceedings of the 23rd Conference on User Modeling, Adaptation and Personalization (UMAP\u201915), Vol. 1388. CEUR-WS.org. Retrieved from http:\/\/ceur-ws.org\/Vol-1388\/demo_paper1.pdf."},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.5555\/3172077.3172127"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2021.08.025"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/2827872"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401063"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/358916.358995"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963772"},{"key":"e_1_3_2_103_2","unstructured":"Bal\u00e1zs Hidasi Alexandros Karatzoglou Linas Baltrunas and Domonkos Tikk. 2015. Session-based recommendations with recurrent neural networks. Retrieved from arXiv:1511.06939."},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.3789\/niso-rp-31-2021"},{"key":"e_1_3_2_105_2","volume-title":"Proceedings of the Workshop on Novelty and Diversity in Recommender Systems (DiveRS\u201911), at the 5th ACM International Conference on Recommender Systems (RecSys\u201911)","author":"Hu Rong","year":"2011","unstructured":"Rong Hu and Pearl Pu. 2011. Helping users perceive recommendation diversity. In Proceedings of the Workshop on Novelty and Diversity in Recommender Systems (DiveRS\u201911), at the 5th ACM International Conference on Recommender Systems (RecSys\u201911). CEUR-WS.org, Article 6, 43\u201350. Retrieved from http:\/\/ceur-ws.org\/Vol-816\/paper6.pdf."},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1109\/icdm.2008.22"},{"key":"e_1_3_2_107_2","article-title":"Surprise, A Python Library for Recommender Systems","author":"Hug Nicolas","year":"2017","unstructured":"Nicolas Hug. 2017. Surprise, A Python Library for Recommender Systems. Retrieved from http:\/\/surpriselib.com.","journal-title":"Retrieved from http:\/\/surpriselib.com"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1145\/1944339.1944341"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287600"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959186"},{"key":"e_1_3_2_111_2","volume-title":"Proceedings of the 1st International Workshop on Value-Aware and Multistakeholder Recommendation (VAMS\u201917)","author":"Jannach Dietmar","year":"2017","unstructured":"Dietmar Jannach and Gediminas Adomavicius. 2017. Price and profit awareness in recommender systems. In Proceedings of the 1st International Workshop on Value-Aware and Multistakeholder Recommendation (VAMS\u201917). Retrieved from https:\/\/arXiv:1707.08029."},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v41i4.5312"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1145\/3370082"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453154"},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09274-4"},{"key":"e_1_3_2_116_2","volume-title":"Proceedings of the 1st Workshop on the Impact of Recommender Systems (ImpactRS\u201919)","volume":"2462","author":"Jannach Dietmar","year":"2019","unstructured":"Dietmar Jannach, Oren Sar Shalom, and Joseph A. Konstan. 2019. Towards more impactful recommender systems research. In Proceedings of the 1st Workshop on the Impact of Recommender Systems (ImpactRS\u201919), Vol. 2462. CEUR-WS.org. Retrieved from http:\/\/ceur-ws.org\/Vol-2462\/short6.pdf."},{"key":"e_1_3_2_117_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_14"},{"key":"e_1_3_2_118_2","doi-asserted-by":"publisher","DOI":"10.1017\/cbo9780511763113"},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-32273-0_7"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/1869446.1869453"},{"key":"e_1_3_2_122_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chbr.2020.100052"},{"key":"e_1_3_2_123_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.12014"},{"key":"e_1_3_2_124_2","doi-asserted-by":"publisher","DOI":"10.1145\/3334480.3382945"},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.1145\/3079628.3079668"},{"key":"e_1_3_2_126_2","doi-asserted-by":"publisher","DOI":"10.1145\/2926720"},{"key":"e_1_3_2_127_2","doi-asserted-by":"publisher","DOI":"10.1109\/bigdata.2014.7004317"},{"key":"e_1_3_2_128_2","doi-asserted-by":"publisher","DOI":"10.17169\/fqs-6.2.466"},{"key":"e_1_3_2_129_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-10-0557-2_112"},{"key":"e_1_3_2_130_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-12630-7_47"},{"key":"e_1_3_2_131_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043960"},{"key":"e_1_3_2_132_2","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639793"},{"key":"e_1_3_2_133_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-7637-6_9"},{"key":"e_1_3_2_134_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9118-4"},{"key":"e_1_3_2_135_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043993"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_3_2_137_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"key":"e_1_3_2_138_2","volume-title":"Proceedings of the 9th International Workshop on News Recommendation and Analytics (INRA\u201921)","volume":"3143","author":"Konstan Joseph A.","year":"2021","unstructured":"Joseph A. Konstan, Robin Burke, and Edward C. Malthouse. 2021. Towards an experimental news user community as infrastructure for recommendation research. In Proceedings of the 9th International Workshop on News Recommendation and Analytics (INRA\u201921), Vol. 3143. CEUR-WS.org, Article 4, 43\u201346. Retrieved from http:\/\/ceur-ws.org\/Vol-3143\/paper4.pdf."},{"key":"e_1_3_2_139_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9112-x"},{"key":"e_1_3_2_140_2","doi-asserted-by":"publisher","DOI":"10.1109\/mc.2009.263"},{"key":"e_1_3_2_141_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_3"},{"key":"e_1_3_2_142_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jenvp.2014.09.003"},{"key":"e_1_3_2_143_2","doi-asserted-by":"publisher","DOI":"10.1140\/epjds\/s13688-021-00268-9"},{"key":"e_1_3_2_144_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2017.02.009"},{"key":"e_1_3_2_145_2","volume-title":"Hawthorne Revisited: Management and the Worker, Its Critics, and Developments in Human Relations in Industry","author":"Landsberger Henry A.","year":"1958","unstructured":"Henry A. Landsberger. 1958. Hawthorne Revisited: Management and the Worker, Its Critics, and Developments in Human Relations in Industry. Cornell University Press."},{"key":"e_1_3_2_146_2","volume-title":"Proceedings of the International Conference on Information Systems (ICIS\u201914)","author":"Lee Dokyun","year":"2014","unstructured":"Dokyun Lee and Kartik Hosanagar. 2014. Impact of recommender systems on sales volume and diversity. In Proceedings of the International Conference on Information Systems (ICIS\u201914). AIS. Retrieved from https:\/\/aisel.aisnet.org\/icis2014\/proceedings\/EBusiness\/40\/."},{"key":"e_1_3_2_147_2","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772758"},{"key":"e_1_3_2_148_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220023"},{"key":"e_1_3_2_149_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_2_150_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3474253"},{"key":"e_1_3_2_151_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3311744"},{"key":"e_1_3_2_152_2","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871643"},{"key":"e_1_3_2_153_2","doi-asserted-by":"publisher","DOI":"10.1109\/icbda.2017.8078860"},{"key":"e_1_3_2_154_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347041"},{"key":"e_1_3_2_155_2","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458205"},{"key":"e_1_3_2_156_2","doi-asserted-by":"publisher","DOI":"10.1108\/intr-06-2018-0274"},{"key":"e_1_3_2_157_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240354"},{"key":"e_1_3_2_158_2","doi-asserted-by":"publisher","DOI":"10.1145\/1125451.1125660"},{"key":"e_1_3_2_159_2","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054197"},{"key":"e_1_3_2_160_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78197-4_5"},{"key":"e_1_3_2_161_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_7"},{"key":"e_1_3_2_162_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_2"},{"key":"e_1_3_2_163_2","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365983"},{"key":"e_1_3_2_164_2","doi-asserted-by":"publisher","DOI":"10.1145\/1142405.1142439"},{"key":"e_1_3_2_165_2","doi-asserted-by":"publisher","DOI":"10.1145\/1454008.1454012"},{"key":"e_1_3_2_166_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-22362-4_22"},{"key":"e_1_3_2_167_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33326-2_7"},{"key":"e_1_3_2_168_2","doi-asserted-by":"publisher","DOI":"10.1145\/2955101"},{"key":"e_1_3_2_169_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-72079-9_10"},{"key":"e_1_3_2_170_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203726389"},{"key":"e_1_3_2_171_2","doi-asserted-by":"publisher","DOI":"10.2753\/mis0742-1222240302"},{"key":"e_1_3_2_172_2","doi-asserted-by":"publisher","DOI":"10.4018\/978-1-4666-2491-7.ch014"},{"key":"e_1_3_2_173_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043962"},{"key":"e_1_3_2_174_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9115-7"},{"key":"e_1_3_2_175_2","doi-asserted-by":"publisher","DOI":"10.1145\/3190616"},{"key":"e_1_3_2_176_2","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109896"},{"key":"e_1_3_2_177_2","first-page":"452","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909)","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909). AUAI Press, 452\u2013461."},{"key":"e_1_3_2_178_2","unstructured":"Steffen Rendle Li Zhang and Yehuda Koren. 2019. On the difficulty of evaluating baselines: A study on recommender systems. Retrieved from arXiv:1905.01395."},{"key":"e_1_3_2_179_2","doi-asserted-by":"publisher","DOI":"10.1145\/245108.245121"},{"key":"e_1_3_2_180_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2018.04.007"},{"key":"e_1_3_2_181_2","doi-asserted-by":"publisher","DOI":"10.1109\/mis.2007.43"},{"key":"e_1_3_2_182_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-2197-4_1"},{"key":"e_1_3_2_183_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10660-018-9321-z"},{"key":"e_1_3_2_184_2","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959176"},{"key":"e_1_3_2_185_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230713"},{"key":"e_1_3_2_186_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645746"},{"key":"e_1_3_2_187_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645712"},{"key":"e_1_3_2_188_2","volume-title":"Proceedings of the 3rd RecSys Workshop on Context-Aware Recommender Systems (CARS\u201911)","volume":"791","author":"Said Alan","year":"2011","unstructured":"Alan Said, Ernesto W. De Luca, and Sahin Albayrak. 2011. Inferring contextual user profiles-improving recommender performance. In Proceedings of the 3rd RecSys Workshop on Context-Aware Recommender Systems (CARS\u201911), Vol. 791. Ceur-WS.org. Retrieved from http:\/\/ceur-ws.org\/Vol-791\/paper7.pdf."},{"key":"e_1_3_2_189_2","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441933"},{"key":"e_1_3_2_190_2","first-page":"21","volume-title":"Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE\u201912)","volume":"910","author":"Said Alan","year":"2012","unstructured":"Alan Said, Domonkos Tikk, Klara Stumpf, Yue Shi, Martha Larson, and Paolo Cremonesi. 2012. Recommender systems evaluation: A 3D benchmark. In Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE\u201912), Vol. 910. CEUR-WS.org, 21\u201323. Retrieved from http:\/\/ceur-ws.org\/Vol-910\/."},{"key":"e_1_3_2_191_2","doi-asserted-by":"publisher","DOI":"10.1023\/a:1009752403260"},{"key":"e_1_3_2_192_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317595"},{"key":"e_1_3_2_193_2","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372071"},{"key":"e_1_3_2_194_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-72079-9_9"},{"key":"e_1_3_2_195_2","doi-asserted-by":"publisher","DOI":"10.1145\/3498366.3505791"},{"key":"e_1_3_2_196_2","volume-title":"Proceedings of the Workshop on Human Decision Making in Recommender Systems (Decisions@RecSys\u201911) and User-Centric Evaluation of Recommender Systems and Their Interfaces-2 (UCERSTI\u201911)","volume":"811","author":"Schr\u00f6der Gunnar","year":"2011","unstructured":"Gunnar Schr\u00f6der, Maik Thiele, and Wolfgang Lehner. 2011. Setting goals and choosing metrics for recommender system evaluations. In Proceedings of the Workshop on Human Decision Making in Recommender Systems (Decisions@RecSys\u201911) and User-Centric Evaluation of Recommender Systems and Their Interfaces-2 (UCERSTI\u201911), Vol. 811. CEUR-WS.org, Article 12, 78\u201385. Retrieved from http:\/\/ceur-ws.org\/Vol-811\/paper12.pdf."},{"key":"e_1_3_2_197_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_3_2_198_2","doi-asserted-by":"publisher","DOI":"10.1145\/223904.223931"},{"key":"e_1_3_2_199_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_2_200_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12124-015-9339-x"},{"key":"e_1_3_2_201_2","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-44593-5_25"},{"key":"e_1_3_2_202_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcps.2015.12.004"},{"key":"e_1_3_2_203_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3470802"},{"key":"e_1_3_2_204_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043957"},{"key":"e_1_3_2_205_2","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507160"},{"key":"e_1_3_2_206_2","doi-asserted-by":"publisher","DOI":"10.4135\/9781446261606"},{"key":"e_1_3_2_207_2","doi-asserted-by":"publisher","DOI":"10.1155\/2009\/421425"},{"issue":"52","key":"e_1_3_2_208_2","first-page":"1731","article-title":"Batch learning from logged bandit feedback through counterfactual risk minimization","volume":"16","author":"Swaminathan Adith","year":"2015","unstructured":"Adith Swaminathan and Thorsten Joachims. 2015. Batch learning from logged bandit feedback through counterfactual risk minimization. J. Mach. Learn. Res. 16, 52 (2015), 1731\u20131755. Retrieved from http:\/\/jmlr.org\/papers\/v16\/swaminathan15a.html.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_209_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.141.3577.245"},{"key":"e_1_3_2_210_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3478848"},{"key":"e_1_3_2_211_2","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2882971"},{"key":"e_1_3_2_212_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3_15"},{"key":"e_1_3_2_213_2","doi-asserted-by":"publisher","DOI":"10.5555\/2999792.2999907"},{"key":"e_1_3_2_214_2","doi-asserted-by":"publisher","DOI":"10.5555\/539927"},{"key":"e_1_3_2_215_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043955"},{"key":"e_1_3_2_216_2","doi-asserted-by":"crossref","unstructured":"Ellen M. Voorhees. 2000. The TREC-8 Question Answering Track Report 77\u201382. Retrieved from https:\/\/tsapps.nist.gov\/publication\/get_pdf.cfm?pub_id=151495.","DOI":"10.6028\/NIST.SP.500-249.qa-overview"},{"key":"e_1_3_2_217_2","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45691-0_34"},{"key":"e_1_3_2_218_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612463078"},{"key":"e_1_3_2_219_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3133025"},{"key":"e_1_3_2_220_2","doi-asserted-by":"publisher","DOI":"10.2307\/25148784"},{"key":"e_1_3_2_221_2","doi-asserted-by":"publisher","DOI":"10.5555\/3294996.3295052"},{"key":"e_1_3_2_222_2","volume-title":"Case Study Research: Design and Methods (5th revised ed.)","author":"Yin Robert K.","year":"1989","unstructured":"Robert K. Yin. 1989. Case Study Research: Design and Methods (5th revised ed.). SAGE."},{"key":"e_1_3_2_223_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219890"},{"key":"e_1_3_2_224_2","doi-asserted-by":"publisher","DOI":"10.1109\/icdm.2012.168"},{"key":"e_1_3_2_225_2","doi-asserted-by":"publisher","DOI":"10.1145\/2957792.2957804"},{"key":"e_1_3_2_226_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2019.06.006"},{"key":"e_1_3_2_227_2","doi-asserted-by":"publisher","DOI":"10.1145\/3285029"},{"key":"e_1_3_2_228_2","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998211"},{"key":"e_1_3_2_229_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3185994"},{"key":"e_1_3_2_230_2","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018665"},{"key":"e_1_3_2_231_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1000488107"},{"key":"e_1_3_2_232_2","doi-asserted-by":"publisher","DOI":"10.1145\/1060745.1060754"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3556536","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3556536","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:32Z","timestamp":1750186832000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3556536"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,23]]},"references-count":231,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2023,8,31]]}},"alternative-id":["10.1145\/3556536"],"URL":"https:\/\/doi.org\/10.1145\/3556536","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,23]]},"assertion":[{"value":"2021-03-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-18","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}