{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T07:58:55Z","timestamp":1792742335862,"version":"4.5.0"},"reference-count":107,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2019,12,10]],"date-time":"2019-12-10T00:00:00Z","timestamp":1575936000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Manage. Inf. Syst."],"published-print":{"date-parts":[[2019,12,31]]},"abstract":"<jats:p>\n                    Recommender Systems are nowadays successfully used by all major web sites\u2014from e-commerce to social media\u2014to filter content and make suggestions in a personalized way. Academic research largely focuses on the value of recommenders for consumers, e.g., in terms of reduced information overload. To what extent and in which ways recommender systems create\n                    <jats:italic>business value<\/jats:italic>\n                    is, however, much less clear, and the literature on the topic is scattered. In this research commentary, we review existing publications on field tests of recommender systems and report which business-related performance measures were used in such real-world deployments. We summarize common challenges of measuring the business value in practice and critically discuss the value of algorithmic improvements and offline experiments as commonly done in academic environments. Overall, our review indicates that various open questions remain both regarding the realistic quantification of the business effects of recommenders and the performance assessment of recommendation algorithms in\u00a0academia.\n                  <\/jats:p>","DOI":"10.1145\/3370082","type":"journal-article","created":{"date-parts":[[2019,12,11]],"date-time":"2019-12-11T08:27:18Z","timestamp":1576052838000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":224,"title":["Measuring the Business Value of Recommender Systems"],"prefix":"10.1145","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4698-8507","authenticated-orcid":false,"given":"Dietmar","family":"Jannach","sequence":"first","affiliation":[{"name":"University of Klagenfurt, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Jugovac","sequence":"additional","affiliation":[{"name":"TU Dortmund, Dortmund, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,12,10]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-5414.2005.00076.x"},{"key":"e_1_2_1_2_1","unstructured":"Himan Abdollahpouri Gediminas Adomavicius Robin Burke Ido Guy Dietmar Jannach Toshihiro Kamishima Jan Krasnodebski and Luiz Pizzato. 2019. Beyond personalization: Research directions in multistakeholder recommendation. Retrieved from https:\/\/arxiv.org\/abs\/1905.01986.  Himan Abdollahpouri Gediminas Adomavicius Robin Burke Ido Guy Dietmar Jannach Toshihiro Kamishima Jan Krasnodebski and Luiz Pizzato. 2019. Beyond personalization: Research directions in multistakeholder recommendation. Retrieved from https:\/\/arxiv.org\/abs\/1905.01986."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079628.3079657"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3241729"},{"key":"e_1_2_1_5_1","volume-title":"Netflix Recommendations: Beyond the 5 stars.","author":"Amatriain Xavier","year":"2012","unstructured":"Xavier Amatriain and Justin Basilico . 2012 . Netflix Recommendations: Beyond the 5 stars. Retrieved from https:\/\/medium.com\/netflix-techblog\/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429. Xavier Amatriain and Justin Basilico. 2012. Netflix Recommendations: Beyond the 5 stars. Retrieved from https:\/\/medium.com\/netflix-techblog\/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646031"},{"key":"e_1_2_1_7_1","volume-title":"Recommender Systems Handbook","author":"Bambini Riccardo","unstructured":"Riccardo Bambini , Paolo Cremonesi , and Roberto Turrin . 2011. Recommender Systems Handbook . Springer , 299--331. Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.). Riccardo Bambini, Paolo Cremonesi, and Roberto Turrin. 2011. Recommender Systems Handbook. Springer, 299--331. Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.)."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645729"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2532508.2532511"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24592-8_12"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the iConference","author":"Beel Joeran","year":"2017","unstructured":"Joeran Beel , Stefan Langer , and Bela Gipp . 2017 . TF-IDuF: A novel term-weighting scheme for user modeling based on users\u2019 personal document collections . In Proceedings of the iConference 2017. 452--459. Joeran Beel, Stefan Langer, and Bela Gipp. 2017. TF-IDuF: A novel term-weighting scheme for user modeling based on users\u2019 personal document collections. In Proceedings of the iConference 2017. 452--459."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1509\/jmkr.45.1.77"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.14763\/2016.1.401"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959166"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL\u201919)","author":"Carraro Diego","year":"2019","unstructured":"Diego Carraro and Derek Bridge . 2019 . Debiased offline evaluation of recommender systems: A weighted-sampling approach (extended abstract) . In Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL\u201919) . Diego Carraro and Derek Bridge. 2019. Debiased offline evaluation of recommender systems: A weighted-sampling approach (extended abstract). In Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL\u201919)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 42nd Hawaii International Conference on System Sciences (HICSS\u201909)","author":"Chen Pei-Yu","unstructured":"Pei-Yu Chen , Yen-Chun Chou , and R. J. Kauffman . 2009. Community-based recommender systems: Analyzing business models from a systems operator\u2019s perspective . In Proceedings of the 42nd Hawaii International Conference on System Sciences (HICSS\u201909) . 1--10. Pei-Yu Chen, Yen-Chun Chou, and R. J. Kauffman. 2009. Community-based recommender systems: Analyzing business models from a systems operator\u2019s perspective. In Proceedings of the 42nd Hawaii International Conference on System Sciences (HICSS\u201909). 1--10."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201911)","author":"Chen Ye","unstructured":"Ye Chen and John F. Canny . 2011. Recommending ephemeral items at web scale . In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201911) . 1013--1022. Ye Chen and John F. Canny. 2011. Recommending ephemeral items at web scale. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201911). 1013--1022."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2209310.2209314"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40477-1_21"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347058"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242610"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864770"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433413"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the ACM Conference on Recommender Systems (RecSys\u201908)","author":"Dias M. Benjamin","unstructured":"M. Benjamin Dias , Dominique Locher , Ming Li , Wael El-Deredy , and Paulo J. G. Lisboa . 2008. The value of personalised recommender systems to e-business: A case study . In Proceedings of the ACM Conference on Recommender Systems (RecSys\u201908) . 291--294. M. Benjamin Dias, Dominique Locher, Ming Li, Wael El-Deredy, and Paulo J. G. Lisboa. 2008. The value of personalised recommender systems to e-business: A case study. In Proceedings of the ACM Conference on Recommender Systems (RecSys\u201908). 291--294."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.2307\/248851"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13735-012-0025-1"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 8th ACM Conference on Recommender Systems (RecSys\u201914)","author":"Ekstrand Michael D.","unstructured":"Michael D. Ekstrand , F. Maxwell Harper , Martijn C. Willemsen , and Joseph A. Konstan . 2014. User perception of differences in recommender algorithms . In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys\u201914) . 161--168. Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User perception of differences in recommender algorithms. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys\u201914). 161--168."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v32i3.2365"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645745"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159687"},{"key":"e_1_2_1_32_1","article-title":"The Netflix recommender system: Algorithms, business value, and innovation","volume":"6","author":"Gomez-Uribe Carlos A.","year":"2015","unstructured":"Carlos A. Gomez-Uribe and Neil Hunt . 2015 . The Netflix recommender system: Algorithms, business value, and innovation . Trans. Manage. Info. Syst. 6 , 4 (2015), 13:1--13:19. Carlos A. Gomez-Uribe and Neil Hunt. 2015. The Netflix recommender system: Algorithms, business value, and innovation. Trans. Manage. Info. Syst. 6, 4 (2015), 13:1--13:19.","journal-title":"Trans. Manage. Info. Syst."},{"key":"e_1_2_1_33_1","volume-title":"The Most Powerful Way to Turn Clicks into Customers","author":"Gumm Bryan","unstructured":"Bryan Gumm . 2013. A\/ B Testing : The Most Powerful Way to Turn Clicks into Customers . Wiley , 180--193. Dan Siroker and Pete Koomen (Eds.). Bryan Gumm. 2013. A\/B Testing: The Most Powerful Way to Turn Clicks into Customers. Wiley, 180--193. Dan Siroker and Pete Koomen (Eds.)."},{"key":"e_1_2_1_34_1","first-page":"1","article-title":"The end-user computing satisfaction (EUCS) on computerized accounting system (CAS): How they perceived","volume":"13","author":"Ilias Azleen","year":"2008","unstructured":"Azleen Ilias , Norazah Bte Mohd Suki , Mohd Rushdan Yasoa , and Mohd Zulkeflee Abd Razak . 2008 . The end-user computing satisfaction (EUCS) on computerized accounting system (CAS): How they perceived ? J. Internet Bank. Comm. 13 , 1 (2008), 1 -- 18 . Azleen Ilias, Norazah Bte Mohd Suki, Mohd Rushdan Yasoa, and Mohd Zulkeflee Abd Razak. 2008. The end-user computing satisfaction (EUCS) on computerized accounting system (CAS): How they perceived? J. Internet Bank. Comm. 13, 1 (2008), 1--18.","journal-title":"J. Internet Bank. Comm."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3290958"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2932206.2932207"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959186"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the Workshop on Value-Aware and Multi-Stakeholder Recommendation (VAMS\u201917)","author":"Jannach Dietmar","year":"2017","unstructured":"Dietmar Jannach and Gediminas Adomavicius . 2017 . Price and profit awareness in recommender systems . In Proceedings of the Workshop on Value-Aware and Multi-Stakeholder Recommendation (VAMS\u201917) . Dietmar Jannach and Gediminas Adomavicius. 2017. Price and profit awareness in recommender systems. In Proceedings of the Workshop on Value-Aware and Multi-Stakeholder Recommendation (VAMS\u201917)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639749"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3019612.3019758"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800176"},{"key":"e_1_2_1_42_1","first-page":"29","article-title":"Item familiarity as a possible confounding factor in user-centric recommender systems evaluation. i-com J","volume":"14","author":"Jannach Dietmar","year":"2015","unstructured":"Dietmar Jannach , Lukas Lerche , and Michael Jugovac . 2015 . Item familiarity as a possible confounding factor in user-centric recommender systems evaluation. i-com J . Interact. Media 14 , 1 (2015), 29 -- 39 . Dietmar Jannach, Lukas Lerche, and Michael Jugovac. 2015. Item familiarity as a possible confounding factor in user-centric recommender systems evaluation. i-com J. Interact. Media 14, 1 (2015), 29--39.","journal-title":"Interact. Media"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-015-9165-3"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-017-9194-1"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2891406"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the ACM RecSys Workshop on the Impact of Recommender Systems (ImpactRS\u201919)","author":"Jannach Dietmar","unstructured":"Dietmar Jannach , Oren Sar Shalom , and Joseph A. Konstan . 2019. Towards more impactful recommender systems research . In Proceedings of the ACM RecSys Workshop on the Impact of Recommender Systems (ImpactRS\u201919) . Dietmar Jannach, Oren Sar Shalom, and Joseph A. Konstan. 2019. Towards more impactful recommender systems research. In Proceedings of the ACM RecSys Workshop on the Impact of Recommender Systems (ImpactRS\u201919)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314288"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2017.03.055"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079628.3079668"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the Workshop on Intelligent Music Interfaces for Listening and Creation (MILC\u201918)","author":"Kamehkhosh Iman","year":"2018","unstructured":"Iman Kamehkhosh , Dietmar Jannach , and Geoffray Bonnin . 2018 . How automated recommendations affect the playlist creation behavior of users . In Proceedings of the Workshop on Intelligent Music Interfaces for Listening and Creation (MILC\u201918) . Iman Kamehkhosh, Dietmar Jannach, and Geoffray Bonnin. 2018. How automated recommendations affect the playlist creation behavior of users. In Proceedings of the Workshop on Intelligent Music Interfaces for Listening and Creation (MILC\u201918)."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2926720"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2009.09.002"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the IEEE International Conference on Big Data","author":"Katukuri J.","year":"2014","unstructured":"J. Katukuri , T. K\u00f6nik , R. Mukherjee , and S. Kolay . 2014. Recommending similar items in large-scale online marketplaces . In Proceedings of the IEEE International Conference on Big Data 2014 . 868--876. J. Katukuri, T. K\u00f6nik, R. Mukherjee, and S. Kolay. 2014. Recommending similar items in large-scale online marketplaces. In Proceedings of the IEEE International Conference on Big Data 2014. 868--876."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the IEEE International Conference on Big Data (BigData\u201915)","author":"Katukuri J.","unstructured":"J. Katukuri , T. Konik , R. Mukherjee , and S. Kolay . 2015. Post-purchase recommendations in large-scale online marketplaces . In Proceedings of the IEEE International Conference on Big Data (BigData\u201915) . 1299--1305. J. Katukuri, T. Konik, R. Mukherjee, and S. Kolay. 2015. Post-purchase recommendations in large-scale online marketplaces. In Proceedings of the IEEE International Conference on Big Data (BigData\u201915). 1299--1305."},{"key":"e_1_2_1_56_1","volume-title":"Machine Learning and Knowledge Discovery in Databases","author":"Kirshenbaum Evan","unstructured":"Evan Kirshenbaum , George Forman , and Michael Dugan . 2012. A live comparison of methods for personalized article recommendation at Forbes.com . In Machine Learning and Knowledge Discovery in Databases . Springer , 51--66. Evan Kirshenbaum, George Forman, and Michael Dugan. 2012. A live comparison of methods for personalized article recommendation at Forbes.com. In Machine Learning and Knowledge Discovery in Databases. Springer, 51--66."},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915)","author":"Kiseleva Julia","year":"2015","unstructured":"Julia Kiseleva , Melanie J. I. M\u00fcller , Lucas Bernardi , Chad Davis , Ivan Kovacek , Mats Stafseng Einarsen , Jaap Kamps , Alexander Tuzhilin , and Djoerd Hiemstra . 2015 . Where to go on your next trip? Optimizing travel destinations based on user preferences . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915) . Julia Kiseleva, Melanie J. I. M\u00fcller, Lucas Bernardi, Chad Davis, Ivan Kovacek, Mats Stafseng Einarsen, Jaap Kamps, Alexander Tuzhilin, and Djoerd Hiemstra. 2015. Where to go on your next trip? Optimizing travel destinations based on user preferences. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915)."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jretai.2018.10.004"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339653"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9112-x"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009835726774"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the International Conference on Information Systems (ICIS\u201914)","author":"Lee Dokyun","year":"2014","unstructured":"Dokyun Lee and Kartik Hosanagar . 2014 . Impact of recommender systems on sales volume and diversity . In Proceedings of the International Conference on Information Systems (ICIS\u201914) . Dokyun Lee and Kartik Hosanagar. 2014. Impact of recommender systems on sales volume and diversity. In Proceedings of the International Conference on Information Systems (ICIS\u201914)."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2018.0800"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2930238.2930244"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935878"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308774.3308781"},{"key":"e_1_2_1_67_1","unstructured":"Z. C. Lipton and J. Steinhardt. 2018. Troubling trends in machine learning scholarship. Retrieved from https:\/\/arxiv.org\/abs\/1807.03341.  Z. C. Lipton and J. Steinhardt. 2018. Troubling trends in machine learning scholarship. Retrieved from https:\/\/arxiv.org\/abs\/1807.03341."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1719970.1719976"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-018-9209-6"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347046"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2792838.2800184"},{"key":"e_1_2_1_72_1","volume-title":"Zahedi.","author":"McKinney Vicki","year":"2002","unstructured":"Vicki McKinney , Yoon Kanghyun , and Fatemeh \u201c Mariam \u201d Zahedi. 2002 . The measurement of web-customer satisfaction: An expectation and disconfirmation approach. Info. Syst. Res . 13, 3 (2002), 296--315. Vicki McKinney, Yoon Kanghyun, and Fatemeh \u201cMariam\u201d Zahedi. 2002. The measurement of web-customer satisfaction: An expectation and disconfirmation approach. Info. Syst. Res. 13, 3 (2002), 296--315."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/587078.587096"},{"key":"e_1_2_1_74_1","volume-title":"Konstan","author":"McNee Sean M.","year":"2006","unstructured":"Sean M. McNee , John Riedl , and Joseph A . Konstan . 2006 . Being accurate is not enough: How accuracy metrics have hurt recommender systems. In CHI\u201906 Extended Abstracts on Human Factors in Computing Systems (CHI EA\u2019 06). 1097--1101. Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In CHI\u201906 Extended Abstracts on Human Factors in Computing Systems (CHI EA\u201906). 1097--1101."},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL\u201919)","author":"More Ajinkya","year":"2019","unstructured":"Ajinkya More , Linas Baltrunas , Nikos Vlassis , and Justin Basilico . 2019 . Recap: Designing a more efficient estimator for off-policy evaluation in bandits with large action spaces . In Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL\u201919) . Ajinkya More, Linas Baltrunas, Nikos Vlassis, and Justin Basilico. 2019. Recap: Designing a more efficient estimator for off-policy evaluation in bandits with large action spaces. In Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL\u201919)."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.elerap.2016.09.003"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-017-9195-0"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611973440.53"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190616"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390255"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-017-0977-1"},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909)","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle , Christoph Freudenthaler , Zeno Gantner , and Lars Schmidt-Thieme . 2009 . BPR: Bayesian personalized ranking from implicit feedback . In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909) . 452--461. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI\u201909). 452--461."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365961"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959176"},{"key":"e_1_2_1_85_1","volume-title":"Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE\u201912)","author":"Said Alan","year":"2012","unstructured":"Alan Said , Domonkos Tikk , Yue Shi , Martha Larson , Klara Stumpf , and Paolo Cremonesi . 2012 . Recommender systems evaluation: A 3D benchmark . In Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE\u201912) . 21--23. Alan Said, Domonkos Tikk, Yue Shi, Martha Larson, Klara Stumpf, and Paolo Cremonesi. 2012. Recommender systems evaluation: A 3D benchmark. In Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE\u201912). 21--23."},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.5555\/1046920.1088715"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1089\/cpb.2007.0117"},{"key":"e_1_2_1_88_1","volume-title":"Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI\u201907)","author":"Smyth Barry","year":"2007","unstructured":"Barry Smyth , Paul Cotter , and Stephen Oman . 2007 . Enabling intelligent content discovery on the mobile Internet . In Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI\u201907) . 1744--1751. Barry Smyth, Paul Cotter, and Stephen Oman. 2007. Enabling intelligent content discovery on the mobile Internet. In Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI\u201907). 1744--1751."},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/1081870.1081956"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835895"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488497"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.1050.0058"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9117-5"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043955"},{"key":"e_1_2_1_95_1","volume-title":"Proceedings of the 29th International Conference on Machine Learning (ICML\u201912)","author":"Wagstaff Kiri","year":"2012","unstructured":"Kiri Wagstaff . 2012 . Machine learning that matters . In Proceedings of the 29th International Conference on Machine Learning (ICML\u201912) . 529--536. Kiri Wagstaff. 2012. Machine learning that matters. In Proceedings of the 29th International Conference on Machine Learning (ICML\u201912). 529--536."},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i3.2599"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017327.2017335"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623368"},{"key":"e_1_2_1_99_1","volume-title":"Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201913)","author":"Yi Jeonghee","unstructured":"Jeonghee Yi , Ye Chen , Jie Li , Swaraj Sett , and Tak W. Yan . 2013. Predictive model performance: Offline and online evaluations . In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201913) . 1294--1302. Jeonghee Yi, Ye Chen, Jie Li, Swaraj Sett, and Tak W. Yan. 2013. Predictive model performance: Offline and online evaluations. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201913). 1294--1302."},{"key":"e_1_2_1_100_1","volume-title":"Persuasive Recommender Systems: Conceptual Background and Implications","author":"Yoo Kyung-Hyan","unstructured":"Kyung-Hyan Yoo , Ulrike Gretzel , and Markus Zanker . 2012. Persuasive Recommender Systems: Conceptual Background and Implications . Springer . Kyung-Hyan Yoo, Ulrike Gretzel, and Markus Zanker. 2012. Persuasive Recommender Systems: Conceptual Background and Implications. Springer."},{"key":"e_1_2_1_101_1","volume-title":"Persuasive Recommender Systems\u2014Conceptual Background and Implications","author":"Yoo Kyung-Hyan","unstructured":"Kyung-Hyan Yoo , Ulrike Gretzel , and Markus Zanker . 2013. Persuasive Recommender Systems\u2014Conceptual Background and Implications . Springer . Kyung-Hyan Yoo, Ulrike Gretzel, and Markus Zanker. 2013. Persuasive Recommender Systems\u2014Conceptual Background and Implications. Springer."},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1007\/11823865_6"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-211-77280-5_3"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1145\/1454008.1454030"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864759"},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1000488107"},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/1060745.1060754"}],"container-title":["ACM Transactions on Management Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3370082","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3370082","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:23:34Z","timestamp":1750188214000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3370082"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,10]]},"references-count":107,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,12,31]]}},"alternative-id":["10.1145\/3370082"],"URL":"https:\/\/doi.org\/10.1145\/3370082","relation":{},"ISSN":["2158-656X","2158-6578"],"issn-type":[{"value":"2158-656X","type":"print"},{"value":"2158-6578","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12,10]]},"assertion":[{"value":"2018-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}