{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:20:02Z","timestamp":1750220402356,"version":"3.41.0"},"reference-count":13,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p>Evaluation is a central step when it comes to developing, optimizing, and deploying recommender systems. The PERSPECTIVES 2021 workshop at the 15th ACM Conference on Recommender Systems brought together academia and industry to critically reflect on the evaluation of recommender systems. The primary goal of the workshop was to capture the current state of evaluation from different, and maybe even diverging or contradictory perspectives.<\/jats:p>\n          <jats:p>\n            <jats:bold>Date<\/jats:bold>\n            : 25 and 30 September 2021.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Website<\/jats:bold>\n            : https:\/\/perspectives-ws.github.io\/2021\/.\n          <\/jats:p>","DOI":"10.1145\/3527546.3527565","type":"journal-article","created":{"date-parts":[[2022,3,17]],"date-time":"2022-03-17T23:24:36Z","timestamp":1647559476000},"page":"1-5","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Report on the 1st workshop on the perspectives on the evaluation of recommender systems (PERSPECTIVES 2021) at RecSys 2021"],"prefix":"10.1145","volume":"55","author":[{"given":"Eva","family":"Zangerle","sequence":"first","affiliation":[{"name":"Universit\u00e4t Innsbruck, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christine","family":"Bauer","sequence":"additional","affiliation":[{"name":"Utrecht University, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alan","family":"Said","sequence":"additional","affiliation":[{"name":"University of Gothenburg, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,3,17]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Angwald Anton","year":"2021","unstructured":"Anton Angwald , Kalle Areskoug , and Alan Said . Prediction accuracy and autonomy . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper6.pdf. Anton Angwald, Kalle Areskoug, and Alan Said. Prediction accuracy and autonomy. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper6.pdf."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Barile Francesco","year":"2021","unstructured":"Francesco Barile , Shabnam Najafian , Tim Draws , Oana Inel , Alisa Rieger , Rishav Hada , and Nava Tintarev . Toward benchmarking group explanations: Evaluating the effect of aggregation strategies versus explanation . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper11.pdf. Francesco Barile, Shabnam Najafian, Tim Draws, Oana Inel, Alisa Rieger, Rishav Hada, and Nava Tintarev. Toward benchmarking group explanations: Evaluating the effect of aggregation strategies versus explanation. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper11.pdf."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Diaz Fernando","year":"2021","unstructured":"Fernando Diaz . On evaluating session-based recommendation with implicit feedback . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper5.pdf. Fernando Diaz. On evaluating session-based recommendation with implicit feedback. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper5.pdf."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Filipovic Milena","year":"2021","unstructured":"Milena Filipovic , Blagoj Mitrevski , Diego Antognini , Emma Lejal Glaude , Boi Faltings , and Claudiu Mjusat . Modeling online behavior in recommender systems: The importance of temporal context . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper4.pdf. Milena Filipovic, Blagoj Mitrevski, Diego Antognini, Emma Lejal Glaude, Boi Faltings, and Claudiu Mjusat. Modeling online behavior in recommender systems: The importance of temporal context. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper4.pdf."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"G\u00f3mez Emilia","year":"2021","unstructured":"Emilia G\u00f3mez , Vicky Charisi , and Chaudron Stephane . Evaluating recommender systems with and for children: towards a multi-perspective framework . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper2.pdf. Emilia G\u00f3mez, Vicky Charisi, and Chaudron Stephane. Evaluating recommender systems with and for children: towards a multi-perspective framework. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper2.pdf."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Ihemelandu Ngozi","year":"2021","unstructured":"Ngozi Ihemelandu and Michael D. Ekstrand . Statistical inference: The missing piece of RecSys experiment reliability discourse . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper9.pdf. Ngozi Ihemelandu and Michael D. Ekstrand. Statistical inference: The missing piece of RecSys experiment reliability discourse. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper9.pdf."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Peska Ladislav","year":"2021","unstructured":"Ladislav Peska and Ladislav Malecek . Coupled or decoupled evaluation for group recommendation methods? In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper1.pdf. Ladislav Peska and Ladislav Malecek. Coupled or decoupled evaluation for group recommendation methods? In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper1.pdf."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Scheidt Teresa","year":"2021","unstructured":"Teresa Scheidt and Joeran Beel . Time-dependent evaluation of recommender systems . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper10.pdf. Teresa Scheidt and Joeran Beel. Time-dependent evaluation of recommender systems. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper10.pdf."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Schr\u00f6der Anna Marie","year":"2021","unstructured":"Anna Marie Schr\u00f6der and Malileh Ghajargar . Unboxing the algorithm: Designing an understandable algorithmic experience in music recommender systems . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper12.pdf. Anna Marie Schr\u00f6der and Malileh Ghajargar. Unboxing the algorithm: Designing an understandable algorithmic experience in music recommender systems. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper12.pdf."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Wang Chunpai","year":"2021","unstructured":"Chunpai Wang , Shaghayegh Sahebi , and Peter Brusilovsky . MOCHI : an offline evaluation framework for educational recommendations . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper3.pdf. Chunpai Wang, Shaghayegh Sahebi, and Peter Brusilovsky. MOCHI: an offline evaluation framework for educational recommendations. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper3.pdf."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Woolridge Daniel","year":"2021","unstructured":"Daniel Woolridge , Sean Wilner , and Madeleine Glick . Sequence or pseudo-sequence? An analysis of sequential recommendation datasets . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper8.pdf. Daniel Woolridge, Sean Wilner, and Madeleine Glick. Sequence or pseudo-sequence? An analysis of sequential recommendation datasets. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper8.pdf."},{"key":"e_1_2_1_12_1","series-title":"CEUR Workshop Proceedings","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop","author":"Zangerle Eva","year":"2021","unstructured":"Eva Zangerle , Christine Bauer , and Alan Said , editors. Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop 2021 , volume 2955 of CEUR Workshop Proceedings , 2021. CEUR-WS.org. URL http:\/\/ceur-ws.org\/Vol-2955\/. Eva Zangerle, Christine Bauer, and Alan Said, editors. Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop 2021, volume 2955 of CEUR Workshop Proceedings, 2021. CEUR-WS.org. URL http:\/\/ceur-ws.org\/Vol-2955\/."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings","author":"Zliobaite Indre","year":"2021","unstructured":"Indre Zliobaite . Recommender systems meet species distribution modelling . In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings , September 2021 . URL http:\/\/ceur-ws.org\/Vol-2955\/paper7.pdf. Indre Zliobaite. Recommender systems meet species distribution modelling. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop, PERSPECTIVES '21. CEUR Workshop Proceedings, September 2021. URL http:\/\/ceur-ws.org\/Vol-2955\/paper7.pdf."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527565","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3527546.3527565","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:53Z","timestamp":1750191533000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527565"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":13,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1145\/3527546.3527565"],"URL":"https:\/\/doi.org\/10.1145\/3527546.3527565","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"2022-03-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}