{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T22:15:05Z","timestamp":1777760105568,"version":"3.51.4"},"reference-count":24,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2016,1,29]],"date-time":"2016-01-29T00:00:00Z","timestamp":1454025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2016,1,29]]},"abstract":"<jats:p>The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms. The goal is to create an algorithm that is able to generate news items that users would click, respecting a strict time constraint. The lab challenges participants to compete in either a \"living lab\" (Task 1) or perform an evaluation that replays recorded streams (Task 2). In this report, we discuss the objectives and challenges of the NewsREEL lab, summarize last year's campaign and outline the main research challenges that can be addressed by participating in NewsREEL 2016.<\/jats:p>","DOI":"10.1145\/2888422.2888443","type":"journal-article","created":{"date-parts":[[2016,2,1]],"date-time":"2016-02-01T20:37:54Z","timestamp":1454359074000},"page":"129-136","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Benchmarking News Recommendations"],"prefix":"10.1145","volume":"49","author":[{"given":"Frank","family":"Hopfgartner","sequence":"first","affiliation":[{"name":"University of Glasgow, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Torben","family":"Brodt","sequence":"additional","affiliation":[{"name":"plista GmbH, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonas","family":"Seiler","sequence":"additional","affiliation":[{"name":"plista GmbH, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Kille","sequence":"additional","affiliation":[{"name":"TU Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Lommatzsch","sequence":"additional","affiliation":[{"name":"TU Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Martha","family":"Larson","sequence":"additional","affiliation":[{"name":"TU Delft, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roberto","family":"Turrin","sequence":"additional","affiliation":[{"name":"ContentWise R&amp;D, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andr\u00e1s","family":"Ser\u00e9ny","sequence":"additional","affiliation":[{"name":"Gravity R&amp;D, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,1,29]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE, RUE 2012, Dublin, Ireland, September 9, 2012, volume 910 of CEUR Workshop Proceedings. CEUR-WS.org","author":"Amatriain Xavier","year":"2012","unstructured":"Xavier Amatriain , Pablo Castells , Arjen P. de Vries , Christian Posse , and Harald Steck , editors. Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE, RUE 2012, Dublin, Ireland, September 9, 2012, volume 910 of CEUR Workshop Proceedings. CEUR-WS.org , 2012 Xavier Amatriain, Pablo Castells, Arjen P. de Vries, Christian Posse, and Harald Steck, editors. Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE, RUE 2012, Dublin, Ireland, September 9, 2012, volume 910 of CEUR Workshop Proceedings. CEUR-WS.org, 2012"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661962"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637002.2637028"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753503"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365968"},{"key":"e_1_2_1_6_1","first-page":"2","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Dumais Susan","year":"2009","unstructured":"Susan Dumais . Evaluating IR in situ . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , page 2 , 2009 . Susan Dumais. Evaluating IR in situ. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, page 2, 2009."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2701583.2701587"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645745"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2776784"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2795403.2795416"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-11301-7_35"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11382-1_21"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390198"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2009.82"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767706"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2516641.2516643"},{"key":"e_1_2_1_17_1","volume-title":"Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum","author":"Kille Benjamin","year":"2015","unstructured":"Benjamin Kille , Andreas Lommatzsch , Roberto Turrin , Andr\u00e1s Ser\u00e9ny , Martha Larson , Torben Brodt , Jonas Seiler , and Frank Hopfgartner . Overview of CLEF newsreel 2015: News recommendation evaluation lab . In Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum , Toulouse, France , September 8-11, 2015 ., 2015. Benjamin Kille, Andreas Lommatzsch, Roberto Turrin, Andr\u00e1s Ser\u00e9ny, Martha Larson, Torben Brodt, Jonas Seiler, and Frank Hopfgartner. Overview of CLEF newsreel 2015: News recommendation evaluation lab. In Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, September 8-11, 2015., 2015."},{"key":"e_1_2_1_18_1","first-page":"497","volume-title":"Proceedings of the Sixth International Conference of the CLEF Association, CLEF'15","author":"Kille Benjamin","year":"2015","unstructured":"Benjamin Kille , Andreas Lommatzsch , Roberto Turrin , Andr\u00e1s Ser\u00e9ny , Martha Larson , Torben Brodt , Jonas Seiler , and Frank Hopfgartner . Stream-Based Recommendations : Online and Offline Evaluation as a Service . In Proceedings of the Sixth International Conference of the CLEF Association, CLEF'15 , pages 497 -- 517 , 2015 . Benjamin Kille, Andreas Lommatzsch, Roberto Turrin, Andr\u00e1s Ser\u00e9ny, Martha Larson, Torben Brodt, Jonas Seiler, and Frank Hopfgartner. Stream-Based Recommendations: Online and Offline Evaluation as a Service. In Proceedings of the Sixth International Conference of the CLEF Association, CLEF'15, pages 497--517, 2015."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2785464"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568388.2568390"},{"key":"e_1_2_1_21_1","first-page":"51","volume-title":"ECIR 2014, Amsterdam, The Netherlands, April 13-16, 2014. Proceedings","author":"Lommatzsch Andreas","year":"2014","unstructured":"Andreas Lommatzsch . Real-time news recommendation using context-aware ensembles. In Advances in Information Retrieval - 36th European Conference on IR Research , ECIR 2014, Amsterdam, The Netherlands, April 13-16, 2014. Proceedings , pages 51 -- 62 , 2014 . Andreas Lommatzsch. Real-time news recommendation using context-aware ensembles. In Advances in Information Retrieval - 36th European Conference on IR Research, ECIR 2014, Amsterdam, The Netherlands, April 13-16, 2014. Proceedings, pages 51--62, 2014."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-85820-3_3"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767695"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1080343.1080347"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2888422.2888443","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2888422.2888443","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:05:25Z","timestamp":1750273525000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2888422.2888443"}},"subtitle":["The CLEF NewsREEL Use Case"],"short-title":[],"issued":{"date-parts":[[2016,1,29]]},"references-count":24,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2016,1,29]]}},"alternative-id":["10.1145\/2888422.2888443"],"URL":"https:\/\/doi.org\/10.1145\/2888422.2888443","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2016,1,29]]},"assertion":[{"value":"2016-01-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}