{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T22:44:57Z","timestamp":1759963497929},"publisher-location":"Cham","reference-count":67,"publisher":"Springer International Publishing","isbn-type":[{"type":"print","value":"9783030229474"},{"type":"electronic","value":"9783030229481"}],"license":[{"start":{"date-parts":[[2019,1,1]],"date-time":"2019-01-01T00:00:00Z","timestamp":1546300800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019]]},"DOI":"10.1007\/978-3-030-22948-1_21","type":"book-chapter","created":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T05:02:59Z","timestamp":1565672579000},"page":"511-543","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Continuous Evaluation of Large-Scale Information Access Systems: A Case for Living Labs"],"prefix":"10.1007","author":[{"given":"Frank","family":"Hopfgartner","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Krisztian","family":"Balog","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Lommatzsch","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liadh","family":"Kelly","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Kille","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anne","family":"Schuth","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Martha","family":"Larson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,8,14]]},"reference":[{"issue":"1","key":"21_CR1","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1145\/2215676.2215678","volume":"46","author":"J Allan","year":"2012","unstructured":"Allan J, Croft B, Moffat A, Sanderson M (2012) Frontiers, challenges, and opportunities for information retrieval: report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne. SIGIR Forum 46(1):2\u201332","journal-title":"SIGIR Forum"},{"key":"21_CR2","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1007\/978-3-642-23708-9_5","volume-title":"Multilingual and multimodal information access evaluation. Proceedings of the second international conference of the cross-language evaluation forum (CLEF 2011)","author":"L Azzopardi","year":"2011","unstructured":"Azzopardi L, Balog K (2011) Towards a living lab for information retrieval research and development - a proposal for a living lab for product search tasks. In: Forner P, Gonzalo J, Kek\u00e4l\u00e4inen J, Lalmas M, de Rijke M (eds) Multilingual and multimodal information access evaluation. Proceedings of the second international conference of the cross-language evaluation forum (CLEF 2011). Lecture notes in computer science (LNCS), vol 6941. Springer, Heidelberg, pp 26\u201337"},{"issue":"1","key":"21_CR3","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1145\/2641383.2641388","volume":"48","author":"K Balog","year":"2014","unstructured":"Balog K, Elsweiler D, Kanoulas E, Kelly L, Smucker MD (2014a) Report on the CIKM workshop on living labs for information retrieval evaluation. SIGIR Forum 48(1):21\u201328","journal-title":"SIGIR Forum"},{"key":"21_CR4","first-page":"1815","volume-title":"Proceedings of the 23rd international conference on information and knowledge management (CIKM\u201914)","author":"K Balog","year":"2014","unstructured":"Balog K, Kelly L, Schuth A (2014b) Head first: living labs for ad-hoc search evaluation. In: Proceedings of the 23rd international conference on information and knowledge management (CIKM\u201914). ACM, New York, pp 1815\u20131818"},{"key":"21_CR5","unstructured":"Beck PD, Blaser M, Michalke A, Lommatzsch A (2017) A system for online news recommendations in real-time with Apache mahout. In: Working notes of the 8th international conference of the CLEF initiative, Dublin, CEUR workshop proceedings"},{"key":"21_CR6","unstructured":"Bons P, Evans N, Kampstra P, van Kessel T (2017) A news recommender engine with a killer sequence. In: Working notes of the 8th international conference of the CLEF initiative, Dublin, CEUR workshop proceedings"},{"issue":"2","key":"21_CR7","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/792550.792552","volume":"36","author":"A Broder","year":"2002","unstructured":"Broder A (2002) A taxonomy of web search. SIGIR Forum 36(2):3\u201310","journal-title":"SIGIR Forum"},{"key":"21_CR8","first-page":"223","volume-title":"Proceedings of the information interaction in context conference, IIiX\u201914","author":"T Brodt","year":"2014","unstructured":"Brodt T, Hopfgartner F (2014) Shedding light on a living lab: the CLEF NewsREEL open recommendation platform. In: Proceedings of the information interaction in context conference, IIiX\u201914. Springer, New York, pp 223\u2013226"},{"key":"21_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2094072.2094078","volume":"30","author":"O Chapelle","year":"2012","unstructured":"Chapelle O, Joachims T, Radlinski F, Yue Y (2012) Large-scale validation and analysis of interleaved search evaluation. ACM Trans Info Syst (TOIS) 30:1\u201341","journal-title":"ACM Trans Info Syst (TOIS)"},{"key":"21_CR10","unstructured":"Ciobanu A, Lommatzsch A (2016) Development of a news recommender system based on Apache flink. In: Working notes of the 7th international conference of the CLEF initiative, Evora, CEUR workshop proceedings"},{"key":"21_CR11","unstructured":"Corsini F, Larson M (2016) CLEF NewsREEL 2016: image based recommendation. In: Working notes of the 7th international conference of the CLEF initiative, Evora, CEUR workshop proceedings"},{"key":"21_CR12","doi-asserted-by":"crossref","unstructured":"Diaz F, White R, Buscher G, Liebling D (2013) Robust models of mouse movement on dynamic web search results pages. In: Proceedings of the 22nd ACM international conference on information and knowledge management (CIKM\u201913), pp 1451\u20131460","DOI":"10.1145\/2505515.2505717"},{"key":"21_CR13","unstructured":"Domann J, Meiners J, Helmers L, Lommatzsch A (2016) Real-time news recommendations using Apache spark. In: Working notes of the 7th international conference of the CLEF initiative, Evora, CEUR workshop proceedings"},{"key":"21_CR14","unstructured":"Freire J, Fuhr N, Rauber A (2016) Reproducibility of data-oriented experiments in e-science (Dagstuhl seminar 16041). In: Dagstuhl reports, Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, vol 6"},{"key":"21_CR15","unstructured":"Gebremeskel G, de Vries AP (2015) The degree of randomness in a live recommender systems evaluation. In: Working notes for CLEF 2015 conference, Toulouse, CEUR"},{"key":"21_CR16","unstructured":"Ghirmatsion AB, Balog K (2015) Probabilistic field mapping for product search. In: CLEF 2015 online working notes"},{"key":"21_CR17","unstructured":"Golian C, Kuchar J (2017) News recommender system based on association rules at CLEF NewsREEL 2017. In: Working notes of the 8th international conference of the CLEF initiative, Dublin, CEUR workshop proceedings"},{"key":"21_CR18","first-page":"2935","volume":"10","author":"A Gunawardana","year":"2009","unstructured":"Gunawardana A, Shani G (2009) A survey of accuracy evaluation metrics of recommendation tasks. J Mach Learn Res 10:2935\u20132962","journal-title":"J Mach Learn Res"},{"key":"21_CR19","unstructured":"Hanbury A, M\u00fcller H, Balog K, Brodt T, Cormack GV, Eggel I, Gollub T, Hopfgartner F, Kalpathy-Cramer J, Kando N, Krithara A, Lin JJ, Mercer S, Potthast M (2015) Evaluation-as-a-service: overview and outlook. CoRR abs\/1512.07454"},{"key":"21_CR20","first-page":"2019","volume-title":"Proceedings of the 22nd ACM international conference on information and knowledge management (CIKM\u201913)","author":"A Hassan","year":"2013","unstructured":"Hassan A, Shi X, Craswell N, Ramsey B (2013) Beyond clicks: query reformulation as a predictor of search satisfaction. In: Proceedings of the 22nd ACM international conference on information and knowledge management (CIKM\u201913). ACM, New York, pp 2019\u20132028"},{"issue":"2015","key":"21_CR21","first-page":"1077","volume":"9\u201313","author":"D Hawking","year":"2015","unstructured":"Hawking D (2015) If SIGIR had an academic track, what would be in it? In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, Santiago, August 9\u201313, 2015, p 1077","journal-title":"August"},{"key":"21_CR22","doi-asserted-by":"crossref","unstructured":"Hofmann K, Whiteson S, de Rijke M (2011) A probabilistic method for inferring preferences from clicks. In: Proceedings of the 20th conference on information and knowledge management (CIKM\u201911). ACM, New York, p 249","DOI":"10.1145\/2063576.2063618"},{"key":"21_CR23","first-page":"826","volume-title":"Lecture Notes in Computer Science","author":"Frank Hopfgartner","year":"2015","unstructured":"Hopfgartner F, Brodt T (2015) Join the living lab: evaluating news recommendations in real-time. In: Advances in information retrieval - 37th European conference on IR research, ECIR 2015, Proceedings, Vienna, March 29\u2013April 2, 2015, pp 826\u2013829"},{"issue":"2","key":"21_CR24","doi-asserted-by":"publisher","first-page":"1029","DOI":"10.1007\/s11042-012-1254-2","volume":"73","author":"F Hopfgartner","year":"2014","unstructured":"Hopfgartner F, Jose JM (2014) An experimental evaluation of ontology-based user profiles. Multimed Tools Appl 73(2):1029\u20131051","journal-title":"Multimed Tools Appl"},{"key":"21_CR25","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1007\/978-3-319-11382-1_21","volume-title":"Information access evaluation \u2013 multilinguality, multimodality, and interaction. Proceedings of the fifth international conference of the CLEF initiative (CLEF 2014)","author":"F Hopfgartner","year":"2014","unstructured":"Hopfgartner F, Kille B, Lommatzsch A, Plumbaum T, Brodt T, Heintz T (2014) Benchmarking news recommendations in a living lab. In: Kanoulas E, Lupu M, Clough P, Sanderson M, Hall M, Hanbury A, Toms E (eds) Information access evaluation \u2013 multilinguality, multimodality, and interaction. Proceedings of the fifth international conference of the CLEF initiative (CLEF 2014). Lecture notes in computer science (LNCS), vol 8685. Springer, Heidelberg, pp 250\u2013267"},{"issue":"2","key":"21_CR26","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1145\/2888422.2888443","volume":"49","author":"F Hopfgartner","year":"2015","unstructured":"Hopfgartner F, Brodt T, Seiler J, Kille B, Lommatzsch A, Larson M, Turrin R, Ser\u00e9ny A (2015a) Benchmarking news recommendations: the CLEF newsreel use case. SIGIR Forum 49(2):129\u2013136","journal-title":"SIGIR Forum"},{"key":"21_CR27","doi-asserted-by":"crossref","unstructured":"Hopfgartner F, Kille B, Heintz T, Turrin R (2015b) Real-time recommendation of streamed data. In: Proceedings of the 9th ACM conference on recommender systems, RecSys 2015, Vienna, September 16\u201320, 2015, pp 361\u2013362","DOI":"10.1145\/2792838.2792839"},{"key":"21_CR28","unstructured":"Hopfgartner F, Lommatzsch A, Kille B, Larson M, Brodt T, Cremonesi P, Karatzoglou A (2016) The potentials of recommender systems challenges for student learning. In: Proceedings of CiML\u201916: challenges in machine learning: gaming and education"},{"issue":"4","key":"21_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3239570","volume":"10","author":"Frank Hopfgartner","year":"2018","unstructured":"Hopfgartner F, Hanbury A, Mueller H, Eggel I, Balog K, Brodt T, Cormack GV, Lin J, Kalpathy-Cramer J, Kando N, Kato MP, Krithara A, Gollub T, Potthast M, Viegas E, Mercer S (2018) Evaluation-as-a-service for the computational sciences: overview and outlook. ACM J Data Inf Qual. https:\/\/doi.org\/10.1145\/3239570","journal-title":"Journal of Data and Information Quality"},{"issue":"3","key":"21_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3239575","volume":"10","author":"Rolf Jagerman","year":"2018","unstructured":"Jagerman R, Balog K, de Rijke M (2018) Opensearch: lessons learned from an online evaluation campaign. J Data Inf Qual 10(3):13:1\u201313:15","journal-title":"Journal of Data and Information Quality"},{"key":"21_CR31","first-page":"79","volume-title":"Text mining","author":"T Joachims","year":"2003","unstructured":"Joachims T (2003) Evaluating retrieval performance using clickthrough data. In: Franke J, Nakhaeizadeh G, Renz I (eds) Text mining. Physica. Springer, Heidelberg, pp 79\u201396"},{"issue":"2","key":"21_CR32","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1145\/1229179.1229181","volume":"25","author":"T Joachims","year":"2007","unstructured":"Joachims T, Granka LA, Pan B, Hembrooke H, Radlinski F, Gay G (2007) Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans Inf Syst 25(2):7","journal-title":"ACM Trans Inf Syst"},{"issue":"2","key":"21_CR33","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1145\/1670564.1670567","volume":"43","author":"J Kamps","year":"2009","unstructured":"Kamps J, Geva S, Peters C, Sakai T, Trotman A, Voorhees E (2009) Report on the SIGIR 2009 workshop on the future of IR evaluation. SIGIR Forum 43(2):13\u201323","journal-title":"SIGIR Forum"},{"issue":"3","key":"21_CR34","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1109\/MC.2009.82","volume":"42","author":"D Kelly","year":"2009","unstructured":"Kelly D, Dumais ST, Pedersen JO (2009) Evaluation challenges and directions for information-seeking support systems. IEEE Comput 42(3):60\u201366","journal-title":"IEEE Comput"},{"key":"21_CR35","first-page":"544","volume-title":"Lecture Notes in Computer Science","author":"Liadh Kelly","year":"2012","unstructured":"Kelly L, Bunbury P, Jones GJF (2012) Evaluating personal information retrieval. In: Proceedings of the 34th European conference on information retrieval (ECIR\u201912). Springer, Berlin"},{"key":"21_CR36","first-page":"14","volume-title":"NRS\u201913: proceedings of the international workshop and challenge on news recommender systems","author":"B Kille","year":"2013","unstructured":"Kille B, Hopfgartner F, Brodt T, Heintz T (2013) The plista dataset. In: NRS\u201913: proceedings of the international workshop and challenge on news recommender systems. ACM, New York, pp 14\u201321"},{"key":"21_CR37","first-page":"497","volume-title":"Lecture Notes in Computer Science","author":"Benjamin Kille","year":"2015","unstructured":"Kille B, Lommatzsch A, Turrin R, Ser\u00e9ny A, Larson M, Brodt T, Seiler J, Hopfgartner F (2015) Stream-based recommendations: online and offline evaluation as a service. In: Mothe J, Savoy J, Kamps J, Pinel-Sauvagnat K, Jones GJF, SanJuan E, Cappellato L, Ferro N (eds) Experimental IR meets multilinguality, multimodality, and interaction. Proceedings of the sixth international conference of the CLEF association (CLEF 2015). Lecture notes in computer science (LNCS), vol 9283. Springer, Heidelberg, pp 497\u2013517"},{"key":"21_CR38","first-page":"228","volume-title":"Lecture Notes in Computer Science","author":"Jinyoung Kim","year":"2009","unstructured":"Kim J, Xue X, Croft WB (2009) A probabilistic retrieval model for semistructured data. In: Proc. of the 31st European conference on information retrieval (ECIR\u201909). Springer, Heidelberg, pp 228\u2013239"},{"key":"21_CR39","doi-asserted-by":"crossref","unstructured":"Kim Y, Hassan A, White R, Zitouni I (2014) Modeling dwell time to predict click-level satisfaction. In: Proc. of the 7th ACM international conference on web search and data mining (WSDM\u201914). ACM, New York, pp 193\u2013202","DOI":"10.1145\/2556195.2556220"},{"key":"21_CR40","doi-asserted-by":"crossref","unstructured":"Kohavi R (2015) Online controlled experiments: lessons from running A\/B\/n tests for 12 Years. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, Sydney, NSW, August 10\u201313, 2015, p 1","DOI":"10.1145\/2783258.2785464"},{"key":"21_CR41","unstructured":"Kumar V, Khattar D, Gupta S, Gupta M, Varma V (2017) Deep neural architecture for news recommendation. In: Working notes of the 8th international conference of the CLEF initiative, Dublin, CEUR workshop proceedings"},{"key":"21_CR42","first-page":"43","volume-title":"Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval (SIGIR \u201909)","author":"J Li","year":"2009","unstructured":"Li J, Huffman S, Tokuda A (2009) Good abandonment in mobile and pc internet search. In: Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval (SIGIR \u201909). ACM, New York, pp 43\u201350"},{"key":"21_CR43","unstructured":"Liang Y, Loni B, Larson M (2017) CLEF NewsREEL 2017: contextual bandit news recommendation. In: Working notes of the 8th international conference of the CLEF initiative, Dublin, CEUR workshop proceedings"},{"issue":"3","key":"21_CR44","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1561\/1500000016","volume":"3","author":"TY Liu","year":"2009","unstructured":"Liu TY (2009) Learning to rank for information retrieval. Found Trends Inf Retr 3(3):225\u2013331","journal-title":"Found Trends Inf Retr"},{"key":"21_CR45","unstructured":"Liu TY, Xu J, Qin T, Xiong W, Li H (2007) LETOR: benchmark dataset for research on learning to rank for information retrieval. In: Proceedings of SIGIR 2007 workshop on learning to rank for information retrieval (LR4IR\u201907), pp 346\u2013374"},{"key":"21_CR46","first-page":"1039","volume-title":"SAC \u201915","author":"Lommatzsch A, Albayrak S (2015) Real-time recommendations for user-item streams. In: Proc. of the 30th symposium on applied computing, SAC","year":"2015","unstructured":"Lommatzsch A, Albayrak S (2015) Real-time recommendations for user-item streams. In: Proc. of the 30th symposium on applied computing, SAC 2015, SAC \u201915. ACM, New York, pp 1039\u20131046"},{"key":"21_CR47","unstructured":"Lommatzsch A, Johannes N, Meiners J, Helmers L, Domann J (2016) Recommender ensembles for news articles based on most-popular strategies. In: Working notes of the 7th international conference of the CLEF initiative, Evora, CEUR workshop proceedings"},{"key":"21_CR48","first-page":"239","volume-title":"Lecture Notes in Computer Science","author":"Andreas Lommatzsch","year":"2017","unstructured":"Lommatzsch A, Kille B, Hopfgartner F, Larson M, Brodt T, Seiler J, \u00d6zg\u00f6bek \u00d6 (2017) CLEF 2017 NewsREEL overview: a stream-based recommender task for evaluation and education. In: Jones GJF, Lawless S, Gonzalo J, Kelly L, Goeuriot L, Mandl T, Cappellato L, Ferro N (eds) Experimental IR meets multilinguality, multimodality, and interaction. Proceedings of the eighth international conference of the CLEF association (CLEF 2017). Lecture notes in computer science (LNCS), vol 10456. Springer, Heidelberg, pp 239\u2013254"},{"key":"21_CR49","unstructured":"Ludmann C (2017) Recommending news articles in the CLEF news recommendation evaluation lab with the data stream management system odysseus. In: Working notes of the 8th international conference of the CLEF initiative, Dublin, CEUR workshop proceedings"},{"key":"21_CR50","doi-asserted-by":"crossref","unstructured":"Radlinski F, Craswell N (2013) Optimized interleaving for online retrieval evaluation. In: Proc. of ACM international conference on web search and data mining (WSDM\u201913). ACM, New York, pp 245\u2013254","DOI":"10.1145\/2433396.2433429"},{"key":"21_CR51","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1145\/1458082.1458092","volume-title":"Proceedings of the 17th conference on information and knowledge management (CIKM\u201908)","author":"F Radlinski","year":"2008","unstructured":"Radlinski F, Kurup M, Joachims T (2008) How does clickthrough data reflect retrieval quality? In: Proceedings of the 17th conference on information and knowledge management (CIKM\u201908). ACM, New York, pp 43\u201352"},{"key":"21_CR52","unstructured":"Said A, Tikk D, Stumpf K, Shi Y, Larson M, Cremonesi P (2012) Recommender systems evaluation: a 3D benchmark. In: Proceedings of the workshop on recommendation utility evaluation: beyond RMSE (RUE 2012), CEUR-WS, vol 910, RUE\u201912, pp 21\u201323"},{"key":"21_CR53","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-1199-4","volume-title":"Laboratory experiments in information retrieval","author":"T Sakai","year":"2018","unstructured":"Sakai T (2018) Laboratory experiments in information retrieval. Springer, Singapore"},{"key":"21_CR54","unstructured":"Schaer P, Tavakolpoursaleh N (2015) GESIS at CLEF LL4IR 2015. In: CLEF 2015 Online Working Notes"},{"key":"21_CR55","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1145\/2661829.2661952","volume-title":"Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM\u201914)","author":"A Schuth","year":"2014","unstructured":"Schuth A, Sietsma F, Whiteson S, Lefortier D, de Rijke M (2014) Multileaved comparisons for fast online evaluation. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM\u201914). ACM, New York, pp 71\u201380"},{"key":"21_CR56","doi-asserted-by":"crossref","unstructured":"Schuth A, Balog K, Kelly L (2015a) Extended overview of the living labs for information retrieval evaluation (LL4IR) CLEF lab 2015. In: CLEF 2015 online working notes","DOI":"10.1007\/978-3-319-24027-5_47"},{"key":"21_CR57","first-page":"955","volume-title":"Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval (SIGIR\u201915)","author":"A Schuth","year":"2015","unstructured":"Schuth A, Bruintjes RJ, B\u00fcttner F, van Doorn J, Groenland C, Oosterhuis H, Tran CN, Veeling B, van der Velde J, Wechsler R, Woudenberg D, de Rijke M (2015b) Probabilistic multileave for online retrieval evaluation. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval (SIGIR\u201915). ACM, New York, pp 955\u2013958"},{"key":"21_CR58","doi-asserted-by":"crossref","unstructured":"Schuth A, Hofmann K, Radlinski F (2015c) Predicting search satisfaction metrics with interleaved comparisons. In: Proceedings of the 38th ACM international conference on information retrieval (SIGIR\u201915). ACM, New York pp 463\u2013472","DOI":"10.1145\/2766462.2767695"},{"key":"21_CR59","first-page":"2016","volume":"17","author":"Scriminaci M, Lommatzsch A, Kille B, Hopfgartner F, Larson M, Malagoli D, Ser\u00e9ny A, Plumbaum T (2016) Idomaar: a framework for multi-dimensional benchmarking of recommender algorithms. In: Proceedings of the poster track of the 10th ACM conference on recommender systems (RecSys","year":"2016","unstructured":"Scriminaci M, Lommatzsch A, Kille B, Hopfgartner F, Larson M, Malagoli D, Ser\u00e9ny A, Plumbaum T (2016) Idomaar: a framework for multi-dimensional benchmarking of recommender algorithms. In: Proceedings of the poster track of the 10th ACM conference on recommender systems (RecSys 2016), Boston, September 17, 2016","journal-title":"September"},{"key":"21_CR60","doi-asserted-by":"crossref","unstructured":"Tavakolifard M, Gulla JA, Almeroth KC, Hopfgartner F, Kille B, Plumbaum T, Lommatzsch A, Brodt T, Bucko A, Heintz T (2013) Workshop and challenge on news recommender systems. In: Seventh ACM conference on recommender systems, RecSys \u201913, Hong Kong, October 12\u201316, 2013, pp 481\u2013482","DOI":"10.1145\/2507157.2508004"},{"key":"21_CR61","first-page":"756","volume-title":"Proceedings of the ACM international conference on information retrieval (SIGIR\u201907)","author":"J Teevan","year":"2007","unstructured":"Teevan J, Dumais S, Horvitz E (2007) The potential value of personalizing search. In: Proceedings of the ACM international conference on information retrieval (SIGIR\u201907). ACM, New York, pp 756\u2013757"},{"key":"21_CR62","doi-asserted-by":"crossref","unstructured":"Turpin A, Scholar F (2006) User performance versus precision measures for simple search tasks. In: Proc. of the ACM international conference on information retrieval (SIGIR\u201906). ACM, New York, pp 11\u201318","DOI":"10.1145\/1148170.1148176"},{"key":"21_CR63","unstructured":"Verbitskiy I, Probst P, Lommatzsch A (2015) Developing and evaluation of a highly scalable news recommender system. In: Working notes for CLEF 2015 conference, Toulouse, CEUR"},{"key":"21_CR64","volume-title":"TREC: Experiment and evaluation in information retrieval","author":"EM Voorhees","year":"2005","unstructured":"Voorhees EM, Harman DK (2005) TREC: Experiment and evaluation in information retrieval, 1st edn. MIT Press, Cambridge, MA","edition":"1"},{"key":"21_CR65","first-page":"211","volume-title":"WSDM\u201910","author":"K Wang","year":"2010","unstructured":"Wang K, Gloy N, Li X (2010) Inferring search behaviors using partially observable Markov (POM) model. In: WSDM\u201910. ACM, New York, pp 211\u2013220"},{"key":"21_CR66","unstructured":"Wilkins P, Byrne D, Jones GJF, Lee H, Keenan G, McGuinness K, O\u2019Connor NE, O\u2019Hare N, Smeaton AF, Adamek T, Troncy R, Amin A, Benmokhtar R, Dumont E, Huet B, M\u00e9rialdo B, Tolias G, Spyrou E, Avrithis YS, Papadopoulos GT, Mezaris V, Kompatsiaris I, M\u00f6rzinger R, Schallauer P, Bailer W, Chandramouli K, Izquierdo E, Goldmann L, Haller M, Samour A, Cobet A, Sikora T, Praks P, Hannah D, Halvey M, Hopfgartner F, Villa R, Punitha P, Goyal A, Jose JM (2008) K-space at TRECVid 2008. In: TRECVID 2008 workshop participants notebook papers, Gaithersburg, MD, Nov 2008"},{"key":"21_CR67","first-page":"91","volume-title":"Proceedings of the 23rd ACM international conference on information and knowledge management (CIKM\u201914)","author":"E Yilmaz","year":"2014","unstructured":"Yilmaz E, Verma M, Craswell N, Radlinski F, Bailey P (2014) Relevance and effort: an analysis of document utility. In: Proceedings of the 23rd ACM international conference on information and knowledge management (CIKM\u201914). ACM, New York, pp 91\u2013100"}],"container-title":["The Information Retrieval Series","Information Retrieval Evaluation in a Changing World"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-22948-1_21","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,16]],"date-time":"2021-01-16T14:52:01Z","timestamp":1610808721000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/978-3-030-22948-1_21"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"ISBN":["9783030229474","9783030229481"],"references-count":67,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-22948-1_21","relation":{},"ISSN":["1387-5264"],"issn-type":[{"type":"print","value":"1387-5264"}],"subject":[],"published":{"date-parts":[[2019]]},"assertion":[{"value":"14 August 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}}]}}