{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T12:16:11Z","timestamp":1763468171286,"version":"3.41.0"},"reference-count":13,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2013,1,21]],"date-time":"2013-01-21T00:00:00Z","timestamp":1358726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2013,12,21]]},"abstract":"<jats:p>How can we run large-scale, community-wide evaluations of information retrieval systems if we lack the ability to distribute the document collection on which the task is based? This was the challenge we faced in the TREC Microblog tracks over the past few years. In this paper, we present a novel evaluation methodology we dub \"evaluation as a service\", which was implemented at TREC 2013 to address restrictions on data redistribution. The basic idea is that instead of distributing the document collection, we (the track organizers) provided a service API \"in the cloud\" with which participants could accomplish the evaluation task. We outline advantages as well as disadvantages of this evaluation methodology, and discuss how the approach might be extended to other evaluation scenarios.<\/jats:p>","DOI":"10.1145\/2568388.2568390","type":"journal-article","created":{"date-parts":[[2014,1,28]],"date-time":"2014-01-28T13:49:22Z","timestamp":1390916962000},"page":"8-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Evaluation as a service for information retrieval"],"prefix":"10.1145","volume":"47","author":[{"given":"Jimmy","family":"Lin","sequence":"first","affiliation":[{"name":"The iSchool, University of Maryland, College Park"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miles","family":"Efron","sequence":"additional","affiliation":[{"name":"Graduate School of Library and Information Science, University of Illinois, Urbana-Champaign"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2013,1,21]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484132"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009000"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718538"},{"key":"e_1_2_1_4_1","volume-title":"TREC 2005 spam track overview. In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005","author":"Cormack G.","year":"2005","unstructured":"G. Cormack and T. Lynam . TREC 2005 spam track overview. In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005 ), Gaithersburg, Maryland , 2005 . G. Cormack and T. Lynam. TREC 2005 spam track overview. In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005), Gaithersburg, Maryland, 2005."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/2031434"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-007-9027-7"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9209-9"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Twentieth Text REtrieval Conference (TREC 2011","author":"Ounis I.","year":"2011","unstructured":"I. Ounis , C. Macdonald , J. Lin , and I. Soboroff . Overview of the TREC-2011 Microblog Track . In Proceedings of the Twentieth Text REtrieval Conference (TREC 2011 ), Gaithersburg, Maryland , 2011 . I. Ounis, C. Macdonald, J. Lin, and I. Soboroff. Overview of the TREC-2011 Microblog Track. In Proceedings of the Twentieth Text REtrieval Conference (TREC 2011), Gaithersburg, Maryland, 2011."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076064"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the Twenty-First Text REtrieval Conference (TREC 2012","author":"Soboroff I.","year":"2012","unstructured":"I. Soboroff , I. Ounis , C. Macdonald , and J. Lin . Overview of the TREC-2012 Microblog Track . In Proceedings of the Twenty-First Text REtrieval Conference (TREC 2012 ), Gaithersburg, Maryland , 2012 . I. Soboroff, I. Ounis, C. Macdonald, and J. Lin. Overview of the TREC-2012 Microblog Track. In Proceedings of the Twenty-First Text REtrieval Conference (TREC 2012), Gaithersburg, Maryland, 2012."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291017"},{"key":"e_1_2_1_12_1","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1007\/3-540-45691-0_34","volume-title":"Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross-Language Evaluation Forum","author":"Voorhees E. M.","year":"2002","unstructured":"E. M. Voorhees . The philosophy of information retrieval evaluation . In Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross-Language Evaluation Forum , Lecture Notes in Computer Science Volume 2406 , pages 355 -- 370 , 2002 . E. M. Voorhees. The philosophy of information retrieval evaluation. In Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross-Language Evaluation Forum, Lecture Notes in Computer Science Volume 2406, pages 355--370, 2002."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564432"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2568388.2568390","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2568388.2568390","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:01:38Z","timestamp":1750230098000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2568388.2568390"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,1,21]]},"references-count":13,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2013,12,21]]}},"alternative-id":["10.1145\/2568388.2568390"],"URL":"https:\/\/doi.org\/10.1145\/2568388.2568390","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2013,1,21]]},"assertion":[{"value":"2013-01-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}