{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T18:01:12Z","timestamp":1760983272489,"version":"3.41.0"},"reference-count":30,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2009,12,14]],"date-time":"2009-12-14T00:00:00Z","timestamp":1260748800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2009,12,14]]},"abstract":"<jats:p>On July 23, 2009 the SIGIR Workshop on the Future of IR Evaluation was held as part of SIGIR in Boston. The program consisted of four keynotes, a boaster and poster session with 20 accepted papers, four breakout groups, and a final panel discussion of the breakout group reports. This report outlines the events of the workshop and summarizes the major outcomes.<\/jats:p>","DOI":"10.1145\/1670564.1670567","type":"journal-article","created":{"date-parts":[[2009,12,21]],"date-time":"2009-12-21T14:08:04Z","timestamp":1261404484000},"page":"13-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Report on the SIGIR 2009 workshop on the future of IR evaluation"],"prefix":"10.1145","volume":"43","author":[{"given":"Jaap","family":"Kamps","sequence":"first","affiliation":[{"name":"University of Amsterdam, The Netherlands &amp; INEX"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shlomo","family":"Geva","sequence":"additional","affiliation":[{"name":"QUT, Australia &amp; INEX"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carol","family":"Peters","sequence":"additional","affiliation":[{"name":"ISTI-CNR, Italy &amp; CLEF"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tetsuya","family":"Sakai","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, China &amp; NTCIR"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew","family":"Trotman","sequence":"additional","affiliation":[{"name":"University of Otago, New Zealand &amp; INEX"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ellen","family":"Voorhees","sequence":"additional","affiliation":[{"name":"NIST, USA &amp; TREC\/TAC"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2009,12,14]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"35","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Ali S.","year":"2009","unstructured":"S. Ali and M. Consens . Enhanced web retrieval task . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 35 -- 36 , 2009 . S. Ali and M. Consens. Enhanced web retrieval task. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 35--36, 2009."},{"key":"e_1_2_1_2_1","first-page":"15","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Alonso O.","year":"2009","unstructured":"O. Alonso and S. Mizzaro . Can we get rid of TREC assessors? using Mechanical Turk for relevance assessment . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 15 -- 16 , 2009 . O. Alonso and S. Mizzaro. Can we get rid of TREC assessors? using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 15--16, 2009."},{"key":"e_1_2_1_3_1","first-page":"25","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Armstrong T.","year":"2009","unstructured":"T. Armstrong , J. Zobel , W. Webber , and A. Moffat . Relative significance is insufficient: Baselines matter too . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 25 -- 26 , 2009 . T. Armstrong, J. Zobel, W. Webber, and A. Moffat. Relative significance is insufficient: Baselines matter too. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 25--26, 2009."},{"key":"e_1_2_1_4_1","first-page":"7","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Belkin N.","year":"2009","unstructured":"N. Belkin , M. Cole , and J. Liu . A model for evaluation of interactive information retrieval . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 7 -- 8 , 2009 . N. Belkin, M. Cole, and J. Liu. A model for evaluation of interactive information retrieval. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 7--8, 2009."},{"key":"e_1_2_1_5_1","first-page":"1","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Buckley C.","year":"2009","unstructured":"C. Buckley . Towards good evaluation of individual topics . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , page 1 , 2009 . C. Buckley. Towards good evaluation of individual topics. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, page 1, 2009."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1328964.1328967"},{"key":"e_1_2_1_7_1","volume-title":"Report on the first stage of an investigation into the comparative efficiency of indexing systems. Technical report","author":"Cleverdon C.W.","year":"1960","unstructured":"C.W. Cleverdon . Report on the first stage of an investigation into the comparative efficiency of indexing systems. Technical report , College of Aeronautics , Cranfield UK , 1960 . C.W. Cleverdon. Report on the first stage of an investigation into the comparative efficiency of indexing systems. Technical report, College of Aeronautics, Cranfield UK, 1960."},{"key":"e_1_2_1_8_1","first-page":"27","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Collins-Thompson K.","year":"2009","unstructured":"K. Collins-Thompson . Accounting for stability of retrieval algorithms using risk-reward curves . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 27 -- 28 , 2009 . K. Collins-Thompson. Accounting for stability of retrieval algorithms using risk-reward curves. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 27--28, 2009."},{"key":"e_1_2_1_9_1","first-page":"37","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Costa M.","year":"2009","unstructured":"M. Costa and M. Silva . Towards information retrieval evaluation over web archives . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 37 -- 38 , 2009 . M. Costa and M. Silva. Towards information retrieval evaluation over web archives. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 37--38, 2009."},{"key":"e_1_2_1_10_1","first-page":"17","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Crecelius T.","year":"2009","unstructured":"T. Crecelius and R. Schenkel . Evaluating network-aware retrieval in social networks . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 17 -- 18 , 2009 . T. Crecelius and R. Schenkel. Evaluating network-aware retrieval in social networks. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 17--18, 2009."},{"key":"e_1_2_1_11_1","first-page":"2","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Dumais S.","year":"2009","unstructured":"S. Dumais . Evaluating IR in situ . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , page 2 , 2009 . S. Dumais. Evaluating IR in situ. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, page 2, 2009."},{"key":"e_1_2_1_12_1","first-page":"3","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Dupret G.","year":"2009","unstructured":"G. Dupret . User models to compare and evaluate web IR metrics . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , page 3 , 2009 . G. Dupret. User models to compare and evaluate web IR metrics. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, page 3, 2009."},{"key":"e_1_2_1_13_1","volume-title":"An online tool for evaluating and comparing ir systems","author":"IR.","year":"2009","unstructured":"Evaluat IR. An online tool for evaluating and comparing ir systems , 2009 . http:\/\/evaluatir.org\/. EvaluatIR. An online tool for evaluating and comparing ir systems, 2009. http:\/\/evaluatir.org\/."},{"key":"e_1_2_1_14_1","first-page":"29","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Hanbury A.","year":"2009","unstructured":"A. Hanbury and H. M\u00fcller . Toward automated component-level evaluation . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 29 -- 30 , 2009 . A. Hanbury and H. M\u00fcller. Toward automated component-level evaluation. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 29--30, 2009."},{"key":"e_1_2_1_15_1","first-page":"5","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Hawking D.","year":"2009","unstructured":"D. Hawking , P. Thomas , T. Gedeon , T. Rowlands , and T. Jones . New methods for creating testfiles: Tuning enterprise search with C-TEST . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 5 -- 6 , 2009 . D. Hawking, P. Thomas, T. Gedeon, T. Rowlands, and T. Jones. New methods for creating testfiles: Tuning enterprise search with C-TEST. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 5--6, 2009."},{"key":"e_1_2_1_16_1","first-page":"19","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Huang W. C. D.","year":"2009","unstructured":"W. C. D. Huang , A. Trotman , and S. Geva . A virtual evaluation forum for cross language link discovery . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 19 -- 20 , 2009 . W. C. D. Huang, A. Trotman, and S. Geva. A virtual evaluation forum for cross language link discovery. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 19--20, 2009."},{"key":"e_1_2_1_17_1","first-page":"21","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Kazai G.","year":"2009","unstructured":"G. Kazai and N. Milic-Frayling . On the evaluation of the quality of relevance assessments collected through crowdsourcing . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 21 -- 22 , 2009 . G. Kazai and N. Milic-Frayling. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 21--22, 2009."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2009.82"},{"key":"e_1_2_1_19_1","first-page":"39","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Kim J.","year":"2009","unstructured":"J. Kim and B. Croft . Building pseudo-desktop collections . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 39 -- 40 , 2009 . J. Kim and B. Croft. Building pseudo-desktop collections. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 39--40, 2009."},{"key":"e_1_2_1_20_1","first-page":"41","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Lathia N.","year":"2009","unstructured":"N. Lathia , S. Hailes , and L. Capra . Evaluating collaborative filtering over time . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 41 -- 42 , 2009 . N. Lathia, S. Hailes, and L. Capra. Evaluating collaborative filtering over time. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 41--42, 2009."},{"key":"e_1_2_1_21_1","unstructured":"Lemur Query Log Project 2009. http:\/\/lemurstudy.cs.umass.edu\/.  Lemur Query Log Project 2009. http:\/\/lemurstudy.cs.umass.edu\/."},{"key":"e_1_2_1_22_1","first-page":"31","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Liu H.","year":"2009","unstructured":"H. Liu , R. Song , J.-Y. Nie , and J.- R. Wen . Building a test collection for evaluating search result diversity: A preliminary study . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 31 -- 32 , 2009 . H. Liu, R. Song, J.-Y. Nie, and J.-R.Wen. Building a test collection for evaluating search result diversity: A preliminary study. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 31--32, 2009."},{"key":"e_1_2_1_23_1","first-page":"43","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Llopis F.","year":"2009","unstructured":"F. Llopis , A. Escapa , A. Ferrandez , S. Navarro , and E. Noguera . How long can you wait for your QA system ? In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 43 -- 44 , 2009 . F. Llopis, A. Escapa, A. Ferrandez, S. Navarro, and E. Noguera. How long can you wait for your QA system? In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 43--44, 2009."},{"key":"e_1_2_1_24_1","first-page":"9","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Paris C.","year":"2009","unstructured":"C. Paris , N. Colineau , P. Thomas , and R. Wilkinson . Stakeholders and their respective costs-benefits in IR evaluation . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 9 -- 10 , 2009 . C. Paris, N. Colineau, P. Thomas, and R. Wilkinson. Stakeholders and their respective costs-benefits in IR evaluation. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 9--10, 2009."},{"key":"e_1_2_1_25_1","first-page":"4","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Robertson S.","year":"2009","unstructured":"S. Robertson . Richer theories, richer experiments . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , page 4 , 2009 . S. Robertson. Richer theories, richer experiments. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, page 4, 2009."},{"key":"e_1_2_1_26_1","first-page":"33","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Shokouhi M.","year":"2009","unstructured":"M. Shokouhi , E. Yilmaz , N. Craswell , and S. Robertson . Are evaluation metrics identical with binary judgements ? In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 33 -- 34 , 2009 . M. Shokouhi, E. Yilmaz, N. Craswell, and S. Robertson. Are evaluation metrics identical with binary judgements? In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 33--34, 2009."},{"key":"e_1_2_1_27_1","first-page":"11","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Smucker M.","year":"2009","unstructured":"M. Smucker . A plan for making information retrieval evaluation synonymous with human performance prediction . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 11 -- 12 , 2009 . M. Smucker. A plan for making information retrieval evaluation synonymous with human performance prediction. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 11--12, 2009."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1147197.1147198"},{"key":"e_1_2_1_29_1","first-page":"13","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Stamou S.","year":"2009","unstructured":"S. Stamou and E. Efthimiadis . Queries without clicks: Successful or failed searches ? In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 13 -- 14 , 2009 . S. Stamou and E. Efthimiadis. Queries without clicks: Successful or failed searches? In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 13--14, 2009."},{"key":"e_1_2_1_30_1","first-page":"23","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Yue Z.","year":"2009","unstructured":"Z. Yue , A. Harplale , D. He , J. Grady , Y. Lin , J. Walker , S. Gopal , and Y. Yang . CiteEval for evaluating personalized social web search . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 23 -- 24 , 2009 . Z. Yue, A. Harplale, D. He, J. Grady, Y. Lin, J. Walker, S. Gopal, and Y. Yang. CiteEval for evaluating personalized social web search. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 23--24, 2009."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1670564.1670567","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1670564.1670567","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T12:40:52Z","timestamp":1750250452000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1670564.1670567"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,12,14]]},"references-count":30,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2009,12,14]]}},"alternative-id":["10.1145\/1670564.1670567"],"URL":"https:\/\/doi.org\/10.1145\/1670564.1670567","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2009,12,14]]},"assertion":[{"value":"2009-12-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}