{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,16]],"date-time":"2025-12-16T12:14:54Z","timestamp":1765887294694,"version":"3.41.0"},"reference-count":12,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2011,1,3]],"date-time":"2011-01-03T00:00:00Z","timestamp":1294012800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2011,1,3]]},"abstract":"<jats:p>The Crowdsourcing for Search Evaluation Workshop (CSE 2010) was held on July 23, 2010 in Geneva, Switzerland, in conjunction with the 33rd Annual ACM SIGIR Conference. The workshop addressed the latest advances in theory and empirical methods in crowdsourcing for search evaluation, as well as novel applications of crowdsourcing for evaluating search systems. Three invited talks were presented, along with seven refereed papers. Proceedings from the workshop, along with presentation slides, have been made available online.<\/jats:p>","DOI":"10.1145\/1924475.1924481","type":"journal-article","created":{"date-parts":[[2011,1,24]],"date-time":"2011-01-24T14:58:13Z","timestamp":1295881093000},"page":"17-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["Crowdsourcing for search evaluation"],"prefix":"10.1145","volume":"44","author":[{"given":"Vitor R.","family":"Carvalho","sequence":"first","affiliation":[{"name":"Microsoft Corporation, Redmond, WA"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"School of Information, University of Texas at Austin, Austin, TX"}]},{"given":"Emine","family":"Yilmaz","sequence":"additional","affiliation":[{"name":"Microsoft Research Cambridge, Cambridge, UK"}]}],"member":"320","published-online":{"date-parts":[[2011,1,3]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"39","volume-title":"Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010","author":"Alonso O.","year":"2010","unstructured":"O. Alonso , C. Carson , D. Gerster , X. Ji , and S. U. Nabar . Detecting uninteresting content in text streams . In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010 ), pages 39 -- 42 , Geneva, Switzerland , July 2010 . O. Alonso, C. Carson, D. Gerster, X. Ji, and S. U. Nabar. Detecting uninteresting content in text streams. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 39--42, Geneva, Switzerland, July 2010."},{"key":"e_1_2_1_2_1","first-page":"15","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Alonso O.","year":"2009","unstructured":"O. Alonso and S. Mizzaro . Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 15 -- 16 , 2009 . O. Alonso and S. Mizzaro. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 15--16, 2009."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148263"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/275537.275544"},{"key":"e_1_2_1_6_1","first-page":"27","volume-title":"Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010","author":"Feild H.","year":"2010","unstructured":"H. Feild , R. Jones , R. C. Miller , R. Nayak , E. F. Churchill , and E. Velipasaoglu . Logging the search self-efficacy of amazon mechanical turkers . In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010 ), pages 27 -- 30 , Geneva, Switzerland , July 2010 . H. Feild, R. Jones, R. C. Miller, R. Nayak, E. F. Churchill, and E. Velipasaoglu. Logging the search self-efficacy of amazon mechanical turkers. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 27--30, Geneva, Switzerland, July 2010."},{"key":"e_1_2_1_7_1","first-page":"17","volume-title":"Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010","author":"Le J.","year":"2010","unstructured":"J. Le , A. Edmonds , V. Hester , and L. Biewald . Ensuring quality in crowdsourced search relevance evaluation . In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010 ), pages 17 -- 20 , Geneva, Switzerland , July 2010 . J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 17--20, Geneva, Switzerland, July 2010."},{"key":"e_1_2_1_8_1","first-page":"31","volume-title":"Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010","author":"McCreadie R. M. C.","year":"2010","unstructured":"R. M. C. McCreadie , C. Macdonald , and I. Ounis . Crowdsourcing a news query classification dataset . In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010 ), pages 31 -- 38 , Geneva, Switzerland , July 2010 . R. M. C. McCreadie, C. Macdonald, and I. Ounis. Crowdsourcing a news query classification dataset. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 31--38, Geneva, Switzerland, July 2010."},{"key":"e_1_2_1_9_1","first-page":"4","volume-title":"Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010","author":"Soleymani M.","year":"2010","unstructured":"M. Soleymani and M. Larson . Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus . In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010 ), pages 4 -- 8 , Geneva, Switzerland , July 2010 . M. Soleymani and M. Larson. Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 4--8, Geneva, Switzerland, July 2010."},{"key":"e_1_2_1_10_1","first-page":"9","volume-title":"Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010","author":"Urbano J.","year":"2010","unstructured":"J. Urbano , J. Morato , M. Marrero , and D. Martin . Crowdsourcing preference judgments for evaluation of music similarity tasks . In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010 ), pages 9 -- 16 , Geneva, Switzerland , July 2010 . J. Urbano, J. Morato, M. Marrero, and D. Martin. Crowdsourcing preference judgments for evaluation of music similarity tasks. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 9--16, Geneva, Switzerland, July 2010."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390437"},{"key":"e_1_2_1_12_1","first-page":"21","volume-title":"Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010","author":"Zhu D.","year":"2010","unstructured":"D. Zhu and B. Carterette . An analysis of assessor behavior in crowdsourced preference judgments . In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010 ), pages 21 -- 26 , Geneva, Switzerland , July 2010 . D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 21--26, Geneva, Switzerland, July 2010."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1924475.1924481","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1924475.1924481","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T10:59:53Z","timestamp":1750244393000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1924475.1924481"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,1,3]]},"references-count":12,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2011,1,3]]}},"alternative-id":["10.1145\/1924475.1924481"],"URL":"https:\/\/doi.org\/10.1145\/1924475.1924481","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2011,1,3]]},"assertion":[{"value":"2011-01-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}