{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T17:09:01Z","timestamp":1791047341967,"version":"4.1.0"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2011,7,24]],"date-time":"2011-07-24T00:00:00Z","timestamp":1311465600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2011,7,24]]},"DOI":"10.1145\/2009916.2009947","type":"proceedings-article","created":{"date-parts":[[2011,7,26]],"date-time":"2011-07-26T14:09:37Z","timestamp":1311689377000},"page":"205-214","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":91,"title":["Crowdsourcing for book search evaluation"],"prefix":"10.1145","author":[{"given":"Gabriella","family":"Kazai","sequence":"first","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaap","family":"Kamps","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marijn","family":"Koolen","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natasa","family":"Milic-Frayling","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2011,7,24]]},"reference":[{"key":"e_1_3_2_1_1_1","series-title":"LNCS","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1007\/978-3-642-20161-5_16","volume-title":"Advances in Information Retrieval -- 33rd European Conference on IR Research (ECIR","author":"Alonso O.","year":"2011","unstructured":"O. Alonso and R. A. Baeza-Yates . Design and implementation of relevance assessments using crowdsourcing . In Advances in Information Retrieval -- 33rd European Conference on IR Research (ECIR 2011 ), volume 6611 of LNCS , pages 153 -- 164 . Springer , 2011. O. Alonso and R. A. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. In Advances in Information Retrieval -- 33rd European Conference on IR Research (ECIR 2011), volume 6611 of LNCS, pages 153--164. Springer, 2011."},{"key":"e_1_3_2_1_2_1","first-page":"15","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Alonso O.","year":"2009","unstructured":"O. Alonso and S. Mizzaro . Can we get rid of TREC assessors? using Mechanical Turk for relevance assessment . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , pages 15 -- 16 , 2009 . O. Alonso and S. Mizzaro. Can we get rid of TREC assessors? using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 15--16, 2009."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb050097"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291009"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753688"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866723"},{"key":"e_1_3_2_1_8_1","volume-title":"Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business","author":"Howe J.","year":"2008","unstructured":"J. Howe . Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business . Crown Publishing Group , 2008 . J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, 2008."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1670564.1670567"},{"key":"e_1_3_2_1_10_1","volume-title":"The World's First Conference on the Future of Distributed Work (CrowdConf2010)","author":"Kapelner A.","year":"2010","unstructured":"A. Kapelner and D. Chandler . Preventing satisficing in online surveys: A 'kapcha' to ensure higher quality data . In The World's First Conference on the Future of Distributed Work (CrowdConf2010) , 2010 . A. Kapelner and D. Chandler. Preventing satisficing in online surveys: A 'kapcha' to ensure higher quality data. In The World's First Conference on the Future of Distributed Work (CrowdConf2010), 2010."},{"key":"e_1_3_2_1_11_1","series-title":"LNCS","first-page":"165","volume-title":"search of quality in crowdsourcing for search engine evaluation. In Advances in Information Retrieval -- 33rd European Conference on IR Research (ECIR","author":"Kazai G.","year":"2011","unstructured":"G. Kazai . In search of quality in crowdsourcing for search engine evaluation. In Advances in Information Retrieval -- 33rd European Conference on IR Research (ECIR 2011 ), volume 6611 of LNCS , pages 165 -- 176 . Springer , 2011. G. Kazai. In search of quality in crowdsourcing for search engine evaluation. In Advances in Information Retrieval -- 33rd European Conference on IR Research (ECIR 2011), volume 6611 of LNCS, pages 165--176. Springer, 2011."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572019"},{"key":"e_1_3_2_1_13_1","volume-title":"Comparative Evaluation of Focused Retrieval : 9th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX","author":"Kazai G.","year":"2010","unstructured":"G. Kazai , M. Koolen , J. Kamps , A. Doucet , and M. Landoni . Overview of the INEX 2010 book track: Scaling up the evaluation using crowdsourcing . In Comparative Evaluation of Focused Retrieval : 9th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2010 ), LNCS. Springer , 2011. G. Kazai, M. Koolen, J. Kamps, A. Doucet, and M. Landoni. Overview of the INEX 2010 book track: Scaling up the evaluation using crowdsourcing. In Comparative Evaluation of Focused Retrieval : 9th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2010), LNCS. Springer, 2011."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_2_1_15_1","first-page":"21","volume-title":"SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation","author":"Le J.","year":"2010","unstructured":"J. Le , A. Edmonds , V. Hester , and L. Biewald . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation , pages 21 -- 26 , 2010 . J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 21--26, 2010."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_3_2_1_18_1","volume-title":"Questionnaire Design, Interviewing and Attitude Measurement","author":"Oppenheim A. N.","year":"1992","unstructured":"A. N. Oppenheim . Questionnaire Design, Interviewing and Attitude Measurement . Pinter Publishers , London , 1992 . A. N. Oppenheim. Questionnaire Design, Interviewing and Attitude Measurement. Pinter Publishers, London, 1992."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031171.1031246"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458092"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_3_2_1_27_1","volume-title":"TREC: Experimentation and Evaluation in Information Retrieval","author":"Voorhees E. M.","year":"2005","unstructured":"E. M. Voorhees and D. K. Harman , editors . TREC: Experimentation and Evaluation in Information Retrieval . MIT Press , 2005 . E. M. Voorhees and D. K. Harman, editors. TREC: Experimentation and Evaluation in Information Retrieval. MIT Press, 2005."},{"key":"e_1_3_2_1_28_1","first-page":"2424","volume-title":"Advances in Neural Information Processing Systems 23","author":"Welinder P.","year":"2010","unstructured":"P. Welinder , S. Branson , S. Belongie , and P. Perona . The multidimensional wisdom of crowds . In Advances in Neural Information Processing Systems 23 , pages 2424 -- 2432 , 2010 . P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems 23, pages 2424--2432, 2010."},{"key":"e_1_3_2_1_29_1","volume-title":"Proceedings of the 2009 Neural Information Processing Systems (NIPS) Conference","author":"Whitehill J.","year":"2009","unstructured":"J. Whitehill , P. Ruvolo , T. Wu , J. Bergsma , and J. Movellan . Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Proceedings of the 2009 Neural Information Processing Systems (NIPS) Conference , 2009 . J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 2009 Neural Information Processing Systems (NIPS) Conference, 2009."},{"key":"e_1_3_2_1_30_1","first-page":"17","volume-title":"SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation","author":"Zhu D.","year":"2010","unstructured":"D. Zhu and B. Carterette . An analysis of assessor behavior in crowdsourced preference judgments . In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation , pages 17 -- 20 , 2010 . D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 17--20, 2010."}],"event":{"name":"SIGIR '11: The 34th International ACM SIGIR conference on research and development in Information Retrieval","location":"Beijing China","acronym":"SIGIR '11","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2009916.2009947","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2009916.2009947","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T11:22:30Z","timestamp":1750245750000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2009916.2009947"}},"subtitle":["impact of hit design on comparative system ranking"],"short-title":[],"issued":{"date-parts":[[2011,7,24]]},"references-count":30,"alternative-id":["10.1145\/2009916.2009947","10.1145\/2009916"],"URL":"https:\/\/doi.org\/10.1145\/2009916.2009947","relation":{},"subject":[],"published":{"date-parts":[[2011,7,24]]},"assertion":[{"value":"2011-07-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}