{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T12:02:27Z","timestamp":1790337747974,"version":"4.1.0"},"publisher-location":"New York, NY, USA","reference-count":21,"publisher":"ACM","license":[{"start":{"date-parts":[[2012,10,29]],"date-time":"2012-10-29T00:00:00Z","timestamp":1351468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2012,10,29]]},"DOI":"10.1145\/2396761.2398697","type":"proceedings-article","created":{"date-parts":[[2012,11,15]],"date-time":"2012-11-15T16:38:00Z","timestamp":1352997480000},"page":"2583-2586","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":71,"title":["The face of quality in crowdsourcing relevance labels"],"prefix":"10.1145","author":[{"given":"Gabriella","family":"Kazai","sequence":"first","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaap","family":"Kamps","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natasa","family":"Milic-Frayling","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2012,10,29]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_3_2_1_3_1","volume-title":"Working paper","author":"Chandler D.","year":"2010","unstructured":"D. Chandler and A. Kapelner . Breaking monotony with meaning: Motivation in crowdsourcing markets. Technical report , Working paper , University of Chicago , 2010 . D. Chandler and A. Kapelner. Breaking monotony with meaning: Motivation in crowdsourcing markets. Technical report, Working paper, University of Chicago, 2010."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753688"},{"key":"e_1_3_2_1_5_1","first-page":"172","volume-title":"Crowdsourcing document relevance assessment with mechanical turk. In phProc. of CSLDAMT '10","author":"Grady C.","year":"2010","unstructured":"C. Grady and M. Lease . Crowdsourcing document relevance assessment with mechanical turk. In phProc. of CSLDAMT '10 , pages 172 -- 179 , 2010 . C. Grady and M. Lease. Crowdsourcing document relevance assessment with mechanical turk. In phProc. of CSLDAMT '10, pages 172--179, 2010."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869094"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_3_2_1_8_1","first-page":"114","volume-title":"Paradigm shift to the integrative big-five trait taxonomy: History, measurement, and conceptual issues. In phHandbook of personality: Theory and research","author":"John O. P.","year":"2008","unstructured":"O. P. John , L. P. Naumann , and C. J. Soto . Paradigm shift to the integrative big-five trait taxonomy: History, measurement, and conceptual issues. In phHandbook of personality: Theory and research , chapter 4, pages 114 -- 212 . Guilford Press , New York NY , 2008 . O. P. John, L. P. Naumann, and C. J. Soto. Paradigm shift to the integrative big-five trait taxonomy: History, measurement, and conceptual issues. In phHandbook of personality: Theory and research, chapter 4, pages 114--212. Guilford Press, New York NY, 2008."},{"key":"e_1_3_2_1_9_1","first-page":"89","volume-title":"Overview of the INEX 2010 book track: At the mercy of crowdsourcing. In phINEX 2010 Workshop Pre-proceedings","author":"Kazai G.","year":"2010","unstructured":"G. Kazai , M. Koolen , A. Doucet , and M. Landoni . Overview of the INEX 2010 book track: At the mercy of crowdsourcing. In phINEX 2010 Workshop Pre-proceedings , pages 89 -- 99 , 2010 . G. Kazai, M. Koolen, A. Doucet, and M. Landoni. Overview of the INEX 2010 book track: At the mercy of crowdsourcing. In phINEX 2010 Workshop Pre-proceedings, pages 89--99, 2010."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009947"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_2_1_12_1","first-page":"21","volume-title":"Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In phSIGIR 2010 Workshop on Crowdsourcing for Search Evaluation","author":"Le J.","year":"2010","unstructured":"J. Le , A. Edmonds , V. Hester , and L. Biewald . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In phSIGIR 2010 Workshop on Crowdsourcing for Search Evaluation , pages 21 -- 26 , 2010 . J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In phSIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 21--26, 2010."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jrp.2006.02.001"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_3_2_1_18_1","first-page":"254","volume-title":"Cheap and fast--but is it good?: evaluating non-expert annotations for natural language tasks. In phProc. of the EMNLP '08","author":"Snow R.","year":"2008","unstructured":"R. Snow , B. O'Connor , D. Jurafsky , and A. Y. Ng . Cheap and fast--but is it good?: evaluating non-expert annotations for natural language tasks. In phProc. of the EMNLP '08 , pages 254 -- 263 , 2008 . R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast--but is it good?: evaluating non-expert annotations for natural language tasks. In phProc. of the EMNLP '08, pages 254--263, 2008."},{"key":"e_1_3_2_1_19_1","volume-title":"phTREC: Experimentation and Evaluation in Information Retrieval","author":"Voorhees E. M.","year":"2005","unstructured":"E. M. Voorhees and D. K. Harman , editors . phTREC: Experimentation and Evaluation in Information Retrieval . MIT Press , 2005 . E. M. Voorhees and D. K. Harman, editors. phTREC: Experimentation and Evaluation in Information Retrieval. MIT Press, 2005."},{"key":"e_1_3_2_1_20_1","volume-title":"How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In phProc. of the ACM SIGIR Workshop on Crowdsourcing for Information Retrieval","author":"Vuurens J.","year":"2011","unstructured":"J. Vuurens , A. de Vries , and C. Eickhoff . How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In phProc. of the ACM SIGIR Workshop on Crowdsourcing for Information Retrieval , 2011 . ACM. J. Vuurens, A. de Vries, and C. Eickhoff. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In phProc. of the ACM SIGIR Workshop on Crowdsourcing for Information Retrieval, 2011. ACM."},{"key":"e_1_3_2_1_21_1","first-page":"17","volume-title":"An analysis of assessor behavior in crowdsourced preference judgments. In phSIGIR 2010 Workshop on Crowdsourcing for Search Evaluation","author":"Zhu D.","year":"2010","unstructured":"D. Zhu and B. Carterette . An analysis of assessor behavior in crowdsourced preference judgments. In phSIGIR 2010 Workshop on Crowdsourcing for Search Evaluation , pages 17 -- 20 , 2010 . D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In phSIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 17--20, 2010."}],"event":{"name":"CIKM'12: 21st ACM International Conference on Information and Knowledge Management","location":"Maui Hawaii USA","acronym":"CIKM'12","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 21st ACM international conference on Information and knowledge management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2396761.2398697","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2396761.2398697","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:34:42Z","timestamp":1750239282000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2396761.2398697"}},"subtitle":["demographics, personality and labeling accuracy"],"short-title":[],"issued":{"date-parts":[[2012,10,29]]},"references-count":21,"alternative-id":["10.1145\/2396761.2398697","10.1145\/2396761"],"URL":"https:\/\/doi.org\/10.1145\/2396761.2398697","relation":{},"subject":[],"published":{"date-parts":[[2012,10,29]]},"assertion":[{"value":"2012-10-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}