{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T01:15:55Z","timestamp":1772759755159,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2012,8,12]],"date-time":"2012-08-12T00:00:00Z","timestamp":1344729600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2012,8,12]]},"DOI":"10.1145\/2348283.2348400","type":"proceedings-article","created":{"date-parts":[[2012,8,14]],"date-time":"2012-08-14T12:18:56Z","timestamp":1344946736000},"page":"871-880","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":122,"title":["Quality through flow and immersion"],"prefix":"10.1145","author":[{"given":"Carsten","family":"Eickhoff","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher G.","family":"Harris","sequence":"additional","affiliation":[{"name":"The University of Iowa, Iowa City, IA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arjen P.","family":"de Vries","sequence":"additional","affiliation":[{"name":"Centrum Wiskunde & Informatica, Amsterdam, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Padmini","family":"Srinivasan","sequence":"additional","affiliation":[{"name":"The University of Iowa, Iowa City, IA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2012,8,12]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Gambit - Payment Solutions for Virtual Currency.burlhttp:\/\/www.getgambit.com\/ 2011.  Gambit - Payment Solutions for Virtual Currency.burlhttp:\/\/www.getgambit.com\/ 2011."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996910"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935831"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_3_2_1_5_1","volume-title":"LingPipe. Available from World Wide Web: http:\/\/alias-i. com\/lingpipe","author":"Baldwin B.","year":"2003","unstructured":"B. Baldwin and B. Carpenter . LingPipe. Available from World Wide Web: http:\/\/alias-i. com\/lingpipe , 2003 . B. Baldwin and B. Carpenter. LingPipe. Available from World Wide Web: http:\/\/alias-i. com\/lingpipe, 2003."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009000"},{"key":"e_1_3_2_1_7_1","volume-title":"Clueweb09 data set.burlhttp:\/\/boston.lti.cs.cmu.edu","author":"Callan J.","year":"2009","unstructured":"J. Callan , M. Hoy , C. Yoo , and L. Zhao . Clueweb09 data set.burlhttp:\/\/boston.lti.cs.cmu.edu , 2009 . J. Callan, M. Hoy, C. Yoo, and L. Zhao. Clueweb09 data set.burlhttp:\/\/boston.lti.cs.cmu.edu, 2009."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_3_2_1_9_1","first-page":"2009","article-title":"overview","volume":"9","author":"Carterette B.","year":"2009","unstructured":"B. Carterette , V. Pavlu , H. Fang , and E. Kanoulas . Million Query track 2009 overview . In Proceedings of TREC , volume 9 , 2009 . B. Carterette, V. Pavlu, H. Fang, and E. Kanoulas. Million Query track 2009 overview. In Proceedings of TREC, volume 9, 2009.","journal-title":"Proceedings of TREC"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924475.1924481"},{"key":"e_1_3_2_1_11_1","volume-title":"Crowdsourcing - Labor on demand.burlhttp:\/\/crowdflower.com\/","year":"2012","unstructured":"CrowdFlower. Crowdsourcing - Labor on demand.burlhttp:\/\/crowdflower.com\/ , 2012 . CrowdFlower. Crowdsourcing - Labor on demand.burlhttp:\/\/crowdflower.com\/, 2012."},{"key":"e_1_3_2_1_12_1","volume-title":"Flow: The psychology of optimal experience","author":"Csikszentmihalyi M.","year":"1991","unstructured":"M. Csikszentmihalyi . Flow: The psychology of optimal experience . Harper Perennial , 1991 . M. Csikszentmihalyi. Flow: The psychology of optimal experience. Harper Perennial, 1991."},{"key":"e_1_3_2_1_13_1","first-page":"11","volume-title":"WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM)","author":"Eickhoff C.","year":"2011","unstructured":"C. Eickhoff and A. de Vries . How crowdsourcable is your task . In WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM) , pages 11 -- 14 , 2011 . C. Eickhoff and A. de Vries. How crowdsourcable is your task. In WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM), pages 11--14, 2011."},{"key":"e_1_3_2_1_14_1","volume-title":"Increasing Cheat Robustness of Crowdsourcing Tasks. Information RetrievaltextitTo appear","author":"Eickhoff C.","year":"2012","unstructured":"C. Eickhoff and A.P. de Vries . Increasing Cheat Robustness of Crowdsourcing Tasks. Information RetrievaltextitTo appear , 2012 . C. Eickhoff and A.P. de Vries. Increasing Cheat Robustness of Crowdsourcing Tasks. Information RetrievaltextitTo appear, 2012."},{"key":"e_1_3_2_1_15_1","volume-title":"NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk","author":"Grady C.","year":"2010","unstructured":"C. Grady and M. Lease . Crowdsourcing document relevance assessment with Mechanical Turk . In NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk , 2010 . C. Grady and M. Lease. Crowdsourcing document relevance assessment with Mechanical Turk. In NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010."},{"key":"e_1_3_2_1_16_1","first-page":"15","volume-title":"An Examination of Crowdsourcing Incentive Models in Human Resource Tasks. In WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM)","author":"Harris C.","year":"2011","unstructured":"C. Harris . You're Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks. In WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM) , pages 15 -- 18 , 2011 . C. Harris. You're Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks. In WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM), pages 15--18, 2011."},{"key":"e_1_3_2_1_17_1","volume-title":"Cheat-Detection Mechanisms for Crowdsourcing","author":"Hirth M.","year":"2010","unstructured":"M. Hirth , T. Ho\u00dffeld , and P. Tran-Gia . Cheat-Detection Mechanisms for Crowdsourcing . University of W\u00fcrzburg , Tech. Rep, 2010 . M. Hirth, T. Ho\u00dffeld, and P. Tran-Gia. Cheat-Detection Mechanisms for Crowdsourcing. University of W\u00fcrzburg, Tech. Rep, 2010."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1982624.1982625"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996911"},{"key":"e_1_3_2_1_21_1","volume-title":"Worker Types and Personality Traits in Crowdsourcing Relevance Labels","author":"Kazai G.","year":"2011","unstructured":"G. Kazai , J. Kamps , and N. Milic-Frayling . Worker Types and Personality Traits in Crowdsourcing Relevance Labels . 2011 . G. Kazai, J. Kamps, and N. Milic-Frayling. Worker Types and Personality Traits in Crowdsourcing Relevance Labels. 2011."},{"key":"e_1_3_2_1_22_1","volume-title":"SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Kazai G.","year":"2009","unstructured":"G. Kazai and N. Milic-Frayling . On the evaluation of the quality of relevance assessments collected through crowdsourcing . In SIGIR 2009 Workshop on the Future of IR Evaluation , 2009 . G. Kazai and N. Milic-Frayling. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In SIGIR 2009 Workshop on the Future of IR Evaluation, 2009."},{"key":"e_1_3_2_1_23_1","volume-title":"Gender differences and related factors affecting online gaming addiction among taiwanese adolescents. The Journal of nervous and mental disease","author":"Ko C.H.","year":"2005","unstructured":"C.H. Ko , J.Y. Yen , C.C. Chen , S.H. Chen , and C.F. Yen . Gender differences and related factors affecting online gaming addiction among taiwanese adolescents. The Journal of nervous and mental disease , 2005 . C.H. Ko, J.Y. Yen, C.C. Chen, S.H. Chen, and C.F. Yen. Gender differences and related factors affecting online gaming addiction among taiwanese adolescents. The Journal of nervous and mental disease, 2005."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1083-6101.2007.00332.x"},{"key":"e_1_3_2_1_25_1","volume-title":"Overview of the TREC 2011 Crowdsourcing Track (Conference Notebook).","author":"Lease M.","year":"2011","unstructured":"M. Lease and G. Kazai . Overview of the TREC 2011 Crowdsourcing Track (Conference Notebook). 2011 . M. Lease and G. Kazai. Overview of the TREC 2011 Crowdsourcing Track (Conference Notebook). 2011."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2093346.2093356"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1645990"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1998076.1998108"},{"key":"e_1_3_2_1_29_1","volume-title":"Reality is broken: Why games make us better and how they can change the world","author":"McGonigal J.","year":"2011","unstructured":"J. McGonigal . Reality is broken: Why games make us better and how they can change the world . Penguin Pr , 2011 . J. McGonigal. Reality is broken: Why games make us better and how they can change the world. Penguin Pr, 2011."},{"key":"e_1_3_2_1_30_1","volume-title":"The Wisdom of the Probability Sports Crowd.burlhttp:\/\/blog.oddhead.com\/2007\/01\/04\/the-wisdom-of-the-probabilitysports%-crowd\/","author":"Pennock David","year":"2007","unstructured":"David Pennock . The Wisdom of the Probability Sports Crowd.burlhttp:\/\/blog.oddhead.com\/2007\/01\/04\/the-wisdom-of-the-probabilitysports%-crowd\/ , 2007 . David Pennock. The Wisdom of the Probability Sports Crowd.burlhttp:\/\/blog.oddhead.com\/2007\/01\/04\/the-wisdom-of-the-probabilitysports%-crowd\/, 2007."},{"key":"e_1_3_2_1_31_1","volume-title":"Futurelab","author":"Richards C.","year":"2003","unstructured":"C. Richards . Teach the world to twitch: An interview with Marc Prensky, CEO and founder Games2train. com . Futurelab , 2003 . C. Richards. Teach the world to twitch: An interview with Marc Prensky, CEO and founder Games2train. com. Futurelab, 2003."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010057"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"key":"e_1_3_2_1_34_1","volume-title":"American Journal of Physics","author":"Surowiecki J.","year":"2007","unstructured":"J. Surowiecki , M.P. Silverman , The wisdom of crowds . American Journal of Physics , 2007 . J. Surowiecki, M.P. Silverman, et al. The wisdom of crowds. American Journal of Physics, 2007."},{"key":"e_1_3_2_1_35_1","volume-title":"Artificial Artificial Intelligence.burlhttp:\/\/mturk.com","author":"Turk Amazon Mechanical","year":"2012","unstructured":"Amazon Mechanical Turk . Artificial Artificial Intelligence.burlhttp:\/\/mturk.com , 2012 . Amazon Mechanical Turk. Artificial Artificial Intelligence.burlhttp:\/\/mturk.com, 2012."},{"key":"e_1_3_2_1_36_1","volume-title":"The University Carlos III of Madrid at TREC 2011 Crowdsourcing Track: Notebook Paper","author":"Urbano J.","year":"2011","unstructured":"J. Urbano , M. Marrero , D. Mart\u00edn , J. Morato , K. Robles , and J. Llor\u00e9ns . The University Carlos III of Madrid at TREC 2011 Crowdsourcing Track: Notebook Paper . 2011 . J. Urbano, M. Marrero, D. Mart\u00edn, J. Morato, K. Robles, and J. Llor\u00e9ns. The University Carlos III of Madrid at TREC 2011 Crowdsourcing Track: Notebook Paper. 2011."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378704.1378719"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2007.367293"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124784"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124782"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/648264.753539"},{"key":"e_1_3_2_1_43_1","volume-title":"National Institute of Standards, and Technology (US). TREC: Experiment and evaluation in information retrieval","author":"Voorhees E.","year":"2005","unstructured":"E. Voorhees , D.K. Harman , National Institute of Standards, and Technology (US). TREC: Experiment and evaluation in information retrieval . MIT press USA , 2005 . E. Voorhees, D.K. Harman, National Institute of Standards, and Technology (US). TREC: Experiment and evaluation in information retrieval. MIT press USA, 2005."},{"key":"e_1_3_2_1_44_1","volume-title":"NIPS","author":"Welinder P.","year":"2010","unstructured":"P. Welinder , S. Branson , S. Belongie , and P. Perona . The multidimensional wisdom of crowds . In NIPS , 2010 . P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In NIPS, 2010."}],"event":{"name":"SIGIR '12: The 35th International ACM SIGIR conference on research and development in Information Retrieval","location":"Portland Oregon USA","acronym":"SIGIR '12","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2348283.2348400","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2348283.2348400","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:07:49Z","timestamp":1750273669000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2348283.2348400"}},"subtitle":["gamifying crowdsourced relevance assessments"],"short-title":[],"issued":{"date-parts":[[2012,8,12]]},"references-count":44,"alternative-id":["10.1145\/2348283.2348400","10.1145\/2348283"],"URL":"https:\/\/doi.org\/10.1145\/2348283.2348400","relation":{},"subject":[],"published":{"date-parts":[[2012,8,12]]},"assertion":[{"value":"2012-08-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}