{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T07:29:12Z","timestamp":1767857352395,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":22,"publisher":"ACM","license":[{"start":{"date-parts":[[2014,6,1]],"date-time":"2014-06-01T00:00:00Z","timestamp":1401580800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000143","name":"Division of Computing and Communication Foundations","doi-asserted-by":"publisher","award":["CCF-0953516"],"award-info":[{"award-number":["CCF-0953516"]}],"id":[{"id":"10.13039\/100000143","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2014,6]]},"DOI":"10.1145\/2600057.2602865","type":"proceedings-article","created":{"date-parts":[[2014,5,30]],"date-time":"2014-05-30T18:18:31Z","timestamp":1401473911000},"page":"507-524","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Trick or treat"],"prefix":"10.1145","author":[{"given":"Xi Alice","family":"Gao","sequence":"first","affiliation":[{"name":"Harvard University, Cambridge, MA, USA"}]},{"given":"Andrew","family":"Mao","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, MA, USA"}]},{"given":"Yiling","family":"Chen","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, MA, USA"}]},{"given":"Ryan Prescott","family":"Adams","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, MA, USA"}]}],"member":"320","published-online":{"date-parts":[[2014,6]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11129-012-9125-8"},{"key":"e_1_3_2_1_2_1","volume-title":"Nonna\u00efvet\u00e9 among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods","author":"Chandler Jesse","year":"2013","unstructured":"Jesse Chandler , Pam Mueller , and Gabriele Paolacci . 2013. Nonna\u00efvet\u00e9 among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods ( 2013 ). DOI:http:\/\/dx.doi.org\/10.3758\/s13428-013-0365-7 10.3758\/s13428-013-0365-7 Jesse Chandler, Pam Mueller, and Gabriele Paolacci. 2013. Nonna\u00efvet\u00e9 among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods (2013). DOI:http:\/\/dx.doi.org\/10.3758\/s13428-013-0365-7"},{"key":"e_1_3_2_1_3_1","volume-title":"Crowdsourced Judgement Elicitation with Endogenous Proficiency. ACM International World Wide Web Conference (WWW)(2013)","author":"Dasgupta Anirban","year":"2013","unstructured":"Anirban Dasgupta and Arpita Ghosh . 2013 . Crowdsourced Judgement Elicitation with Endogenous Proficiency. ACM International World Wide Web Conference (WWW)(2013) . Anirban Dasgupta and Arpita Ghosh. 2013. Crowdsourced Judgement Elicitation with Endogenous Proficiency. ACM International World Wide Web Conference (WWW)(2013)."},{"key":"e_1_3_2_1_4_1","unstructured":"Xi Alice Gao Yoram Bachrach Peter Key and Thore Graepel. 2012. Quality Expectation-Variance Tradeoffs in Crowdsourcing Contests. In AAAI.  Xi Alice Gao Yoram Bachrach Peter Key and Thore Graepel. 2012. Quality Expectation-Variance Tradeoffs in Crowdsourcing Contests. In AAAI."},{"key":"e_1_3_2_1_5_1","volume-title":"201 The online laboratory: conducting experiments in a real labor market. Experimental Economics 14, 3 (Feb","author":"Horton John J","year":"2011","unstructured":"John J Horton , David G Rand , and Richard J Zeckhauser . 201 The online laboratory: conducting experiments in a real labor market. Experimental Economics 14, 3 (Feb . 2011 ), 399--425. John J Horton, David G Rand, and Richard J Zeckhauser. 201 The online laboratory: conducting experiments in a real labor market. Experimental Economics 14, 3 (Feb. 2011), 399--425."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611430953"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622716.1622723"},{"key":"e_1_3_2_1_8_1","volume-title":"Proceedings of the 4th Workshop on Human Computation (HCOMP'12)","author":"Mao Andrew","year":"2012","unstructured":"Andrew Mao , Yiling Chen , Krzysztof Z. Gajos , David Parkes , Ariel D. Procaccia , and Haoqi Zhang . 2012 . TurkServer: Enabling Synchronous and Longitudinal Online Experiments . In Proceedings of the 4th Workshop on Human Computation (HCOMP'12) . Andrew Mao, Yiling Chen, Krzysztof Z. Gajos, David Parkes, Ariel D. Procaccia, and Haoqi Zhang. 2012. TurkServer: Enabling Synchronous and Longitudinal Online Experiments. In Proceedings of the 4th Workshop on Human Computation (HCOMP'12)."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1050.0379"},{"key":"e_1_3_2_1_12_1","first-page":"257","article-title":"An algorithm that finds truth even if most people are wrong. (2006). Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition","volume":"77","author":"Prelec Drazen","year":"2006","unstructured":"Drazen Prelec and HS Seung . 2006 . An algorithm that finds truth even if most people are wrong. (2006). Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition . Proc. IEEE 77 , 2 (1989), 257 -- 286 . Drazen Prelec and HS Seung. 2006. An algorithm that finds truth even if most people are wrong. (2006). Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2 (1989), 257--286.","journal-title":"Proc. IEEE"},{"key":"e_1_3_2_1_13_1","volume-title":"Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI'13)","author":"Radanovic Goran","year":"2013","unstructured":"Goran Radanovic and Boi Faltings . 2013 . A Robust Bayesian Truth Serum for Nonbinary Signals . In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI'13) . 833--839. Goran Radanovic and Boi Faltings. 2013. A Robust Bayesian Truth Serum for Nonbinary Signals. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI'13). 833--839."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jtbi.2011.03.004"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176344136"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1257\/aer.98.3.938"},{"key":"e_1_3_2_1_17_1","volume-title":"A hidden Markov model for the detection of pure and mixed strategy play in games. Available at SSRN","author":"Shachat Jason","year":"2012","unstructured":"Jason Shachat , J Swarthout , and Lijia Wei . 2012. A hidden Markov model for the detection of pure and mixed strategy play in games. Available at SSRN ( 2012 ). Jason Shachat, J Swarthout, and Lijia Wei. 2012. A hidden Markov model for the detection of pure and mixed strategy play in games. Available at SSRN (2012)."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1967.1054010"},{"key":"e_1_3_2_1_20_1","volume-title":"Proceedings of the 3rd Workshop on Social Computing and User Generated Content (SC13)","author":"Waggoner Bo","year":"2013","unstructured":"Bo Waggoner and Yiling Chen . 2013 . Information Elicitation Sans Verification . In Proceedings of the 3rd Workshop on Social Computing and User Generated Content (SC13) . Bo Waggoner and Yiling Chen. 2013. Information Elicitation Sans Verification. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content (SC13)."},{"key":"e_1_3_2_1_21_1","volume-title":"Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP'13)","author":"Witkowski Jens","unstructured":"Jens Witkowski , Yoram Bachrach , Peter Key , and David C. Parkes . 2013. Dwelling on the Negative: Incentivizing Effort in Peer Prediction . In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP'13) . Jens Witkowski, Yoram Bachrach, Peter Key, and David C. Parkes. 2013. Dwelling on the Negative: Incentivizing Effort in Peer Prediction. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP'13)."},{"key":"e_1_3_2_1_22_1","volume-title":"Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI '12)","author":"Witkowski Jens","unstructured":"Jens Witkowski and David C. Parkes . 2012a. A Robust Bayesian Truth Serum for Small Populations . In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI '12) . Jens Witkowski and David C. Parkes. 2012a. A Robust Bayesian Truth Serum for Small Populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI '12)."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2229012.2229085"}],"event":{"name":"EC '14: ACM Conference on Economics and Computation","location":"Palo Alto California USA","acronym":"EC '14","sponsor":["SIGecom Special Interest Group on Economics and Computation"]},"container-title":["Proceedings of the fifteenth ACM conference on Economics and computation"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2600057.2602865","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2600057.2602865","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:01:21Z","timestamp":1750230081000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2600057.2602865"}},"subtitle":["putting peer prediction to the test"],"short-title":[],"issued":{"date-parts":[[2014,6]]},"references-count":22,"alternative-id":["10.1145\/2600057.2602865","10.1145\/2600057"],"URL":"https:\/\/doi.org\/10.1145\/2600057.2602865","relation":{},"subject":[],"published":{"date-parts":[[2014,6]]},"assertion":[{"value":"2014-06-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}