{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T19:50:49Z","timestamp":1779306649608,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":42,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,4,30]],"date-time":"2023-04-30T00:00:00Z","timestamp":1682812800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,4,30]]},"DOI":"10.1145\/3578503.3583622","type":"proceedings-article","created":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T23:03:17Z","timestamp":1682550197000},"page":"335-345","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Who Broke Amazon Mechanical Turk?"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-2635-2915","authenticated-orcid":false,"given":"Catherine C.","family":"Marshall","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, Texas A and M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-3472-4371","authenticated-orcid":false,"given":"Partha S.R.","family":"Goguladinne","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Texas A&M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9768-5729","authenticated-orcid":false,"given":"Mudit","family":"Maheshwari","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Texas A&M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9797-4224","authenticated-orcid":false,"given":"Apoorva","family":"Sathe","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Texas A&M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1469-9835","authenticated-orcid":false,"given":"Frank M.","family":"Shipman","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Texas A&M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,4,30]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Proc. 2019 Meeting of Midwest Political Science Assn, ret.","author":"Ahler Douglas J.","year":"2019","unstructured":"Douglas J. Ahler, Carolyn E. Rous, Gaurav Sood. 2019. The Micro-Task Market for Lemons: Data Quality on Amazon's Mechanical Turk. Proc. 2019 Meeting of Midwest Political Science Assn, ret. July 22, 2019."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2756406.2757741"},{"key":"e_1_3_2_2_3_1","volume-title":"MTurk Worker Quality and Identity. Blog post dated","author":"Turk Amazon Mechanical","year":"2019","unstructured":"[3] Amazon Mechanical Turk. 2019. MTurk Worker Quality and Identity. Blog post dated 25 March 2019. https:\/\/blog.mturk.com\/mturk-worker-identity-and-task-quality-d3be46d83d0d"},{"key":"e_1_3_2_2_4_1","volume-title":"Worker identity and Requester tools to manage task quality. Blog post dated","author":"Turk Amazon Mechanical","year":"2020","unstructured":"Amazon Mechanical Turk. 2020. Important updates on MTurk marketplace integrity, Worker identity and Requester tools to manage task quality. Blog post dated 20 March 2020."},{"key":"e_1_3_2_2_5_1","first-page":"1092","volume-title":"Proc. CHI EA '17","author":"Bentley F.","unstructured":"F. Bentley, N. Daskalova, and B. White. 2017 Comparing the Reliability of Amazon Mechanical Turk and Survey Monkey to Traditional Market Research Surveys. Proc. CHI EA '17. ACM, New York, NY, USA, 1092-1099."},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-018-1035-6"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13332"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550619875149"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","unstructured":"S.A. Dennis B.M. Goodson and C. Pearson. 2018. MTurk workers\u2019 use of low-cost \u201cvirtual private servers\u201d to circumvent screening methods. Research note. https:\/\/doi.org\/10.2139\/ssrn.3233954.","DOI":"10.2139\/ssrn.3233954"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_3_2_2_11_1","volume-title":"Proc. CHI '10","author":"Downs Julie S.","year":"2010","unstructured":"Julie S. Downs, Mandy B. Holbrook, Steve Sheng, and Lorrie Faith Cranor. 2010. Are your participants gaming the system? screening mechanical turk workers. In Proc. CHI '10. ACM, New York, NY, USA, 2399\u20132402."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3537674.3555783"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130914"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819942"},{"key":"e_1_3_2_2_17_1","volume-title":"US Gov't","author":"Democratic People's Guidance","year":"2022","unstructured":"Guidance on the Democratic People's Republic of Korea Information Technology Workers. 2022. US Gov't 16 May 2022 https:\/\/home.treasury.gov\/system\/files\/126\/20220516_dprk_it_worker_advisory.pdf."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/0013164415627349"},{"issue":"1","key":"e_1_3_2_2_19_1","first-page":"1","article-title":"Comparing Internet Experiences and Prosociality in Amazon Mechanical Turk and Population-Based Survey Samples","volume":"6","author":"Hargittai Eszter","year":"2020","unstructured":"Eszter Hargittai and Aaron Shaw. 2020. Comparing Internet Experiences and Prosociality in Amazon Mechanical Turk and Population-Based Survey Samples. Socius 6 (1), 1\u201311.","journal-title":"Socius"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-015-0578-z"},{"key":"e_1_3_2_2_21_1","volume-title":"ITWorld","author":"M.","year":"2009","unstructured":"Jakobsson, M. 2009. Experimenting on Mechanical Turk: 5 How Tos. ITWorld, September 3, 2009."},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580645"},{"key":"e_1_3_2_2_23_1","first-page":"70","volume-title":"The Sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP","author":"Kaplan Toni","year":"2018","unstructured":"Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey P. Bigham. 2018, Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers. In The Sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018). AAAI Press, pp 70-78."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1017\/psrm.2020.6"},{"key":"e_1_3_2_2_25_1","first-page":"453","volume-title":"Proc. CHI\u201908","author":"B.","unstructured":"Kittur, A., Chi, E., Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc. CHI\u201908, 453-456."},{"key":"e_1_3_2_2_26_1","volume-title":"Shipman","author":"Marshall Catherine C.","year":"2013","unstructured":"Catherine C. Marshall and Frank M. Shipman. 2013. Experiences Surveying the Crowd: Reflections on Methods, Participation, and Reliability, Proc. WebSci\u201913, 234-243."},{"key":"e_1_3_2_2_27_1","volume-title":"Shipman","author":"Marshall Catherine C.","year":"2015","unstructured":"Catherine C. Marshall and Frank M. Shipman. 2015. Exploring the Ownership and Persistent Value of Facebook Content. Proc. CSCW\u201915, ACM Press, NY."},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2996181"},{"key":"e_1_3_2_2_29_1","volume-title":"HCOMP '09","author":"Mason Winter","unstructured":"Winter Mason and Duncan J. Watts. 2009. Financial incentives and the \"performance of crowds\". HCOMP '09. ACM, New York, NY, USA, 77\u201385."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0028085"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-021-01694-3"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-013-0434-y"},{"key":"e_1_3_2_2_33_1","volume-title":"Mazurek","author":"Redmiles Elissa M.","year":"2019","unstructured":"Elissa M. Redmiles, Sean Kross, and Michelle L. Mazurek. 2019. How Well Do My Results Generalize? Comparing Security and Privacy Survey Results from MTurk, Web, and Telephone Samples. Proc. IEEE Security and Privacy"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_3_2_2_35_1","volume-title":"Fraudulent responses on Amazon Mechanical Turk: A Fresh Cautionary Tale. Blog post published","author":"Ryan Timothy J.","year":"2020","unstructured":"Timothy J. Ryan. 2020. Fraudulent responses on Amazon Mechanical Turk: A Fresh Cautionary Tale. Blog post published 22 December 2020 on https:\/\/timryan.web.unc.edu."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300512"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-90238-4_12"},{"key":"e_1_3_2_2_38_1","volume-title":"Proc CHI","author":"Frank","year":"2020","unstructured":"Frank M. Shipman and Catherine C. Marshall. 2020. Ownership, Privacy, and Control in the Wake of Cambridge Analytica. Proc CHI 2020. ACM, New York."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.20982\/tqmp.16.5.p472"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11135-005-8081-8"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359126"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","unstructured":"Bingbing Zhang and Sherice Gearhart. 2020. Collecting Online Survey Data: A Comparison of Data Quality among a Commercial Panel and MTurk. Survey Practice 13 (1). https:\/\/doi.org\/10.29115\/SP-2020-0015.","DOI":"10.29115\/SP-2020-0015"}],"event":{"name":"WebSci '23: 15th ACM Web Science Conference 2023","location":"Austin TX USA","acronym":"WebSci '23","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"]},"container-title":["Proceedings of the 15th ACM Web Science Conference 2023"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578503.3583622","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3578503.3583622","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:08:37Z","timestamp":1750183717000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578503.3583622"}},"subtitle":["An Analysis of Crowdsourcing Data Quality over Time"],"short-title":[],"issued":{"date-parts":[[2023,4,30]]},"references-count":42,"alternative-id":["10.1145\/3578503.3583622","10.1145\/3578503"],"URL":"https:\/\/doi.org\/10.1145\/3578503.3583622","relation":{},"subject":[],"published":{"date-parts":[[2023,4,30]]},"assertion":[{"value":"2023-04-30","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}