{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T18:24:52Z","timestamp":1785003892725,"version":"3.55.0"},"reference-count":53,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T00:00:00Z","timestamp":1570665600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Social Psychological and Personality Science"],"published-print":{"date-parts":[[2020,5]]},"abstract":"<jats:p>Amazon\u2019s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.<\/jats:p>","DOI":"10.1177\/1948550619875149","type":"journal-article","created":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T06:35:50Z","timestamp":1570689350000},"page":"464-473","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":897,"title":["An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results"],"prefix":"10.1177","volume":"11","author":[{"given":"Michael","family":"Chmielewski","sequence":"first","affiliation":[{"name":"Department of Psychology, Southern Methodist University, Dallas, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah C.","family":"Kucker","sequence":"additional","affiliation":[{"name":"Department of Psychology, Oklahoma State University, Stillwater, OK, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2019,10,10]]},"reference":[{"key":"bibr1-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1080\/13645579.2018.1563966"},{"key":"bibr2-1948550619875149","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-012-0265-2"},{"key":"bibr3-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1016\/0165-0327(96)00060-2"},{"key":"bibr4-1948550619875149","unstructured":"Bai H. (2018). Evidence that a large amount of low quality responses on MTurk can be detected with repeated GPS coordinates. Retrieved February 4, 2019, from Sights + Sounds website: http:\/\/www.maxhuibai.com\/1\/post\/2018\/08\/evidence-that-responses-from-repeating-gps-are-random.html"},{"key":"bibr5-1948550619875149","first-page":"11","volume":"49","author":"Barger P.","year":"2011","journal-title":"The Industrial-Organizational Psychologist"},{"key":"bibr6-1948550619875149","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0081-0"},{"key":"bibr80-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1111\/ajps.12081"},{"key":"bibr7-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"bibr8-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/1745691617706516"},{"key":"bibr9-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2013.05.009"},{"key":"bibr10-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/abn0000069"},{"key":"bibr11-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/pas0000322"},{"key":"bibr12-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/0021-843X.100.3.316"},{"key":"bibr13-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/pas0000706"},{"key":"bibr14-1948550619875149","doi-asserted-by":"crossref","unstructured":"Dennis S. A., Goodson B. M., Pearson C. (2018). MTurk workers\u2019 use of low-cost \u201cvirtual private servers\u201d to circumvent screening methods: A research note (SSRN scholarly paper no. ID 3233954). Retrieved from Social Science Research Network website: https:\/\/papers.ssrn.com\/abstract=3233954","DOI":"10.2139\/ssrn.3233954"},{"key":"bibr15-1948550619875149","unstructured":"Dreyfuss E. (2018, August 17). A bot panic hits Amazon\u2019s Mechanical Turk. Wired. Retrieved from https:\/\/www.wired.com\/story\/amazon-mechanical-turk-bot-panic\/"},{"key":"bibr16-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1017\/S1930297500001030"},{"key":"bibr17-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/1948550617693063"},{"key":"bibr18-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1002\/bdm.1753"},{"key":"bibr19-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/2158244015584617"},{"key":"bibr20-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1007\/s10683-011-9273-9"},{"key":"bibr21-1948550619875149","volume-title":"The Big Five Inventory\u2014versions 4a and 54","author":"John O. P.","year":"1991"},{"key":"bibr22-1948550619875149","first-page":"102","volume-title":"Handbook of personality: Theory and research","author":"John O. P.","year":"1999","edition":"2"},{"key":"bibr23-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1080\/00913367.2016.1269304"},{"key":"bibr24-1948550619875149","doi-asserted-by":"crossref","unstructured":"Kennedy R., Clifford S., Burleigh T., Jewell R., Waggoner P. (2018). The shape of and solutions to the MTurk quality crisis (SSRN Scholarly Paper No. ID 3272468). Retrieved from Social Science Research Network website: https:\/\/papers.ssrn.com\/abstract=3272468","DOI":"10.2139\/ssrn.3272468"},{"key":"bibr25-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"bibr26-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/a0020327"},{"key":"bibr27-1948550619875149","volume-title":"Personality and psychopathology","author":"Krueger R. F.","year":"2006"},{"key":"bibr81-1948550619875149","unstructured":"Kucker S. C., Zimmerman C., Chmielewski M. (submitted for publication). Taking parent personality and child temperament into account in child language development."},{"key":"bibr82-1948550619875149","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-014-0483-x"},{"key":"bibr28-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1007\/s10862-005-5384-y"},{"key":"bibr29-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2010.5494979"},{"key":"bibr30-1948550619875149","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"bibr31-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"bibr32-1948550619875149","volume-title":"Crowdsourcing a news query classification dataset (31\u201338)","author":"McCreadie R. M.","year":"2010"},{"key":"bibr33-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2009.03.009"},{"key":"bibr34-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/0963721414531598"},{"key":"bibr35-1948550619875149","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-013-0434-y"},{"key":"bibr36-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/2515245919838479"},{"key":"bibr37-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/a0033323"},{"key":"bibr38-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/2167702612469015"},{"key":"bibr39-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1080\/03637751.2017.1342043"},{"key":"bibr40-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.94.4.718"},{"key":"bibr41-1948550619875149","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1017\/S1930297500005611","volume":"10","author":"Stewart N.","year":"2015","journal-title":"Judgment and Decision Making"},{"key":"bibr42-1948550619875149","unstructured":"Stokel-Walker C. (2018, October 1). Bots on Amazon\u2019s Mechanical Turk are ruining psychology studies. Retrieved February 4, 2019, from New Scientist website: https:\/\/www.newscientist.com\/article\/2176436-bots-on-amazons-mechanical-turk-are-ruining-psychology-studies\/"},{"key":"bibr43-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0016836"},{"key":"bibr44-1948550619875149","volume-title":"It\u2019s 2019: Do we need super attention check items to conduct web-based survey research? The evolution of MTurk survey respondents","author":"Sylaska K.","year":"2019"},{"key":"bibr45-1948550619875149","unstructured":"U.S. Census Bureau. (2018). Historical households tables, households by size. Retrieved February 4, 2019, from https:\/\/www.census.gov\/data\/tables\/time-series\/demo\/families\/households.html"},{"key":"bibr83-1948550619875149","volume-title":"Paper presented at the 71st Annual Conference of the American Association for Public Opinion Research","author":"Vannette D. L.","year":"2016"},{"key":"bibr46-1948550619875149","unstructured":"Vannette D. (2017, June 29). Using attention checks in your surveys may harm data quality. Retrieved July 18, 2019, from Qualtrics website: https:\/\/www.qualtrics.com\/blog\/using-attention-checks-in-your-surveys-may-harm-data-quality\/"},{"key":"bibr47-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.62.2.71"},{"key":"bibr48-1948550619875149","doi-asserted-by":"publisher","DOI":"10.1177\/1948550617703168"},{"key":"bibr49-1948550619875149","unstructured":"Zhu D., Carterette B. (2010). An analysis of assessor behavior in crowdsourced preference judgments. Presented at the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluations, Geneva, Switzerland, 17\u201320."}],"container-title":["Social Psychological and Personality Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1948550619875149","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1948550619875149","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1948550619875149","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T22:10:35Z","timestamp":1777587035000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1948550619875149"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,10]]},"references-count":53,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,5]]}},"alternative-id":["10.1177\/1948550619875149"],"URL":"https:\/\/doi.org\/10.1177\/1948550619875149","relation":{},"ISSN":["1948-5506","1948-5514"],"issn-type":[{"value":"1948-5506","type":"print"},{"value":"1948-5514","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,10]]}}}