{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T15:15:50Z","timestamp":1788880550359,"version":"build-2803163510"},"reference-count":43,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2019,12,16]],"date-time":"2019-12-16T00:00:00Z","timestamp":1576454400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"DOI":"10.1371\/journal.pone.0226394","type":"journal-article","created":{"date-parts":[[2019,12,16]],"date-time":"2019-12-16T13:30:57Z","timestamp":1576503057000},"page":"e0226394","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":161,"title":["Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool"],"prefix":"10.1371","volume":"14","author":[{"given":"Jonathan","family":"Robinson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cheskie","family":"Rosenzweig","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4396-4128","authenticated-orcid":true,"given":"Aaron J.","family":"Moss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leib","family":"Litman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2019,12,16]]},"reference":[{"issue":"1","key":"pone.0226394.ref001","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/1745691610393980","article-title":"Amazon\u2019s Mechanical Turk: A new source of inexpensive, yet high-quality, data?","volume":"6","author":"M Buhrmester","year":"2011","journal-title":"Perspect Psychol Sci"},{"key":"pone.0226394.ref002","doi-asserted-by":"crossref","unstructured":"Bentley JW. Challenges with Amazon Mechanical Turk research in accounting. SSRN 2924876. 2018 Mar 30.","DOI":"10.2139\/ssrn.2924876"},{"key":"pone.0226394.ref003","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1146\/annurev-clinpsy-021815-093623","article-title":"Conducting clinical research using crowdsourced convenience samples","volume":"12","author":"J Chandler","year":"2016","journal-title":"Annu Rev Clin Psychol"},{"issue":"1","key":"pone.0226394.ref004","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1093\/jcr\/ucx047","article-title":"Crowdsourcing consumer research","volume":"44","author":"JK Goodman","year":"2017","journal-title":"J Consum Res"},{"issue":"6291","key":"pone.0226394.ref005","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1126\/science.352.6291.1263","article-title":"Mechanical Turk upends social sciences","volume":"352","author":"J Bohannon","year":"2016","journal-title":"Science"},{"issue":"10","key":"pone.0226394.ref006","doi-asserted-by":"crossref","first-page":"736","DOI":"10.1016\/j.tics.2017.06.007","article-title":"Crowdsourcing samples in cognitive science","volume":"21","author":"N Stewart","year":"2017","journal-title":"Trends in Cogn Sci"},{"issue":"4","key":"pone.0226394.ref007","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1037\/pspa0000056","article-title":"The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions","volume":"111","author":"H Zhou","year":"2016","journal-title":"J Pers Soc Psychol"},{"issue":"3","key":"pone.0226394.ref008","article-title":"Under the hood of Mechanical Turk","volume":"29","author":"KA DeSoto","year":"2016","journal-title":"APS Obs"},{"key":"pone.0226394.ref009","doi-asserted-by":"crossref","unstructured":"Sorokin A, Forsyth D. Utility data annotation with Amazon Mechanical Turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2008 Jun 23 (pp. 1\u20138). IEEE.","DOI":"10.1109\/CVPRW.2008.4562953"},{"issue":"5","key":"pone.0226394.ref010","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1017\/S1930297500005611","article-title":"The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers","volume":"10","author":"N Stewart","year":"2015","journal-title":"Judgm Decis Mak"},{"issue":"6054","key":"pone.0226394.ref011","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1126\/science.334.6054.307","article-title":"Social science for pennies","volume":"334","author":"J Bohannon","year":"2011","journal-title":"Science"},{"key":"pone.0226394.ref012","volume-title":"Handbook of research methods in consumer psychology","author":"D Hauser"},{"issue":"7","key":"pone.0226394.ref013","doi-asserted-by":"crossref","first-page":"1131","DOI":"10.1177\/0956797615585115","article-title":"Using nonnaive participants can reduce effect sizes","volume":"26","author":"J Chandler","year":"2015","journal-title":"Psychol Sci"},{"key":"pone.0226394.ref014","first-page":"1","article-title":"Online panels in social science research: Expanding sampling methods beyond Mechanical Turk","volume":"11","author":"J Chandler","year":"2019","journal-title":"Behav Res Methods"},{"key":"pone.0226394.ref015","doi-asserted-by":"crossref","unstructured":"Stagnaro M, Pennycook G, Rand DG. Performance on the Cognitive Reflection Test is stable across time. SSRN 3115809. 2018.","DOI":"10.2139\/ssrn.3115809"},{"key":"pone.0226394.ref016","first-page":"1","article-title":"Generalizing from survey experiments conducted on Mechanical Turk: A replication approach","author":"A Coppock","year":"2018","journal-title":"Political Sci Res Methods"},{"issue":"2","key":"pone.0226394.ref017","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1017\/XPS.2015.19","article-title":"The generalizability of survey experiments","volume":"2","author":"KJ Mullinix","year":"2015","journal-title":"Journal of Experimental Political Science"},{"issue":"1","key":"pone.0226394.ref018","doi-asserted-by":"crossref","first-page":"400","DOI":"10.3758\/s13428-015-0578-z","article-title":"Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants","volume":"48","author":"DJ Hauser","year":"2016","journal-title":"Behav Res Methods"},{"issue":"2","key":"pone.0226394.ref019","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1177\/2167702612469015","article-title":"Using Mechanical Turk to study clinical populations","volume":"1","author":"DN Shapiro","year":"2013","journal-title":"Clin Psychol Sci"},{"issue":"3","key":"pone.0226394.ref020","doi-asserted-by":"crossref","first-page":"e57410","DOI":"10.1371\/journal.pone.0057410","article-title":"Evaluating Amazon\u2019s Mechanical Turk as a tool for experimental behavioral research","volume":"8","author":"MJ Crump","year":"2013","journal-title":"PloS One"},{"key":"pone.0226394.ref021","doi-asserted-by":"crossref","unstructured":"Difallah D, Filatova E, Ipeirotis P. Demographics and dynamics of Mechanical Turk workers. In Proceedings of the eleventh ACM international conference on web search and data mining 2018 Feb 2 (pp. 135\u2013143). ACM.","DOI":"10.1145\/3159652.3159661"},{"key":"pone.0226394.ref022","unstructured":"Ipeirotis PG. Demographics of Mechanical Turk. CeDER-10\u201301 working paper, New York University. https:\/\/archive.nyu.edu\/bitstream\/2451\/29585\/2\/CeDER-10-01.pdf?__hstc=214931602.e72c280a7921bf0d7ab734f9822a9c39.1532390400111.1532390400112.1532390400113.1&__hssc=214931602.1.1532390400114&__hsfp=1773666937"},{"issue":"2","key":"pone.0226394.ref023","doi-asserted-by":"crossref","first-page":"433","DOI":"10.3758\/s13428-016-0727-z","article-title":"TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences","volume":"49","author":"L Litman","year":"2017","journal-title":"Behavior Res Methods"},{"issue":"2","key":"pone.0226394.ref024","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1162\/COLI_a_00057","article-title":"Amazon Mechanical Turk: Gold mine or coal mine?","volume":"37","author":"K Fort","year":"2011","journal-title":"Comput Linguist"},{"key":"pone.0226394.ref025","unstructured":"MTurk Crowd. Your first 1000 HITs [Internet]. Message posted by user Jklmnop 2016 Jan 12 to https:\/\/www.mturkcrowd.com\/threads\/your-first-1000-hits.23\/"},{"issue":"4","key":"pone.0226394.ref026","doi-asserted-by":"crossref","first-page":"496","DOI":"10.3758\/BF03210601","article-title":"Virtual subjects: Using the Internet as an alternative source of subjects and research environment","volume":"29","author":"MA Smith","year":"1997","journal-title":"Behav Res Methods Instrum Comput"},{"key":"pone.0226394.ref027","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1016\/j.chb.2015.12.049","article-title":"Psychological research in the internet age: The quality of web-based data","volume":"58","author":"SR Ramsey","year":"2016","journal-title":"Comput Human Behav"},{"issue":"4","key":"pone.0226394.ref028","doi-asserted-by":"crossref","first-page":"1023","DOI":"10.3758\/s13428-013-0434-y","article-title":"Reputation as a sufficient condition for data quality on Amazon Mechanical Turk","volume":"46","author":"E Peer","year":"2014","journal-title":"Behavior Res Methods"},{"issue":"3","key":"pone.0226394.ref029","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1093\/pan\/mpr057","article-title":"Evaluating online labor markets for experimental research: Amazon. com\u2019s Mechanical Turk","volume":"20","author":"AJ Berinsky","year":"2012","journal-title":"Polit Anal"},{"issue":"3","key":"pone.0226394.ref030","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1002\/bdm.1753","article-title":"Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples","volume":"26","author":"JK Goodman","year":"2013","journal-title":"J Behav Decis Mak"},{"issue":"5","key":"pone.0226394.ref031","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1017\/S1930297500002205","article-title":"Running experiments on amazon mechanical turk","volume":"5","author":"G Paolacci","year":"2010","journal-title":"Judgm Decis Mak"},{"issue":"4481","key":"pone.0226394.ref032","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1126\/science.7455683","article-title":"The framing of decisions and the psychology of choice","volume":"211","author":"A Tversky","year":"1981","journal-title":"Science"},{"issue":"11","key":"pone.0226394.ref033","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1177\/01461672952111004","article-title":"Measures of anchoring in estimation tasks","volume":"21","author":"KE Jacowitz","year":"1995","journal-title":"Pers Soc Psychol Rev"},{"issue":"2","key":"pone.0226394.ref034","doi-asserted-by":"crossref","first-page":"204","DOI":"10.5840\/monist197659224","article-title":"Killing, letting die, and the trolley problem","volume":"59","author":"JJ Thomson","year":"1976","journal-title":"The Monist"},{"issue":"1","key":"pone.0226394.ref035","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.1468-0017.2006.00297.x","article-title":"A dissociation between moral judgments and justifications","volume":"22","author":"M Hauser","year":"2007","journal-title":"Mind Lang"},{"key":"pone.0226394.ref036","first-page":"114","volume-title":"Handbook of personality: Theory and research","author":"OP John","year":"2008"},{"issue":"2","key":"pone.0226394.ref037","doi-asserted-by":"crossref","first-page":"519","DOI":"10.3758\/s13428-014-0483-x","article-title":"The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk","volume":"47","author":"L Litman","year":"2015","journal-title":"Behavior Res Methods"},{"issue":"4","key":"pone.0226394.ref038","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1257\/089533005775196732","article-title":"Cognitive reflection and decision making","volume":"19","author":"S Frederick","year":"2005","journal-title":"J Econ Perspect"},{"key":"pone.0226394.ref039","volume-title":"American National Election Study: 2016 Pilot Study","author":"American National Election Studies, Stanford University, and University of Michigan","year":"2016"},{"key":"pone.0226394.ref040","unstructured":"Moss AJ, Litman L. The TurkPrime Blog [Internet]. After the bot scare: Understanding what\u2019s been happening with data collection on MTurk and how to stop it. 2018, Sept 18. [cited 2019 June 3]. https:\/\/blog.turkprime.com\/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it"},{"key":"pone.0226394.ref041","unstructured":"Kennedy, R, Clifford, S, Burleigh, T, Waggoner, P, Jewell, R. How Venezuela\u2019s economic crisis is undermining social science research\u2014about everything. The Washington Post. 2018, Nov 7. https:\/\/www.washingtonpost.com\/news\/monkey-cage\/wp\/2018\/11\/07\/how-the-venezuelan-economic-crisis-is-undermining-social-science-research-about-everything-not-just-venezuela\/?noredirect=on&utm_term=.e9f29ab7f1e6"},{"key":"pone.0226394.ref042","unstructured":"Chandler J., L. Litman & Y. Robinson. Predicting retention in longitudinal studies conducted on Mechanical Turk.\" Presentation at the American Association for Public Opinion Research Conference, Toronto, May, 2019."},{"key":"pone.0226394.ref043","unstructured":"Robinson J, Litman L, 2020. Online research on Mechanical Turk and other platforms. In Press. SAGE Publications."}],"updated-by":[{"DOI":"10.1371\/journal.pone.0342218","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T00:00:00Z","timestamp":1770163200000}}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0226394","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T22:54:10Z","timestamp":1695509650000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0226394"}},"subtitle":[],"editor":[{"given":"Frantisek","family":"Sudzina","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2019,12,16]]},"references-count":43,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,12,16]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0226394","relation":{"has-preprint":[{"id-type":"doi","id":"10.31234\/osf.io\/jq589","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12,16]]}}}