{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T06:25:08Z","timestamp":1795155908790,"version":"4.10.3"},"reference-count":52,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T00:00:00Z","timestamp":1667433600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T00:00:00Z","timestamp":1667433600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Behav Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch\u2019s vetting. In a pre-registered study, participants (\n                    <jats:italic>N<\/jats:italic>\n                    = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk\u2019s Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.\n                  <\/jats:p>","DOI":"10.3758\/s13428-022-01999-x","type":"journal-article","created":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T10:03:37Z","timestamp":1667469817000},"page":"3953-3964","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":222,"title":["Evaluating CloudResearch\u2019s Approved Group as a solution for problematic data quality on MTurk"],"prefix":"10.3758","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5480-5213","authenticated-orcid":false,"given":"David J.","family":"Hauser","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4396-4128","authenticated-orcid":false,"given":"Aaron J.","family":"Moss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2403-6940","authenticated-orcid":false,"given":"Cheskie","family":"Rosenzweig","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7523-5213","authenticated-orcid":false,"given":"Shalom N.","family":"Jaffe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5353-7969","authenticated-orcid":false,"given":"Jonathan","family":"Robinson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6598-2701","authenticated-orcid":false,"given":"Leib","family":"Litman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,11,3]]},"reference":[{"key":"1999_CR1","doi-asserted-by":"publisher","unstructured":"Ahler, D. J., Roush, C. E., & Sood, G. (2019). The micro-task market for lemons: Data quality on Amazon\u2019s Mechanical Turk. Political Science Research and Methods, 1\u201320. https:\/\/doi.org\/10.1017\/psrm.2021.57","DOI":"10.1017\/psrm.2021.57"},{"key":"1999_CR2","unstructured":"Bai, H. (2018). Evidence that a large amount of low quality responses on MTurk can be detected with repeated GPS coordinates. Retrieved from: https:\/\/www.maxhuibai.com\/blog\/evidence-that-responses-from-repeating-gps-are-random"},{"issue":"3","key":"1999_CR3","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1111\/ajps.12081","volume":"58","author":"AJ Berinsky","year":"2014","unstructured":"Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58(3), 739\u2013753.","journal-title":"American Journal of Political Science"},{"key":"1999_CR4","doi-asserted-by":"publisher","unstructured":"Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., ...Van\u2019t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217\u2013224.https:\/\/doi.org\/10.1016\/j.jesp.2013.10.005","DOI":"10.1016\/j.jesp.2013.10.005"},{"key":"1999_CR5","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/1745691610393980","volume":"6","author":"M Buhrmester","year":"2011","unstructured":"Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon\u2019s Mechanical Turk: A new source of inexpensive, yet high-quality data? Perspectives on Psychological Science, 6, 3\u20135. https:\/\/doi.org\/10.1177\/1745691610393980","journal-title":"Perspectives on Psychological Science"},{"key":"1999_CR6","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1177\/1745691617706516","volume":"13","author":"MD Buhrmester","year":"2018","unstructured":"Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon\u2019s Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science, 13, 149\u2013154. https:\/\/doi.org\/10.1177\/1745691617706516","journal-title":"Perspectives on Psychological Science"},{"issue":"4","key":"1999_CR7","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1093\/poq\/nfw030","volume":"80","author":"S Clifford","year":"2016","unstructured":"Clifford, S., & Jerit, J. (2016). Cheating on political knowledge questions in online surveys: An assessment of the problem and solutions. Public Opinion Quarterly, 80(4), 858\u2013887.","journal-title":"Public Opinion Quarterly"},{"key":"1999_CR8","first-page":"95","volume-title":"Conducting online research on Amazon mechanical Turk and beyond","author":"J Chandler","year":"2020","unstructured":"Chandler, J., Paolacci, G., & Hauser, D. J. (2020). Data quality issues on mechanical turk. In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon mechanical Turk and beyond (pp. 95\u2013120). Sage Academic Publishing. Thousand Oaks."},{"issue":"4","key":"1999_CR9","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1177\/1948550619875149","volume":"11","author":"M Chmielewski","year":"2020","unstructured":"Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464\u2013473.","journal-title":"Social Psychological and Personality Science"},{"key":"1999_CR10","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1016\/j.jesp.2015.07.006","volume":"66","author":"PG Curran","year":"2016","unstructured":"Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4\u201319.","journal-title":"Journal of Experimental Social Psychology"},{"issue":"1","key":"1999_CR11","doi-asserted-by":"publisher","first-page":"119","DOI":"10.2308\/bria-18-044","volume":"32","author":"SA Dennis","year":"2020","unstructured":"Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2020). Online worker fraud and evolving threats to the integrity of MTurk data: A discussion of virtual private servers and the limitations of IP-based screening procedures. Behavioral Research in Accounting, 32(1), 119\u2013134. https:\/\/doi.org\/10.2308\/bria-18-044","journal-title":"Behavioral Research in Accounting"},{"key":"1999_CR12","doi-asserted-by":"publisher","first-page":"175","DOI":"10.3758\/BF03193146","volume":"39","author":"F Faul","year":"2007","unstructured":"Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175\u2013191. https:\/\/doi.org\/10.3758\/BF03193146","journal-title":"Behavior Research Methods"},{"key":"1999_CR13","unstructured":"Gautam, R., Kerstein, M., Moss, A. J., & Litman, L. (2018). Understanding geolocations and their connection to data quality. [blog post]. Retrieved from: https:\/\/www.cloudresearch.com\/resources\/blog\/understanding-geolocations-and-their-connection-to-data-quality\/"},{"issue":"4","key":"1999_CR14","doi-asserted-by":"publisher","first-page":"700","DOI":"10.1037\/xge0000920","volume":"150","author":"R Gomila","year":"2020","unstructured":"Gomila, R. (2020). Logistic or linear? Estimating causal effects of experimental treatments on binary outcomes using regression analysis. Journal of Experimental Psychology: General, 150(4), 700.","journal-title":"Journal of Experimental Psychology: General"},{"issue":"3","key":"1999_CR15","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1002\/bdm.1753","volume":"26","author":"JK Goodman","year":"2013","unstructured":"Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213\u2013224.","journal-title":"Journal of Behavioral Decision Making"},{"key":"1999_CR16","doi-asserted-by":"publisher","first-page":"400","DOI":"10.3758\/s13428-015-0578-z","volume":"48","author":"DJ Hauser","year":"2016","unstructured":"Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400\u2013407. https:\/\/doi.org\/10.3758\/s13428-015-0578-z","journal-title":"Behavior Research Methods"},{"key":"1999_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1111\/j.1468-0017.2006.00297.x","volume":"22","author":"M Hauser","year":"2007","unstructured":"Hauser, M., Cushman, F., Young, L., Kang-Xing Jin, R., & Mikhail, J. (2007). A dissociation between moral judgments and justifications. Mind & Language, 22, 1\u201321. https:\/\/doi.org\/10.1111\/j.1468-0017.2006.00297.x","journal-title":"Mind & Language"},{"key":"1999_CR18","volume-title":"Handbook of research methods in consumer psychology","author":"DJ Hauser","year":"2019","unstructured":"Hauser, D. J., Paolacci, G., & Chandler, J. J. (2019). Common concerns with MTurk as a participant pool: Evidence and solutions. In F. R. Kardes, P. M. Herr, & N. Schwarz (Eds.), Handbook of research methods in consumer psychology. Routledge."},{"key":"1999_CR19","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1177\/01461672952111004","volume":"21","author":"KE Jacowitz","year":"1995","unstructured":"Jacowitz, K. E., & Kahneman, D. (1995). Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21, 1161\u20131166. https:\/\/doi.org\/10.1177\/01461672952111004","journal-title":"Personality and Social Psychology Bulletin"},{"key":"1999_CR20","first-page":"114","volume-title":"Handbook of personality: Theory and research","author":"OP John","year":"2008","unstructured":"John, O. P., Naumann, L. P., & Soto, C. J. (2008). Paradigm shift to the integrative Big Five trait taxonomy: History, measurement, and conceptual issues. In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality: Theory and research (3rd ed., pp. 114\u2013158). Guilford Press.","edition":"3"},{"key":"1999_CR21","doi-asserted-by":"publisher","DOI":"10.33774\/apsa-2020-96t72","author":"JV Kane","year":"2020","unstructured":"Kane, J. V., Velez, Y. R., & Barabas, J. (2020). Analyze the attentive and bypass bias: Mock vignette checks in survey experiments. APSA Preprints. https:\/\/doi.org\/10.33774\/apsa-2020-96t72","journal-title":"APSA Preprints"},{"key":"1999_CR22","unstructured":"Kennedy, C., Hatley, N., Lau, A., Mercer, A., Keeter, S., Ferno, J., & Asare-Marfo, D. (2020a). Assessing the risks to online polls from bogus respondents. Pew Research Center Methods. https:\/\/www.pewresearch.org\/methods\/2020a\/02\/18\/assessing-the-risks-to-online-polls-from-bogus-respondents\/"},{"key":"1999_CR23","doi-asserted-by":"publisher","first-page":"614","DOI":"10.1017\/psrm.2020.6","volume":"8","author":"R Kennedy","year":"2020","unstructured":"Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. (2020b). The shape of and solutions to the MTurk quality crisis. Political Science Research and Methods, 8, 614\u2013629.","journal-title":"Political Science Research and Methods"},{"issue":"3","key":"1999_CR24","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1002\/acp.2350050305","volume":"5","author":"JA Krosnick","year":"1991","unstructured":"Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive Psychology, 5(3), 213\u2013236.","journal-title":"Applied cognitive Psychology"},{"key":"1999_CR25","unstructured":"Levesque, H., Davis, E., & Morgenstern, L. (2012). The Winograd schema challenge. In Proceedings of KR, pp 362\u2013372, Rome, Italy."},{"key":"1999_CR26","first-page":"1","volume-title":"Conducting online research on Amazon mechanical turk and beyond","author":"L Litman","year":"2020","unstructured":"Litman, L., & Robinson, J. (2020a). Introduction. In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon mechanical turk and beyond (pp. 1\u201326). Sage Academic Publishing."},{"key":"1999_CR27","first-page":"234","volume-title":"Conducting online research on Amazon mechanical turk and beyond","author":"L Litman","year":"2020","unstructured":"Litman, L., & Robinson, J. (2020b). Conducting ethical online research: A data-driven approach. In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon mechanical turk and beyond (pp. 234\u2013263). Sage Academic Publishing."},{"key":"1999_CR28","doi-asserted-by":"publisher","first-page":"519","DOI":"10.3758\/s13428-014-0483-x","volume":"47","author":"L Litman","year":"2015","unstructured":"Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods, 47, 519\u2013528. https:\/\/doi.org\/10.3758\/s13428-014-0483-x","journal-title":"Behavior Research Methods"},{"issue":"2","key":"1999_CR29","doi-asserted-by":"publisher","first-page":"433","DOI":"10.3758\/s13428-016-0727-z","volume":"49","author":"L Litman","year":"2017","unstructured":"Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49(2), 433\u2013442.","journal-title":"Behavior Research Methods"},{"key":"1999_CR30","unstructured":"Litman, L., Rosenzweig, C., & Moss, A (2020). New solutions dramatically improve research data quality on MTurk. CloudResearch. https:\/\/www.cloudresearch.com\/resources\/blog\/new-tools-improve-research-data-quality-mturk\/"},{"key":"1999_CR31","doi-asserted-by":"publisher","unstructured":"Litman, L., Rosenzweig, C., Jaffe, S. N., Gautam, R., Robinson, J., & Moss, A. J. (2021). Bots or inattentive humans? Identifying sources of low-quality data in online platforms. https:\/\/doi.org\/10.31234\/osf.io\/wr8ds","DOI":"10.31234\/osf.io\/wr8ds"},{"key":"1999_CR32","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/j.jesp.2016.09.006","volume":"69","author":"A Luttrell","year":"2017","unstructured":"Luttrell, A., Petty, R. E., & Xu, M. (2017). Replicating and fixing failed replications: The case of need for cognition and argument quality. Journal of Experimental Social Psychology, 69, 178\u2013183.","journal-title":"Journal of Experimental Social Psychology"},{"key":"1999_CR33","unstructured":"Mechanical Turk Replication Project (2021). #MTRP: Mechanical Turk Replication Project. https:\/\/www.mtrp.info\/index.html"},{"key":"1999_CR34","unstructured":"Moss, A. J., & Litman, L. (2018). After the bot scare: Understanding what\u2019s been happening with data collection on MTurk and how to stop it [blog post]."},{"key":"1999_CR35","doi-asserted-by":"publisher","unstructured":"Moss, A. J., Rosenzweig, C., Robinson, J., Jaffe, S. N., & Litman, L. (2021). Is it ethical to use mechanical turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. https:\/\/doi.org\/10.31234\/osf.io\/jbc9d","DOI":"10.31234\/osf.io\/jbc9d"},{"issue":"4","key":"1999_CR36","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1093\/ijpor\/edw027","volume":"29","author":"MP Motta","year":"2017","unstructured":"Motta, M. P., Callaghan, T. H., & Smith, B. (2017). Looking for answers: Identifying search behavior and improving knowledge-based data quality in online surveys. International Journal of Public Opinion Research, 29(4), 575\u2013603. https:\/\/doi.org\/10.1093\/ijpor\/edw027","journal-title":"International Journal of Public Opinion Research"},{"issue":"6","key":"1999_CR37","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0157732","volume":"11","author":"EA Necka","year":"2016","unstructured":"Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016). Measuring the prevalence of problematic respondent behaviors among MTurk, campus, and community participants. PLOS ONE, 11(6), e0157732.","journal-title":"PLOS ONE"},{"issue":"4","key":"1999_CR38","doi-asserted-by":"publisher","first-page":"867","DOI":"10.1016\/j.jesp.2009.03.009","volume":"45","author":"DM Oppenheimer","year":"2009","unstructured":"Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867\u2013872.","journal-title":"Journal of Experimental Social Psychology"},{"key":"1999_CR39","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1017\/S1930297500002205","volume":"5","author":"G Paolacci","year":"2010","unstructured":"Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411\u2013419.","journal-title":"Judgment and Decision Making"},{"key":"1999_CR40","doi-asserted-by":"publisher","first-page":"1023","DOI":"10.3758\/s13428-013-0434-y","volume":"46","author":"E Peer","year":"2014","unstructured":"Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023\u20131031. https:\/\/doi.org\/10.3758\/s13428-013-0434-y","journal-title":"Behavior Research Methods"},{"key":"1999_CR41","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1016\/j.jesp.2017.01.006","volume":"70","author":"E Peer","year":"2017","unstructured":"Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153\u2013163.","journal-title":"Journal of Experimental Social Psychology"},{"issue":"2","key":"1999_CR42","doi-asserted-by":"publisher","first-page":"188","DOI":"10.1177\/2515245919838479","volume":"2","author":"S Permut","year":"2019","unstructured":"Permut, S., Fisher, M., & Oppenheimer, D. M. (2019). TaskMaster: A tool for determining when subjects are on task. Advances in Methods and Practices in Psychological Science, 2(2), 188\u2013196.","journal-title":"Advances in Methods and Practices in Psychological Science"},{"key":"1999_CR43","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1016\/j.copsyc.2016.07.001","volume":"12","author":"M Ramscar","year":"2016","unstructured":"Ramscar, M. (2016). Learning and the replicability of priming effects. Current Opinion in Psychology, 12, 80\u201384.","journal-title":"Current Opinion in Psychology"},{"issue":"12","key":"1999_CR44","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0226394","volume":"14","author":"J Robinson","year":"2019","unstructured":"Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLOS ONE, 14(12), e0226394. https:\/\/doi.org\/10.1371\/journal.pone.0226394","journal-title":"PLOS ONE"},{"key":"1999_CR45","unstructured":"Ryan, T. J. (2018). Data contamination on MTurk. Retrieved from: https:\/\/timryan.web.unc.edu\/2018\/08\/12\/data-contamination-on-mturk\/"},{"issue":"4","key":"1999_CR46","first-page":"305","volume":"45","author":"N Schwarz","year":"2014","unstructured":"Schwarz, N., & Strack, F. (2014). Does merely going through the same moves make for a \u201cdirect\u201d replication? Concepts, contexts, and operationalizations. Social Psychology, 45(4), 305\u2013306.","journal-title":"Social Psychology"},{"issue":"1","key":"1999_CR47","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1177\/1065912919882101","volume":"73","author":"B Smith","year":"2020","unstructured":"Smith, B., Clifford, S., & Jerit, J. (2020). TRENDS: How internet search undermines the validity of political knowledge measures. Political Research Quarterly, 73(1), 141\u2013155. https:\/\/doi.org\/10.1177\/1065912919882101","journal-title":"Political Research Quarterly"},{"issue":"5","key":"1999_CR48","doi-asserted-by":"publisher","first-page":"472","DOI":"10.20982\/tqmp.16.5.p472","volume":"16","author":"A Storozuk","year":"2020","unstructured":"Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020). Got bots? Practical recommendations to protect online survey data from bot attacks. The Quantitative Methods for Psychology, 16(5), 472\u2013481.","journal-title":"The Quantitative Methods for Psychology"},{"key":"1999_CR49","volume-title":"Exploratory data analysis","author":"JW Tukey","year":"1977","unstructured":"Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley."},{"issue":"2","key":"1999_CR50","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1016\/0010-0285(73)90033-9","volume":"5","author":"A Tversky","year":"1973","unstructured":"Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability. Cognitive Psychology, 5(2), 207\u2013232.","journal-title":"Cognitive Psychology"},{"key":"1999_CR51","unstructured":"Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merri\u00ebnboer, B., Joulin, A., & Mikolov, T. (2015). Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698."},{"key":"1999_CR52","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1037\/pspa0000056","volume":"111","author":"H Zhou","year":"2016","unstructured":"Zhou, H., & Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111, 493\u2013504. https:\/\/doi.org\/10.1037\/pspa0000056","journal-title":"Journal of Personality and Social Psychology"}],"container-title":["Behavior Research Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-022-01999-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-022-01999-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-022-01999-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,6]],"date-time":"2023-12-06T11:03:47Z","timestamp":1701860627000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.3758\/s13428-022-01999-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,3]]},"references-count":52,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["1999"],"URL":"https:\/\/doi.org\/10.3758\/s13428-022-01999-x","relation":{},"ISSN":["1554-3528"],"issn-type":[{"value":"1554-3528","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,3]]},"assertion":[{"value":"2 October 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 November 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"\u00a0All procedures performed in studies involving human participants were in accordance with the ethical standards of the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Informed consent was obtained from all individual participants included in the study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"N\/A.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Author DJH declares no competing financial or non-financial interests. Authors AJM, CR, SNJ, JR, and LL are employees of and receive salaries from CloudResearch.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"The data and materials for all studies are available at\n                      \n                      . The study was preregistered\u00a0\n                      \n                      .","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Open Practices Statement"}}]}}