{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,4]],"date-time":"2026-11-04T09:09:53Z","timestamp":1793783393582,"version":"4.6.1"},"reference-count":72,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T00:00:00Z","timestamp":1713744000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T00:00:00Z","timestamp":1713744000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Perspect Psychol Sci"],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:p>\n                    A recent article in\n                    <jats:italic toggle=\"yes\">Perspectives on Psychological Science<\/jats:italic>\n                    (Webb &amp; Tangney, 2022) reported a study in which just 2.6% of participants recruited on Amazon\u2019s Mechanical Turk (MTurk) were deemed \u201cvalid.\u201d The authors highlighted some well-established limitations of MTurk, but their central claims\u2014that MTurk is \u201ctoo good to be true\u201d and that it captured \u201conly 14 human beings . . . [out of]\n                    <jats:italic toggle=\"yes\">N<\/jats:italic>\n                    = 529\u201d\u2014are radically misleading, yet have been repeated widely. This commentary aims to (a) correct the record (i.e., by showing that Webb and Tangney\u2019s approach to data collection led to unusually low data quality) and (b) offer a shift in perspective for running high-quality studies online. Negative attitudes toward MTurk sometimes reflect a fundamental misunderstanding of what the platform offers and how it should be used in research. Beyond pointing to research that details strategies for effective design and recruitment on MTurk, we stress that MTurk is not suitable for every study. Effective use requires specific expertise and design considerations. Like all tools used in research\u2014from advanced hardware to specialist software\u2014the tool itself places constraints on what one should use it for. Ultimately, high-quality data is the responsibility of the researcher, not the crowdsourcing platform.\n                  <\/jats:p>","DOI":"10.1177\/17456916241242734","type":"journal-article","created":{"date-parts":[[2024,4,22]],"date-time":"2024-04-22T15:50:57Z","timestamp":1713801057000},"page":"891-899","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":34,"title":["The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms"],"prefix":"10.1177","volume":"19","author":[{"given":"Christine","family":"Cuskley","sequence":"first","affiliation":[{"name":"Language Evolution, Acquisition, and Development Group, Newcastle University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0978-9496","authenticated-orcid":false,"given":"Justin","family":"Sulik","sequence":"additional","affiliation":[{"name":"Cognition, Values and Behavior, LMU Munich"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2024,4,22]]},"reference":[{"key":"e_1_3_4_2_1","first-page":"885","volume-title":"Behavioral Research Methods","author":"Agley J.","year":"2022","unstructured":"Agley J., Xiao Y., Nolan R., Golzarri-Arroyo L. (2022). Quality control questions on Amazon\u2019s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7. Behavioral Research Methods, 54, 885\u2013897. https:\/\/doi.org\/10.3758\/s13428-021-01665-8"},{"key":"e_1_3_4_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/0149206320969787"},{"key":"e_1_3_4_4_1","doi-asserted-by":"publisher","DOI":"10.1017\/psrm.2021.57"},{"key":"e_1_3_4_5_1","doi-asserted-by":"publisher","DOI":"10.1177\/0146167218798821"},{"key":"e_1_3_4_6_1","doi-asserted-by":"crossref","unstructured":"Bauer B. Larsen K. L. Caulfield N. Elder D. Jordan S. Capron D. (2020 November 14). Review of best practice recommendations for ensuring high quality data with Amazon\u2019s Mechanical Turk. https:\/\/doi.org\/10.31234\/osf.io\/m78sf","DOI":"10.31234\/osf.io\/m78sf"},{"key":"e_1_3_4_7_1","doi-asserted-by":"publisher","DOI":"10.1080\/00913367.2022.2079026"},{"key":"e_1_3_4_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10869-021-09787-8"},{"key":"e_1_3_4_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.metip.2020.100022"},{"key":"e_1_3_4_10_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-018-1035-6"},{"key":"e_1_3_4_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"e_1_3_4_12_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691617706516"},{"key":"e_1_3_4_13_1","unstructured":"Cape P. Phillips K. (2015) Questionnaire length and fatigue effects: The latest thinking and practical solutions [White paper]. Survey Sampling International. https:\/\/silo.tips\/download\/white-paper-questionnaire-length-and-fatigue-effects-the-latest-thinking-and-pra"},{"key":"e_1_3_4_14_1","doi-asserted-by":"publisher","DOI":"10.1017\/9781009010054.010"},{"key":"e_1_3_4_15_1","doi-asserted-by":"publisher","DOI":"10.1037\/abn0000479"},{"key":"e_1_3_4_16_1","unstructured":"Charalambides N. (2021). We recently went viral on TikTok - here\u2019s what we learned. Prolific Blog. https:\/\/www.prolific.co\/blog\/we-recently-went-viral-on-tiktok-heres-what-we-learned"},{"key":"e_1_3_4_17_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550619875149"},{"key":"e_1_3_4_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/089443930101900202"},{"key":"e_1_3_4_19_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0057410"},{"key":"e_1_3_4_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2015.07.006"},{"key":"e_1_3_4_21_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-014-0458-y"},{"key":"e_1_3_4_22_1","doi-asserted-by":"publisher","DOI":"10.2308\/bria-18-044"},{"key":"e_1_3_4_23_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0279720"},{"key":"e_1_3_4_24_1","doi-asserted-by":"publisher","DOI":"10.1037\/met0000267"},{"key":"e_1_3_4_25_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-018-1103-y"},{"key":"e_1_3_4_26_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-022-01955-9"},{"key":"e_1_3_4_27_1","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfp031"},{"key":"e_1_3_4_28_1","doi-asserted-by":"publisher","DOI":"10.1177\/0049124118769083"},{"key":"e_1_3_4_29_1","doi-asserted-by":"crossref","unstructured":"Hauser D. J. Moss A. J. Rosenzweig C. Jaffe S. N. Robinson J. Litman L. (2022). Evaluating CloudResearch\u2019s Approved Group as a solution for problematic data quality on MTurk. Behavior Research Methods. Advance online publication. https:\/\/doi.org\/10.3758\/s13428-022-01999-x","DOI":"10.31234\/osf.io\/48yxj"},{"key":"e_1_3_4_30_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781351137713-17"},{"key":"e_1_3_4_31_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-015-0578-z"},{"key":"e_1_3_4_32_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-017-0984-5"},{"key":"e_1_3_4_33_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2017.01359"},{"key":"e_1_3_4_34_1","doi-asserted-by":"publisher","DOI":"10.1017\/psrm.2020.6"},{"key":"e_1_3_4_35_1","doi-asserted-by":"publisher","DOI":"10.2307\/2088175"},{"key":"e_1_3_4_36_1","author":"Litman L.","year":"2020","unstructured":"Litman L., Robinson J. (2020). Conducting online research on Amazon Mechanical Turk and beyond. Sage.","journal-title":"Conducting online research on Amazon Mechanical Turk and beyond"},{"key":"e_1_3_4_37_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-014-0483-x"},{"key":"e_1_3_4_38_1","doi-asserted-by":"publisher","DOI":"10.1177\/0894439317695581"},{"key":"e_1_3_4_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsis.2016.06.002"},{"key":"e_1_3_4_40_1","doi-asserted-by":"publisher","DOI":"10.1177\/19389655211025475"},{"key":"e_1_3_4_41_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_3_4_42_1","doi-asserted-by":"crossref","unstructured":"Moss A. J. Hauser D. J. Rosenzweig C. Jaffe S. Robinson J. Litman L. (2023). Using market-research panels for behavioral science: An overview and tutorial. Advances in Methods and Practices in Psychological Science 6(2). https:\/\/doi.org\/10.1177\/25152459221140388","DOI":"10.1177\/25152459221140388"},{"key":"e_1_3_4_43_1","doi-asserted-by":"crossref","unstructured":"Moss A. J. Rosenzweig C. Jaffe S. N. Gautam R. Robinson J. Litman L. (2021 June 11). Bots or inattentive humans? Identifying sources of low-quality data in online platforms. https:\/\/doi.org\/10.31234\/osf.io\/wr8ds","DOI":"10.31234\/osf.io\/wr8ds"},{"key":"e_1_3_4_44_1","doi-asserted-by":"crossref","unstructured":"Moss A. J. Rosenzweig C. Robinson J. Jaffe S. N. Litman L. (2020 April 28). Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. https:\/\/doi.org\/10.31234\/osf.io\/jbc9d","DOI":"10.31234\/osf.io\/jbc9d"},{"key":"e_1_3_4_45_1","doi-asserted-by":"crossref","unstructured":"Necka E. A. Cacioppo S. Norman G. J. Cacioppo J. T. (2016). Measuring the prevalence of problematic respondent behaviors among MTurk campus and community participants. PLOS ONE 11(6) Article e0157732. https:\/\/doi.org\/10.1371\/journal.pone.0157732","DOI":"10.1371\/journal.pone.0157732"},{"key":"e_1_3_4_46_1","doi-asserted-by":"publisher","DOI":"10.1177\/2167702619865973"},{"key":"e_1_3_4_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2009.03.009"},{"key":"e_1_3_4_48_1","doi-asserted-by":"publisher","DOI":"10.1177\/0963721414531598"},{"key":"e_1_3_4_49_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1930297500002205"},{"key":"e_1_3_4_50_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-021-01694-3"},{"key":"e_1_3_4_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380195"},{"key":"e_1_3_4_52_1","doi-asserted-by":"publisher","DOI":"10.1080\/23736992.2016.1228811"},{"key":"e_1_3_4_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-012099980-4\/50005-8"},{"key":"e_1_3_4_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/1470785320943049"},{"issue":"5","key":"e_1_3_4_55_1","first-page":"557","article-title":"Ideal and maximum length for a web survey. International","volume":"59","author":"Revilla M.","year":"2017","unstructured":"Revilla M., Ochoa C. (2017). Ideal and maximum length for a web survey. International. Journal of Market Research, 59(5), 557\u2013565.","journal-title":"Journal of Market Research"},{"key":"e_1_3_4_56_1","doi-asserted-by":"crossref","unstructured":"Robinson J. Rosenzweig C. Moss A. J. Litman L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLOS ONE 14(12) Article e0226394. https:\/\/doi.org\/10.1371\/journal.pone.0226394","DOI":"10.1371\/journal.pone.0226394"},{"key":"e_1_3_4_57_1","doi-asserted-by":"crossref","unstructured":"Rodd J. M. (2023 April 13). Moving experimental psychology online: How to maintain data quality when we can\u2019t see our participants. https:\/\/doi.org\/10.31234\/osf.io\/2fhcb","DOI":"10.31234\/osf.io\/2fhcb"},{"key":"e_1_3_4_58_1","doi-asserted-by":"publisher","DOI":"10.1017\/iop.2015.20"},{"key":"e_1_3_4_59_1","doi-asserted-by":"publisher","DOI":"10.1177\/1525822X221115830"},{"key":"e_1_3_4_60_1","doi-asserted-by":"publisher","DOI":"10.1017\/iop.2015.29"},{"key":"e_1_3_4_61_1","doi-asserted-by":"crossref","unstructured":"Springer V. A. Martini P. J. Lindsey S. C. Vezich I. S. (2016). Practice-based considerations for using multi-stage survey design to reach special populations on Amazon\u2019s Mechanical Turk. Survey Practice 9(5). https:\/\/doi.org\/10.29115\/SP-2016-0029","DOI":"10.29115\/SP-2016-0029"},{"key":"e_1_3_4_62_1","doi-asserted-by":"publisher","DOI":"10.20982\/tqmp.16.5.p472"},{"key":"e_1_3_4_63_1","doi-asserted-by":"publisher","DOI":"10.1037\/abn0000844"},{"key":"e_1_3_4_64_1","doi-asserted-by":"publisher","DOI":"10.2471\/BLT.14.141390"},{"key":"e_1_3_4_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2017.08.038"},{"key":"e_1_3_4_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.12.074"},{"key":"e_1_3_4_67_1","unstructured":"Veselovsky V. Ribeiro M. H. Cozzolino P. Gordon A. Rothschild D. West R. (2023 October 24). Prevalence and prevention of large language model use in crowd work. https:\/\/doi.org\/10.48550\/arXiv.2310.15683"},{"key":"e_1_3_4_68_1","doi-asserted-by":"publisher","DOI":"10.1177\/17456916221120027"},{"key":"e_1_3_4_69_1","author":"Wickham H.","year":"2016","unstructured":"Wickham H., Grolemund G. (2016). R for data science: Import, tidy, transform, visualize, and model data. O\u2019Reilly Media.","journal-title":"R for data science: Import, tidy, transform, visualize, and model data"},{"key":"e_1_3_4_70_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550617703168"},{"key":"e_1_3_4_71_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781003025245-12"},{"key":"e_1_3_4_72_1","doi-asserted-by":"publisher","DOI":"10.1037\/pspa0000056"},{"key":"e_1_3_4_73_1","doi-asserted-by":"crossref","unstructured":"Zorowitz S. Niv Y. Bennett D. (2023). Inattentive responding can induce spurious associations between task behavior and symptom measures. Nature Human Behavior. Advance online publication. https:\/\/doi.org\/10.1038\/s41562-023-01640-7","DOI":"10.1038\/s41562-023-01640-7"}],"container-title":["Perspectives on Psychological Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/17456916241242734","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/17456916241242734","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/17456916241242734","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T08:45:01Z","timestamp":1777711501000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/17456916241242734"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,22]]},"references-count":72,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["10.1177\/17456916241242734"],"URL":"https:\/\/doi.org\/10.1177\/17456916241242734","relation":{},"ISSN":["1745-6916","1745-6924"],"issn-type":[{"value":"1745-6916","type":"print"},{"value":"1745-6924","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,22]]}}}