{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T10:12:18Z","timestamp":1777889538402,"version":"3.51.4"},"reference-count":81,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,2,2]],"date-time":"2024-02-02T00:00:00Z","timestamp":1706832000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Online questionnaires that use crowdsourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial intelligence (AI)-based large language models (LLMs) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested whether text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. While humans were able to correctly identify the authorship of such text above chance level (76% accuracy), their performance was still below what would be required to ensure satisfactory data quality. Researchers currently have to rely on a lack of interest among bad actors to successfully use open-ended responses as a useful tool for ensuring data quality. Automatic AI detection systems are currently completely unusable. If AI submissions of responses become too prevalent, then the costs associated with detecting fraudulent submissions will outweigh the benefits of online questionnaires. Individual attention checks will no longer be a sufficient tool to ensure good data quality. This problem can only be systematically addressed by crowdsourcing platforms. They cannot rely on automatic AI detection systems and it is unclear how they can ensure data quality for their paying clients.<\/jats:p>","DOI":"10.3389\/frobt.2023.1277635","type":"journal-article","created":{"date-parts":[[2024,2,2]],"date-time":"2024-02-02T04:19:42Z","timestamp":1706847582000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Detecting the corruption of online questionnaires by artificial intelligence"],"prefix":"10.3389","volume":"10","author":[{"given":"Benjamin","family":"Lebrun","sequence":"first","affiliation":[]},{"given":"Sharon","family":"Temtsin","sequence":"additional","affiliation":[]},{"given":"Andrew","family":"Vonasch","sequence":"additional","affiliation":[]},{"given":"Christoph","family":"Bartneck","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,2,2]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"15","DOI":"10.17705\/1atrr.00058","article-title":"A replication of beyond the turk: alternative platforms for crowdsourcing behavioral research\u2013sometimes preferable to student groups","volume":"6","author":"Adams","year":"2020","journal-title":"AIS Trans. Replication Res."},{"key":"B2","first-page":"294","article-title":"Captcha: using hard ai problems for security","volume-title":"International conference on the theory and applications of cryptographic techniques","author":"Ahn","year":"2003"},{"key":"B3","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1017\/pan.2023.2","article-title":"Out of one, many: using language models to simulate human samples","volume":"31","author":"Argyle","year":"2023","journal-title":"Polit. Anal."},{"key":"B4","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1007\/s12369-010-0082-7","article-title":"The benefits of interactions with physically present robots over video-displayed agents","volume":"3","author":"Bainbridge","year":"2011","journal-title":"Int. J. Soc. Robotics"},{"key":"B5","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"Baker","year":"2016","journal-title":"Nat. News"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0121595","article-title":"Comparing the similarity of responses received from studies in amazon\u2019s mechanical turk to studies conducted online and with direct recruitment","volume":"10","author":"Bartneck","year":"2015","journal-title":"PLOS ONE"},{"key":"B7","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1016\/j.jesp.2016.02.003","article-title":"Charting the future of social psychology on stormy seas: winners, losers, and recommendations","volume":"66","author":"Baumeister","year":"2016","journal-title":"J. Exp. Soc. Psychol."},{"key":"B8","first-page":"391","article-title":"From characterising three years of hri to methodology and reporting recommendations","volume-title":"The eleventh ACM\/IEEE international conference on human robot interaction (IEEE press), HRI \u201916","author":"Baxter","year":"2016"},{"key":"B9","first-page":"1","article-title":"Towards methodological principles for user studies in human-robot interaction","volume-title":"Test methods and metrics for effective HRI in collaborative human-robot teams workshop","author":"Belhassein","year":"2019"},{"key":"B10","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1007\/978-3-030-42307-0_14","volume-title":"Advice to new human-robot interaction researchers","author":"Belpaeme","year":"2020"},{"key":"B11","doi-asserted-by":"publisher","first-page":"552","DOI":"10.7334\/psicothema2016.383","article-title":"Non-normal data: is anova still a valid option?","volume":"29","author":"Blanca","year":"2017","journal-title":"Psicothema"},{"key":"B12","volume-title":"Chatgpt participates in a computer science exam","author":"Bordt","year":"2023"},{"key":"B13","doi-asserted-by":"crossref","DOI":"10.21203\/rs.3.rs-2895792\/v1","volume-title":"A categorical archive of chatgpt failures","author":"Borji","year":"2023"},{"key":"B14","doi-asserted-by":"publisher","first-page":"2586","DOI":"10.3758\/S13428-018-1035-6","article-title":"Methods to detect low quality data and its implication for psychological research","volume":"50","author":"Buchanan","year":"2018","journal-title":"Behav. Res. Methods"},{"key":"B15","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/1745691610393980","article-title":"Amazon\u2019s mechanical turk: a new source of inexpensive, yet high-quality, data?","volume":"6","author":"Buhrmester","year":"2011","journal-title":"Perspect. Psychol. Sci."},{"key":"B16","volume-title":"Readability revisited: the new Dale-Chall readability formula","author":"Chall","year":"1995"},{"key":"B17","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/978-1-4020-6710-5_9","article-title":"Turing\u2019s test: a philosophical and historical guide","volume-title":"Parsing the turing test: philosophical and methodological issues in the quest for the thinking computer","author":"Copeland","year":"2009"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3148148","article-title":"Quality control in crowdsourcing: a survey of quality attributes, assessment techniques, and assurance actions","volume":"51","author":"Daniel","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1910.11399","article-title":"Comparison of quality indicators in user-generated content using social media and scholarly text","author":"Das","year":"2019"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04805","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2019"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0279720","article-title":"Data quality in online human-subjects research: comparisons between mturk, prolific, cloudresearch, qualtrics, and sona","volume":"18","author":"Douglas","year":"2023","journal-title":"PLOS ONE"},{"key":"B22","first-page":"631","article-title":"The principles of readability","volume":"92627949","author":"Dubay","year":"2004","journal-title":"CA"},{"key":"B23","volume-title":"Smart language: readers, readability, and the grading of text","author":"DuBay","year":"2007"},{"key":"B24","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1145\/602382.602400","article-title":"Some challenges and grand challenges for computational intelligence","volume":"50","author":"Feigenbaum","year":"2003","journal-title":"J. ACM (JACM)"},{"key":"B25","doi-asserted-by":"publisher","first-page":"p221","DOI":"10.1037\/h0057532","article-title":"A new readability yardstick","volume":"32","author":"Flesch","year":"1948","journal-title":"J. Appl. Psychol."},{"key":"B26","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1007\/s12144-015-9403-1","article-title":"Comparing in-person, sona, and mechanical turk measurements of three prejudice-relevant constructs","volume":"36","author":"Gamblin","year":"2017","journal-title":"Curr. Psychol."},{"key":"B27","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1038\/s41746-023-00819-6","article-title":"Comparing scientific abstracts generated by chatgpt to real abstracts with detectors and blinded human reviewers","volume":"6","author":"Gao","year":"2023","journal-title":"npj Digit. Med."},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1101\/2022.12.23.22283901","article-title":"How does chatgpt perform on the medical licensing exams? the implications of large language models for medical education and knowledge assessment","author":"Gilson","year":"2022"},{"key":"B29","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1080\/08897077.2019.1691131","article-title":"Out damn bot, out: recruiting real people into substance use studies on the internet","volume":"41","author":"Godinho","year":"2020","journal-title":"Subst. Abuse"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.13053\/cys-22-1-2882","article-title":"Stylometry-based approach for detecting writing style changes in literary texts","volume":"22","author":"Gomez Adorno","year":"2018","journal-title":"Comput. Sist."},{"key":"B31","doi-asserted-by":"publisher","first-page":"2841","DOI":"10.1007\/s11135-021-01252-1","article-title":"Ensuring survey research data integrity in the era of internet bots","volume":"56","author":"Griffin","year":"2022","journal-title":"Qual. Quantity"},{"key":"B32","volume-title":"The technique of clear writing","author":"Gunning","year":"1952"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2301.07597","article-title":"How close is chatgpt to human experts? comparison corpus, evaluation, and detection","author":"Guo","year":"2023"},{"key":"B34","first-page":"1","article-title":"Evaluating large language models in generating synthetic hci research data: a case study","volume-title":"CHI \u201923: CHI conference on human factors in computing systems","author":"H\u00e4m\u00e4l\u00e4inen","year":"2023"},{"key":"B35","doi-asserted-by":"publisher","first-page":"912","DOI":"10.1177\/0013164415627349","article-title":"Survey satisficing inflates reliability and validity measures: an experimental comparison of college and amazon mechanical turk samples","volume":"76","author":"Hamby","year":"2016","journal-title":"Educ. Psychol. Meas."},{"key":"B36","first-page":"13","article-title":"Social psychology and human-robot interaction: an uneasy marriage","author":"Irfan","year":"2018"},{"key":"B37","doi-asserted-by":"publisher","first-page":"196","DOI":"10.1207\/s15327957pspr0203_4","article-title":"Harking: hypothesizing after the results are known","volume":"2","author":"Kerr","year":"1998","journal-title":"Personality Soc. Psychol. Rev. official J. Soc. Personality Soc. Psychol. Inc"},{"key":"B38","doi-asserted-by":"crossref","DOI":"10.21236\/ADA006655","volume-title":"Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel","author":"Kincaid","year":"1975"},{"key":"B39","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1002\/acp.2350050305","article-title":"Response strategies for coping with the cognitive demands of attitude measures in surveys","volume":"5","author":"Krosnick","year":"1991","journal-title":"Appl. Cogn. Psychol."},{"key":"B40","volume-title":"The global opportunity in online outsourcing","author":"Kuek","year":"2015"},{"key":"B41","volume-title":"Stylometric detection of ai-generated text in twitter timelines","author":"Kumarage","year":"2023"},{"key":"B42","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit. Health"},{"key":"B43","doi-asserted-by":"publisher","first-page":"101386","DOI":"10.1016\/j.jenvp.2019.101386","article-title":"How much distance do humans keep toward robots? literature review, meta-analysis, and theoretical considerations on personal space in human-robot interaction","volume":"68","author":"Leichtmann","year":"","journal-title":"J. Environ. Psychol."},{"key":"B44","doi-asserted-by":"publisher","first-page":"1013","DOI":"10.1007\/s12369-020-00688-z","article-title":"Is the social desirability effect in human\u2013robot interaction overestimated? a conceptual replication study indicates less robust effects","volume":"13","author":"Leichtmann","year":"","journal-title":"Int. J. Soc. Robotics"},{"key":"B45","doi-asserted-by":"publisher","first-page":"838116","DOI":"10.3389\/frobt.2022.838116","article-title":"Crisis ahead? why human-robot interaction user studies may have replicability problems and directions for improvement","volume":"9","author":"Leichtmann","year":"2022","journal-title":"Front. Robotics AI"},{"key":"B46","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1016\/j.ijhcs.2015.01.001","article-title":"The benefit of being physically present: a survey of experimental works comparing copresent robots, telepresent robots and virtual agents","volume":"77","author":"Li","year":"2015","journal-title":"Int. J. Human-Computer Stud."},{"key":"B47","doi-asserted-by":"publisher","first-page":"570","DOI":"10.1002\/asi.24750","article-title":"Chatgpt and a new academic reality: artificial intelligence-written research papers and the ethics of the large language models in scholarly publishing","volume":"74","author":"Lund","year":"2023","journal-title":"J. Assoc. Inf. Sci. Technol."},{"key":"B48","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1609\/hcomp.v1i1.13075","article-title":"Volunteering versus work for pay: incentives and tradeoffs in crowdsourcing","volume":"1","author":"Mao","year":"2013","journal-title":"Proc. AAAI Conf. Hum. Comput. Crowdsourcing"},{"key":"B49","first-page":"639","article-title":"Smog grading-a new readability formula","volume":"12","author":"Mc Laughlin","year":"1969","journal-title":"J. Read."},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2301.13852","article-title":"Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text","author":"Mitrovic","year":"2023"},{"key":"B51","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1023\/A:1011218925467","article-title":"The status and future of the turing test","volume":"11","author":"Moor","year":"2001","journal-title":"Minds Mach."},{"key":"B52","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1037\/0003-066X.59.3.150","article-title":"Psychological testing on the internet: new problems, old issues","volume":"59","author":"Naglieri","year":"2004","journal-title":"Am. Psychol."},{"key":"B53","doi-asserted-by":"crossref","DOI":"10.2139\/ssrn.4413305","volume-title":"Putting chatgpt\u2019s medical advice to the (turing) test","author":"Nov","year":"2023"},{"key":"B54","doi-asserted-by":"publisher","first-page":"106547","DOI":"10.1016\/j.chb.2020.106547","article-title":"Towards prosocial design: a scoping review of the use of robots and virtual agents to trigger prosocial behaviour","volume":"114","author":"Oliveira","year":"2020","journal-title":"Comput. Hum. Behav."},{"key":"B55","doi-asserted-by":"publisher","first-page":"aac4716","DOI":"10.1126\/science.aac4716","article-title":"PSYCHOLOGY. Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","journal-title":"Science"},{"key":"B56","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1016\/j.jesp.2017.01.006","article-title":"Beyond the turk: alternative platforms for crowdsourcing behavioral research","volume":"70","author":"Peer","year":"2017","journal-title":"J. Exp. Soc. Psychol."},{"key":"B57","doi-asserted-by":"publisher","first-page":"1226","DOI":"10.1126\/science.1213847","article-title":"Reproducible research in computational science","volume":"334","author":"Peng","year":"2011","journal-title":"Science"},{"key":"B58","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1145\/1228716.1228736","article-title":"Comparing a computer agent with a humanoid robot","volume-title":"Proceedings of the ACM\/IEEE international conference on human-robot interaction","author":"Powers","year":"2007"},{"key":"B59","doi-asserted-by":"publisher","first-page":"e23021","DOI":"10.2196\/23021","article-title":"Threats of bots and other bad actors to data quality following research participant recruitment through social media: cross-sectional questionnaire","volume":"22","author":"Pozzar","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"B60","doi-asserted-by":"publisher","first-page":"5783","DOI":"10.3390\/app13095783","article-title":"Chatgpt for education and research: opportunities, threats, and strategies","volume":"13","author":"Rahman","year":"2023","journal-title":"Appl. Sci."},{"key":"B61","doi-asserted-by":"publisher","DOI":"10.37074\/jalt.2023.6.1.9","article-title":"Chatgpt: bullshit spewer or the end of traditional assessments in higher education?","volume":"6","author":"Rudolph","year":"2023","journal-title":"J. Appl. Learn. Teach."},{"key":"B62","volume-title":"An empirical study and evaluation of modern captchas","author":"Searles","year":"2023"},{"key":"B63","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1145\/502585.502695","article-title":"A statistical model for scientific readability","volume-title":"Proceedings of the tenth international conference on information and knowledge management","author":"Si","year":"2001"},{"key":"B64","doi-asserted-by":"publisher","first-page":"1359","DOI":"10.1177\/0956797611417632","article-title":"False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant","volume":"22","author":"Simmons","year":"2011","journal-title":"Psychol. Sci."},{"key":"B65","first-page":"1","article-title":"Automated readability index","author":"Smith","year":"1967","journal-title":"Amrl Tr."},{"key":"B66","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1145\/3319502.3374783","article-title":"A three-site reproduction of the joint simon effect with the nao robot","volume-title":"Proceedings of the 2020 ACM\/IEEE international conference on human-robot interaction","author":"Strait","year":"2020"},{"key":"B67","volume-title":"Chatgpt: the end of online exam integrity?","author":"Susnjak","year":"2022"},{"key":"B68","doi-asserted-by":"publisher","first-page":"111","DOI":"10.5334\/irsp.66","article-title":"Replicability crisis in social psychology: looking at the past to find new pathways for the future","volume":"30","author":"Swiatkowski","year":"2017","journal-title":"Int. Rev. Soc. Psychol."},{"key":"B69","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1111\/jlme.12200","article-title":"Detecting, preventing, and responding to \u201cfraudsters\u201d in internet research: ethics and tradeoffs","volume":"43","author":"Teitcher","year":"2015","journal-title":"J. Law Med. Ethics"},{"key":"B70","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1016\/j.obhdp.2020.10.015","article-title":"Open science and reform practices in organizational behavior research over time (2011 to 2019)","volume":"162","author":"Tenney","year":"2021","journal-title":"Organ. Behav. Hum. Decis. Process."},{"key":"B71","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1007\/978-3-319-47665-0_44","article-title":"Physical vs. virtual agent embodiment and effects on social interaction","volume":"10011","author":"Thellman","year":"2016","journal-title":"Int. Conf. Intelligent Virtual Agents"},{"key":"B72","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.13971","article-title":"Llama: open and efficient foundation language models","author":"Touvron","year":"2023"},{"key":"B73","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1093\/oso\/9780198250791.003.0020","article-title":"Can automatic calculating machines be said to think? (1952)","volume-title":"The essential turing","author":"Turing","year":"2004"},{"key":"B74","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1093\/mind\/LIX.236.433","article-title":"I.\u2014COMPUTING MACHINERY AND INTELLIGENCE","author":"Turing","year":"1950","journal-title":"Mind LIX"},{"key":"B75","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1093\/oso\/9780198250791.003.0016","article-title":"Intelligent machinery","volume-title":"The essential turing","author":"Turing","year":"2004"},{"key":"B76","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1145\/3434073.3444652","article-title":"Challenges and opportunities for replication science in hri: a case study in human-robot trust","volume-title":"Proceedings of the 2021 ACM\/IEEE international conference on human-robot interaction","author":"Ullman","year":"2021"},{"key":"B77","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1145\/3029798.3038423","article-title":"Human-robot trust: just a button press away","volume-title":"Proceedings of the companion of the 2017 ACM\/IEEE international conference on human-robot interaction","author":"Ullman","year":"2017"},{"key":"B78","first-page":"45","article-title":"Chatgpt and academic integrity concerns: detecting artificial intelligence generated content","volume":"3","author":"Uzun","year":"2023","journal-title":"Lang. Educ. Technol."},{"key":"B79","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/fcery","article-title":"When people reject free money: phantom costs and the psychology of economic exchange","author":"Vonasch","year":"2022"},{"key":"B80","doi-asserted-by":"publisher","first-page":"100206","DOI":"10.1016\/j.chbr.2022.100206","article-title":"Response rates of online surveys in published research: a meta-analysis","volume":"7","author":"Wu","year":"2022","journal-title":"Comput. Hum. Behav. Rep."},{"key":"B81","first-page":"235","article-title":"Finding the signal in the noise: minimizing responses from bots and inattentive humans in online research","volume":"42","author":"Yarrish","year":"2019","journal-title":"Behav. Ther."}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1277635\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,9]],"date-time":"2024-11-09T21:40:06Z","timestamp":1731188406000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1277635\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,2]]},"references-count":81,"alternative-id":["10.3389\/frobt.2023.1277635"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1277635","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,2]]},"article-number":"1277635"}}