{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T08:08:17Z","timestamp":1788250097714,"version":"build-2803163510"},"reference-count":37,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T00:00:00Z","timestamp":1696464000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100002429","name":"Amgen","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100002429","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:sec id=\"sec001\">\n<jats:title>Background<\/jats:title>\n<jats:p>Online administration of surveys has a number of advantages but can also lead to increased exposure to bad actors (human and non-human bots) who can try to influence the study results or to benefit financially from the survey. We analyze data collected through an online discrete-choice experiment (DCE) survey to evaluate the likelihood that bad actors can affect the quality of the data collected.<\/jats:p>\n<\/jats:sec>\n<jats:sec id=\"sec002\">\n<jats:title>Methods<\/jats:title>\n<jats:p>We developed and fielded a survey instrument that included two sets of DCE questions asking respondents to select their preferred treatments for multiple myeloma therapies. The survey also included questions to assess respondents\u2019 attention while completing the survey and their understanding of the DCE questions. We used a latent-class model to identify a class associated with perverse preferences or high model variance, and the degree to which the quality checks included in the survey were correlated with class membership. Class-membership probabilities for the problematic class were used as weights in a random-parameters logit to recover population-level estimates that minimizes exposure to potential bad actors.<\/jats:p>\n<\/jats:sec>\n<jats:sec id=\"sec003\">\n<jats:title>Results<\/jats:title>\n<jats:p>Results show a significant proportion of respondents provided answers with a high degree of variability consistent with responses from bad actors. We also found that a wide-ranging selection of conditions in the survey screener is more consistent with choice patterns expected from bad actors looking to qualify for the study. The relationship between the number of incorrect answers to comprehension questions and problematic choice patterns peaked around 5 out of 10 questions.<\/jats:p>\n<\/jats:sec>\n<jats:sec id=\"sec004\">\n<jats:title>Conclusions<\/jats:title>\n<jats:p>Our results highlight the need for a robust discussion around the appropriate way to handle bad actors in online preference surveys. While exclusion of survey respondents must be avoided under most circumstances, the impact of \u201cbots\u201d on preference estimates can be significant.<\/jats:p>\n<\/jats:sec>","DOI":"10.1371\/journal.pone.0287766","type":"journal-article","created":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T17:40:02Z","timestamp":1696527602000},"page":"e0287766","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":15,"title":["Did a bot eat your homework? An assessment of the potential impact of bad actors in online administration of preference surveys"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5386-0907","authenticated-orcid":true,"given":"Juan Marcos","family":"Gonzalez","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kiran","family":"Grover","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas W.","family":"Leblanc","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bryce B.","family":"Reeve","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2023,10,5]]},"reference":[{"issue":"3","key":"pone.0287766.ref001","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1007\/s11135-005-8081-8","article-title":"Conducting online surveys","volume":"40","author":"M. Van Selm","year":"2006","journal-title":"Quality and quantity"},{"issue":"2","key":"pone.0287766.ref002","doi-asserted-by":"crossref","first-page":"54","DOI":"10.4018\/IJAVET.2016040105","article-title":"A primer for conducting survey research using MTurk: Tips for the field","volume":"7","author":"S. Chambers","year":"2016","journal-title":"International Journal of Adult Vocational Education and Technology (IJAVET)"},{"key":"pone.0287766.ref003","first-page":"50001","article-title":"Introduction to psychological experiments on the internet","author":"M.H. Birnbaum","year":"2000","journal-title":"Psychological experiments on the Internet"},{"issue":"7","key":"pone.0287766.ref004","first-page":"235","article-title":"Finding the signal in the noise: Minimizing responses from bots and inattentive humans in online research","volume":"42","author":"C. Yarrish","year":"2019","journal-title":"The Behavior Therapist"},{"key":"pone.0287766.ref005","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.npbr.2020.05.003","article-title":"Patient preferences for ketamine-based antidepressant treatments in treatment-resistant depression: Results from a clinical trial and panel","volume":"37","author":"A.O. Fairchild","year":"2020","journal-title":"Neurology, Psychiatry and Brain Research"},{"issue":"5","key":"pone.0287766.ref006","doi-asserted-by":"crossref","first-page":"1975","DOI":"10.1287\/mnsc.2016.2704","article-title":"When waiting to see a doctor is less irritating: Understanding patient preferences and choice behavior in appointment scheduling","volume":"64","author":"N. Liu","year":"2018","journal-title":"Management Science"},{"issue":"5","key":"pone.0287766.ref007","doi-asserted-by":"crossref","first-page":"472","DOI":"10.20982\/tqmp.16.5.p472","article-title":"Got bots? Practical recommendations to protect online survey data from bot attacks","volume":"16","author":"A. Storozuk","year":"2020","journal-title":"The Quantitative Methods for Psychology"},{"issue":"5","key":"pone.0287766.ref008","doi-asserted-by":"crossref","first-page":"2228","DOI":"10.3758\/s13428-018-1103-y","article-title":"Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices","volume":"51","author":"M. Dupuis","year":"2019","journal-title":"Behavior research methods"},{"issue":"6","key":"pone.0287766.ref009","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1007\/s40271-019-00380-z","article-title":"Does device or connection type affect health preferences in online surveys?","volume":"12","author":"J.D. Hartman","year":"2019","journal-title":"The Patient-Patient-Centered Outcomes Research"},{"issue":"1","key":"pone.0287766.ref010","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1007\/s40271-020-00484-x","article-title":"Mobilising the Next Generation of Stated-Preference Studies: the Association of Access Device with Choice Behaviour and Data Quality","volume":"14","author":"C.M. Vass","year":"2021","journal-title":"The Patient-Patient-Centered Outcomes Research"},{"issue":"23","key":"pone.0287766.ref011","doi-asserted-by":"crossref","first-page":"3651","DOI":"10.1002\/cncr.28940","article-title":"Patient preferences in advanced or recurrent ovarian cancer","volume":"120","author":"L.J. Havrilesky","year":"2014","journal-title":"Cancer"},{"issue":"2","key":"pone.0287766.ref012","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.wep.2013.10.001","article-title":"Online, face-to-face and telephone surveys\u2014Comparing different sampling methods in wine consumer research","volume":"2","author":"G. Szolnoki","year":"2013","journal-title":"Wine Economics and Policy"},{"issue":"3","key":"pone.0287766.ref013","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1007\/s10640-015-9876-2","article-title":"Investigating Internet and mail implementation of stated-preference surveys while controlling for differences in sample frames","volume":"64","author":"K.J. Boyle","year":"2016","journal-title":"Environmental and Resource Economics"},{"issue":"3","key":"pone.0287766.ref014","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1093\/pan\/mpm003","article-title":"The effect of survey mode and sampling on inferences about political attitudes and behavior: Comparing the 2000 and 2004 ANES to Internet surveys with nonprobability samples","volume":"15","author":"N. Malhotra","year":"2007","journal-title":"Political Analysis"},{"issue":"7","key":"pone.0287766.ref015","doi-asserted-by":"crossref","first-page":"953","DOI":"10.1016\/j.jval.2017.02.007","article-title":"Impact of survey administration mode on the results of a health-related discrete choice experiment: online and paper comparison","volume":"20","author":"D. Determann","year":"2017","journal-title":"Value in Health"},{"key":"pone.0287766.ref016","volume-title":"Seminars in oncology","author":"D. Kazandjian","year":"2016"},{"issue":"4","key":"pone.0287766.ref017","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/j.jval.2010.11.013","article-title":"Conjoint Analysis Applications in Health\u2014a Checklist: A Report of the ISPOR Good Research Practices for Conjoint Analysis Task Force","volume":"14","author":"J.F.P. Bridges","year":"2011","journal-title":"Value in Health"},{"issue":"1","key":"pone.0287766.ref018","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.jval.2012.08.2223","article-title":"Constructing Experimental Designs for Discrete-Choice Experiments: Report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force","volume":"16","author":"F. Reed Johnson","year":"2013","journal-title":"Value in Health"},{"issue":"2","key":"pone.0287766.ref019","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.jval.2018.07.876","article-title":"The Internal Validity of Discrete Choice Experiment Data: A Testing Tool for Quantitative Assessments","volume":"22","author":"F.R. Johnson","year":"2019","journal-title":"Value in Health"},{"key":"pone.0287766.ref020","unstructured":"Kuhfeld, W.F., Marketing Research Methods in SAS. 2010, SAS Institute Inc."},{"issue":"1","key":"pone.0287766.ref021","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.jval.2017.07.001","article-title":"Conducting a Discrete-Choice Experiment Study Following Recommendations for Good Research Practices: An Application for Eliciting Patient Preferences for Diabetes Treatments","volume":"21","author":"E.M. Janssen","year":"2018","journal-title":"Value in Health"},{"issue":"11","key":"pone.0287766.ref022","doi-asserted-by":"crossref","first-page":"e0166635","DOI":"10.1371\/journal.pone.0166635","article-title":"The prevalence of anemia and moderate-severe anemia in the US population (NHANES 2003\u20132012)","volume":"11","author":"C.H.H. Le","year":"2016","journal-title":"PloS one"},{"key":"pone.0287766.ref023","unstructured":"Desai, D.S. and S. Hajouli, Arrhythmias, in StatPearls [Internet]. 2022, StatPearls Publishing."},{"issue":"7","key":"pone.0287766.ref024","doi-asserted-by":"crossref","first-page":"1582","DOI":"10.1002\/art.39692","article-title":"Updated projected prevalence of self-reported doctor-diagnosed arthritis and arthritis-attributable activity limitation among US adults, 2015\u20132040","volume":"68","author":"J.M. Hootman","year":"2016","journal-title":"Arthritis & rheumatology"},{"issue":"1","key":"pone.0287766.ref025","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1378\/chest.13-0809","article-title":"COPD surveillance\u2014United States, 1999\u20132011","volume":"144","author":"E.S. Ford","year":"2013","journal-title":"Chest"},{"key":"pone.0287766.ref026","unstructured":"CDC, National diabetes statistics report, C. US Department of Health and Human Services, Editor. 2020."},{"key":"pone.0287766.ref027","unstructured":"Fryar, C.D., et al., Hypertension prevalence and control among adults: United States, 2015\u20132016. 2017."},{"issue":"7","key":"pone.0287766.ref028","doi-asserted-by":"crossref","first-page":"473","DOI":"10.7326\/M16-0273","article-title":"Trends in prevalence of chronic kidney disease in the United States","volume":"165","author":"D. Murphy","year":"2016","journal-title":"Annals of internal medicine"},{"issue":"5","key":"pone.0287766.ref029","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1212\/01.wnl.0000252808.97649.21","article-title":"Migraine prevalence, disease burden, and the need for preventive therapy","volume":"68","author":"R.B. Lipton","year":"2007","journal-title":"Neurology"},{"key":"pone.0287766.ref030","unstructured":"Ogden, C.L., et al. Prevalence of obesity among adults and youth: United States, 2011\u20132014. 2015: p. 1\u20138."},{"issue":"4","key":"pone.0287766.ref031","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1016\/j.jval.2016.04.004","article-title":"Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force","volume":"19","author":"A.B. Hauber","year":"2016","journal-title":"Value in health"},{"issue":"3","key":"pone.0287766.ref032","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1007\/s40271-019-00360-3","article-title":"A guide to measuring and interpreting attribute importance","volume":"12","author":"J.M. Gonzalez","year":"2019","journal-title":"The Patient-Patient-Centered Outcomes Research"},{"issue":"2","key":"pone.0287766.ref033","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1016\/j.jtrangeo.2007.07.001","article-title":"Combining RP and SP data: biases in using the nested logit \u2018trick\u2019\u2013contrasts with flexible mixed logit incorporating panel and scale effects","volume":"16","author":"D.A. Hensher","year":"2008","journal-title":"Journal of Transport Geography"},{"key":"pone.0287766.ref034","unstructured":"Risk Factors for Multiple Myeloma. [Website] 2018 [cited 2021 11\/30\/2021]; https:\/\/www.cancer.org\/cancer\/multiple-myeloma\/causes-risks-prevention\/risk-factors.html."},{"key":"pone.0287766.ref035","doi-asserted-by":"crossref","first-page":"108857","DOI":"10.1016\/j.drugalcdep.2021.108857","article-title":"Preferences for research design and treatment of comorbid depression among patients with an opioid use disorder: A cross-sectional discrete choice experiment","volume":"226","author":"G. Bastien","year":"2021","journal-title":"Drug and alcohol dependence"},{"key":"pone.0287766.ref036","doi-asserted-by":"crossref","first-page":"112439","DOI":"10.1016\/j.enpol.2021.112439","article-title":"Effects of rescaling the EU energy label on household preferences for top-rated appliances","volume":"156","author":"C. Faure","year":"2021","journal-title":"Energy Policy"},{"key":"pone.0287766.ref037","unstructured":"Zhang, C. and F. Conrad. Speeding in web surveys: The tendency to answer very fast and its association with straightlining. in Survey research methods. 2014."}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0287766","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T17:40:25Z","timestamp":1696527625000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0287766"}},"subtitle":[],"editor":[{"given":"Stefano","family":"Occhipinti","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2023,10,5]]},"references-count":37,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10,5]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0287766","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,5]]}}}