{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T20:24:35Z","timestamp":1780431875742,"version":"3.54.1"},"reference-count":86,"publisher":"SAGE Publications","issue":"10","license":[{"start":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T00:00:00Z","timestamp":1699920000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006751","name":"U.S. Army","doi-asserted-by":"crossref","award":["Advanced Strategic Planning and Policy Program"],"award-info":[{"award-number":["Advanced Strategic Planning and Policy Program"]}],"id":[{"id":"10.13039\/100006751","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Hum Factors"],"published-print":{"date-parts":[[2024,10]]},"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We test the effects of three aids on individuals\u2019 ability to detect social bots among Twitter personas: a bot indicator score, a training video, and a warning.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Detecting social bots can prevent online deception. We use a simulated social media task to evaluate three aids.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Method<\/jats:title>\n                    <jats:p>Lay participants judged whether each of 60 Twitter personas was a human or social bot in a simulated online environment, using agreement between three machine learning algorithms to estimate the probability of each persona being a bot. Experiment 1 compared a control group and two intervention groups, one provided a bot indicator score for each tweet; the other provided a warning about social bots. Experiment 2 compared a control group and two intervention groups, one receiving the bot indicator scores and the other a training video, focused on heuristics for identifying social bots.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The bot indicator score intervention improved predictive performance and reduced overconfidence in both experiments. The training video was also effective, although somewhat less so. The warning had no effect. Participants rarely reported willingness to share content for a persona that they labeled as a bot, even when they agreed with it.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Informative interventions improved social bot detection; warning alone did not.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Application<\/jats:title>\n                    <jats:p>We offer an experimental testbed and methodology that can be used to evaluate and refine interventions designed to reduce vulnerability to social bots. We show the value of two interventions that could be applied in many settings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/00187208231210145","type":"journal-article","created":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T14:02:21Z","timestamp":1699970541000},"page":"2323-2344","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":2,"title":["Improving Social Bot Detection Through Aid and Training"],"prefix":"10.1177","volume":"66","author":[{"given":"Ryan","family":"Kenny","sequence":"first","affiliation":[{"name":"United States Army, Fayetteville, NC, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3030-6874","authenticated-orcid":false,"given":"Baruch","family":"Fischhoff","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Davis","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5325-3798","authenticated-orcid":false,"given":"Casey","family":"Canfield","sequence":"additional","affiliation":[{"name":"Missouri University of Science and Technology, Rolla, MO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,11,14]]},"reference":[{"key":"bibr1-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v6i1.14236"},{"key":"bibr2-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1080\/01605682.2019.1595193"},{"key":"bibr3-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1007\/s11747-019-00695-1"},{"key":"bibr4-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2008.06.001"},{"key":"bibr5-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1177\/1745691620986135"},{"key":"bibr6-00187208231210145","unstructured":"Beskow D., Carley K. (2018a). Bot-hunter: A tiered approach to detecting and characterizing automated activity on Twitter. Conference: SBP-BRiMS: International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, Washington, USA, July 2018."},{"key":"bibr7-00187208231210145","volume-title":"International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation","author":"Beskow D.","year":"2018"},{"key":"bibr8-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1207\/s15327957pspr1003_2"},{"key":"bibr9-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1007\/s10979-007-9110-z"},{"key":"bibr10-00187208231210145","unstructured":"Botometer (2021). CNetS. Retrieved March 22, 2021, from https:\/\/cnets.indiana.edu\/blog\/tag\/botometer\/"},{"key":"bibr11-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1212729110"},{"key":"bibr12-00187208231210145","doi-asserted-by":"publisher","DOI":"10.2196\/28916"},{"key":"bibr13-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1002\/bdm.2155"},{"key":"bibr14-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1177\/0018720816665025"},{"key":"bibr15-00187208231210145","doi-asserted-by":"crossref","unstructured":"Chu Z., Gianvecchio S., Wang H., Jajodia S. (2010). Who is tweeting on Twitter: Human, bot, or cyborg? In Proceedings of the 26th Annual Computer Security Applications Conference (pp. 21\u201330). Austin, Texas, USA, 6\u201310 December 2010.","DOI":"10.1145\/1920261.1920265"},{"key":"bibr16-00187208231210145","volume-title":"Elon Musk\u2019s new enemy: An Army of good bots","author":"Cohen B.","year":"2023"},{"key":"bibr17-00187208231210145","volume-title":"How data poisoning attacks corrupt machine learning models","volume":"14","author":"Constantin L.","year":"2022"},{"issue":"1","key":"bibr18-00187208231210145","first-page":"58","volume":"13","author":"Cook D.","year":"2014","journal-title":"Journal of Information Warfare"},{"key":"bibr19-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1145\/3409116"},{"key":"bibr20-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1145\/2872518.2889302"},{"key":"bibr21-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.34.7.571"},{"key":"bibr22-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.3.2.186"},{"key":"bibr23-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1088\/1748-9326\/10\/12\/124009"},{"key":"bibr24-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-2389.2011.00548.x"},{"key":"bibr25-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1037\/xge0000033"},{"key":"bibr26-00187208231210145","doi-asserted-by":"crossref","unstructured":"Downs J. S., Holbrook M. B., Sheng S., Cranor L. F. (2010). Are your participants gaming the system? Screening mechanical Turk workers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2399\u20132402). Atlanta, GA, USA, 10\u201315 April 2010.","DOI":"10.1145\/1753326.1753688"},{"key":"bibr27-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2019.08.002"},{"key":"bibr28-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1002\/wcs.1382"},{"issue":"4","key":"bibr29-00187208231210145","first-page":"1","volume":"54","author":"Eyal P.","year":"2021","journal-title":"Behavior Research Methods"},{"key":"bibr30-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1145\/2818717"},{"key":"bibr31-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809477.032"},{"key":"bibr32-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00385-9"},{"key":"bibr33-00187208231210145","volume-title":"Package \u2018arm\u2019: Data analysis using regression and multilevel\/hierarchical models","author":"Gelman A.","year":"2016"},{"key":"bibr34-00187208231210145","doi-asserted-by":"publisher","DOI":"10.5465\/annals.2018.0057"},{"key":"bibr35-00187208231210145","volume-title":"Signal detection theory and psychophysics","author":"Green D. M.","year":"1966"},{"key":"bibr36-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1257\/aer.100.2.267"},{"key":"bibr37-00187208231210145","unstructured":"Harari Y. (2023, March 27). You can have the blue pill or the red pill and we\u2019re out of blue pills (p. 18). New York Times."},{"key":"bibr38-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1177\/0093650214534974"},{"key":"bibr39-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1109\/SocialCom-PASSAT.2012.129"},{"key":"bibr40-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814547570"},{"key":"bibr41-00187208231210145","volume-title":"Learning user latent attributes on social media","author":"Huang B.","year":"2020"},{"key":"bibr42-00187208231210145","unstructured":"Huang B., Carley K. M. (2020). Disinformation and misinformation on twitter during the novel coronavirus outbreak. ArXiv Preprint arXiv:2006.04278."},{"key":"bibr43-00187208231210145","doi-asserted-by":"crossref","unstructured":"Jagielski M., Oprea A., Biggio B., Liu C., Nita-Rotaru C., Li B. (2018, May). Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP) (pp. 19\u201335). San Francisco, CA, USA, 20\u201324 May 2018.","DOI":"10.1109\/SP.2018.00057"},{"key":"bibr44-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21149"},{"key":"bibr45-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1037\/a0016755"},{"key":"bibr46-00187208231210145","unstructured":"Kats D. (2022). Introducing BotSight: A new tool to detect bots on twitter in real-time. https:\/\/www.nortonlifelock.com\/blogs\/norton-labs\/botsight-tool-detect-twitter-bots"},{"key":"bibr47-00187208231210145","first-page":"187208211072642","author":"Kenny R.","journal-title":"Human Factors"},{"issue":"3","key":"bibr48-00187208231210145","first-page":"853","volume":"34","author":"Kluttz D. N.","year":"2019","journal-title":"Berkeley Technology Law Journal"},{"key":"bibr49-00187208231210145","doi-asserted-by":"crossref","unstructured":"Lee C., Kwak H., Park H., Moon S. (2010). Finding influentials based on the temporal order of information adoption in twitter. In Proceedings of the 19th international conference on world wide web (pp. 1137\u20131138). Raleigh, NC, USA, 26\u201330 April 2010. for detailed perspectives on influence in social media environments.","DOI":"10.1145\/1772690.1772842"},{"key":"bibr50-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50_30392"},{"key":"bibr51-00187208231210145","doi-asserted-by":"crossref","unstructured":"Lerman K., Jain P., Ghosh R., Kang J. H., Kumaraguru P. (2013, May). Limited attention and centrality in social networks. In 2013 IEEE International Conference on Social Intelligence and Technology (pp. 80\u201389). State College, PA, USA, 08\u201310 May 2013.","DOI":"10.1109\/SOCIETY.2013.11"},{"key":"bibr52-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809477.023"},{"key":"bibr53-00187208231210145","doi-asserted-by":"publisher","DOI":"10.4324\/9781410611147"},{"key":"bibr54-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-1210-7_13"},{"key":"bibr55-00187208231210145","doi-asserted-by":"publisher","DOI":"10.4324\/9781410611949"},{"key":"bibr56-00187208231210145","unstructured":"Meaker M. (2022). This student\u2019s side project will help to decide Musk vs. Twitter. https:\/\/www.wired.com\/story\/musk-twitter-botometer\/ (accessed 6\/12\/23)."},{"key":"bibr57-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1037\/11281-000"},{"key":"bibr58-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-6924.2009.01142.x"},{"key":"bibr59-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.j5416"},{"key":"bibr60-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1805450115"},{"key":"bibr61-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2020.102250"},{"key":"bibr62-00187208231210145","volume-title":"Uncovering coordinated networks on social media","author":"Pacheco D.","year":"2020"},{"key":"bibr63-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbef.2017.12.004"},{"key":"bibr64-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4684-2529-1_26"},{"key":"bibr65-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1080\/10864415.2003.11044275"},{"key":"bibr66-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2017.01.006"},{"key":"bibr67-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2018.06.011"},{"key":"bibr68-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2016.04.003"},{"key":"bibr69-00187208231210145","unstructured":"Sarno D. M., Black J. Who gets caught in the web of lies? Understanding susceptibility to phishing emails, fake news headlines, and scam text messages. Human Factors, 187208231173263. (in press). https:\/\/doi.org\/10.1177\/0018720823117326"},{"key":"bibr70-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1037\/xap0000410"},{"key":"bibr71-00187208231210145","unstructured":"Satariano A., Mozur. (2023, February 9). The people onscreen are fake. The disinformation is real (p. 3). New York Times."},{"key":"bibr72-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-06930-7"},{"key":"bibr73-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2021.102577"},{"issue":"4","key":"bibr74-00187208231210145","first-page":"394","volume":"61","author":"Simon H. A.","year":"1973","journal-title":"American Scientist"},{"key":"bibr75-00187208231210145","doi-asserted-by":"publisher","DOI":"10.3758\/BF03207704"},{"key":"bibr76-00187208231210145","unstructured":"Stieglitz S., Brachten F., Ross B., Jung A. K. (2017). Do social bots dream of electric sheep? A categorisation of social media bot accounts. ArXiv Preprint arXiv:1710.04044."},{"key":"bibr77-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0221-y"},{"key":"bibr78-00187208231210145","unstructured":"Thompson S. A. (2023, May 20). A. I.generated content discovered on news sites, content farms and product reviews (p. 5). New York Times."},{"key":"bibr79-00187208231210145","unstructured":"Timberg C., Dwoskin E. (2018). Twitter is sweeping out fake accounts like never before, putting user growth at risk. Retrieved November 3, 2020 https:\/\/www.washingtonpost.com\/technology\/2018\/07\/06\/twitter-is-sweeping-out-fake-accounts-like-never-before-putting-user-growth-risk\/"},{"key":"bibr80-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1007\/s42001-020-00087-4"},{"key":"bibr81-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14871"},{"key":"bibr82-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1109\/IRI.2013.6642447"},{"key":"bibr83-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1518\/001872008X312152"},{"key":"bibr84-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1123\/jsep.2020-0040"},{"key":"bibr86-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1145\/3373464.3373475"},{"key":"bibr87-00187208231210145","doi-asserted-by":"publisher","DOI":"10.1007\/s42001-022-00177-5"}],"container-title":["Human Factors: The Journal of the Human Factors and Ergonomics Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00187208231210145","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/00187208231210145","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00187208231210145","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00187208231210145","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T07:54:00Z","timestamp":1777449240000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/00187208231210145"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,14]]},"references-count":86,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2024,10]]}},"alternative-id":["10.1177\/00187208231210145"],"URL":"https:\/\/doi.org\/10.1177\/00187208231210145","relation":{},"ISSN":["0018-7208","1547-8181"],"issn-type":[{"value":"0018-7208","type":"print"},{"value":"1547-8181","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,14]]}}}