{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T10:54:40Z","timestamp":1790679280211,"version":"4.1.0"},"reference-count":45,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,1,31]],"date-time":"2022-01-31T00:00:00Z","timestamp":1643587200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Social Science Foundation Youth Project \u201cResearch on the Influence of Social bots on the Climate of False Opinions in International Communication\u201d","award":["21CXW028"],"award-info":[{"award-number":["21CXW028"]}]},{"name":"Major projects of Jiangsu Provincial Department of Education \u201cResearch on the Influence of Social Bots on the Climate of False Opinions in International Communication and its Regulation\u201d","award":["2021SJZDA151"],"award-info":[{"award-number":["2021SJZDA151"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>During the COVID-19 pandemic, social media served as an important channel for the public to obtain health information and disseminate opinions when offline communication was severely hindered. Yet the emergence of social bots influencing social media conversations about public health threats will require researchers and practitioners to develop new communication strategies considering their influence. So far, little is known as to what extent social bots have been involved in COVID-19 vaccine-related discussions and debates on social media. This work selected a period of nearly 9 months after the approval of the first COVID-19 vaccines to detect social bots and performed high-frequency word analysis for both social bot-generated and human-generated tweets, thus working out the extent to which social bots participated in the discussion on the COVID-19 vaccine on Twitter and their participation features. Then, a textual analysis was performed on the content of tweets. The findings revealed that 8.87% of the users were social bots, with 11% of tweets in the corpus. Besides, social bots remained active over three periods. High-frequency words in the discussions of social bots and human users on vaccine topics were similar within the three peaks of discourse.<\/jats:p>","DOI":"10.3390\/ijerph19031651","type":"journal-article","created":{"date-parts":[[2022,2,1]],"date-time":"2022-02-01T09:59:21Z","timestamp":1643709561000},"page":"1651","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Social Bots\u2019 Involvement in the COVID-19 Vaccine Discussions on Twitter"],"prefix":"10.3390","volume":"19","author":[{"given":"Menghan","family":"Zhang","sequence":"first","affiliation":[{"name":"Centre for Chinese Urbanization Studies of Soochow University & Collaborative Innovation Center for New Urbanization and Social Governance of Universities, Suzhou 215006, China"},{"name":"School of Communication, Soochow University, Suzhou 215123, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xue","family":"Qi","sequence":"additional","affiliation":[{"name":"School of Communication, Soochow University, Suzhou 215123, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ze","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Communication, Soochow University, Suzhou 215123, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6066-4130","authenticated-orcid":false,"given":"Jun","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Communication, University of Copenhagen, DK-2300 Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1080\/10584609.2020.1718257","article-title":"Troll factories: Manufacturing specialized disinformation on Twitter","volume":"37","author":"Linvill","year":"2020","journal-title":"Political Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2586","DOI":"10.1080\/21645515.2020.1780846","article-title":"Social media and vaccine hesitancy: New updates for the era of COVID-19 and globalized infectious diseases","volume":"16","author":"Puri","year":"2020","journal-title":"Hum. Vaccines Immunother."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1586\/14760584.2015.964212","article-title":"Vaccine hesitancy, vaccine refusal and the anti-vaccine movement: Influence, impact and implications","volume":"14","author":"Vivion","year":"2015","journal-title":"Expert Rev. Vaccines"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Benis, A., Sei Dm Ann, A., and Ashkenazi, S. (2021). Reasons for taking the COVID-19 vaccines by us social media users. Vaccines, 9.","DOI":"10.3390\/vaccines9040315"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.jclinepi.2020.06.028","article-title":"Social media can have an impact on how we manage and investigate the COVID-19 pandemic","volume":"127","year":"2020","journal-title":"J. Clin. Epidemiol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1007\/s00296-019-04493-4","article-title":"Social media for research, education and practice in rheumatology","volume":"40","author":"Zimba","year":"2020","journal-title":"Rheumatol. Int."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1145\/2818717","article-title":"The rise of social bots","volume":"59","author":"Ferrara","year":"2016","journal-title":"Commun. ACM"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1378","DOI":"10.2105\/AJPH.2018.304567","article-title":"Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate","volume":"108","author":"Broniatowski","year":"2018","journal-title":"Am. J. Public Health"},{"key":"ref_9","first-page":"1","article-title":"Examining emergent communities and social bots within the polarized online vaccination debate in Twitter","volume":"5","author":"Yuan","year":"2019","journal-title":"Soc. Media Soc."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Shi, W., Liu, D., Yang, J., Zhang, J., and Su, J. (2020). Social bots\u2019 sentiment engagement in health emergencies: A topic-based analysis of the COVID-19 pandemic discussions on Twitter. Int. J. Environ. Res. Public Health, 17.","DOI":"10.3390\/ijerph17228701"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Varol, O., Ferrara, E., Davis, C., Menczer, F., and Flammini, A. (2017, January 15\u201318). Online human-bot interactions: Detection, estimation, and characterization. Proceedings of the International AAAI Conference on Web and Social Media, Montreal, QC, Canada.","DOI":"10.1609\/icwsm.v11i1.14871"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.ijinfomgt.2017.12.002","article-title":"Social media analytics\u2014Challenges in topic discovery, data collection, and data preparation","volume":"39","author":"Stieglitz","year":"2018","journal-title":"Int. J. Inf. Manag."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Jolley, D., and Douglas, K.M. (2014). The effects of anti-vaccine conspiracy theories on vaccination intentions. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0089177"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1093\/her\/cys110","article-title":"The effect of falsely balanced reporting of the autism\u2013vaccine controversy on vaccine safety perceptions and behavioral intentions","volume":"28","author":"Dixon","year":"2013","journal-title":"Health Educ. Res."},{"key":"ref_15","unstructured":"Kim, A. (2020, September 10). Nearly Half of the Twitter Accounts Discussing \u2018Reopening America\u2019 May Be Bots, Researchers Say. CNN. 22 May 2020. Available online: https:\/\/edition.cnn.com\/2020\/05\/22\/tech\/twitter-bots-trnd\/index.html."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1285","DOI":"10.1038\/s41562-020-00994-6","article-title":"Assessing the risks of \u2018infodemics\u2019 in response to COVID-19 epidemics","volume":"4","author":"Gallotti","year":"2020","journal-title":"Nat. Hum. Behav."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.bushor.2009.09.003","article-title":"Users of the world, unite! The challenges and opportunities of social media","volume":"53","author":"Kaplan","year":"2010","journal-title":"Bus. Horiz."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Charles-Smith, L.E., Reynolds, T.L., Cameron, M.A., Conway, M., Lau, E.H.Y., Olsen, J.M., Pavlin, J.A., Shigematsu, M., Streichert, L.C., and Suda, K.J. (2015). Using social media for actionable disease surveillance and outbreak management: A systematic literature review. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0139701"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3727","DOI":"10.1016\/j.vaccine.2012.02.025","article-title":"Opportunities and challenges of web 2.0 for vaccination decisions","volume":"30","author":"Betsch","year":"2012","journal-title":"Vaccine"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"17","DOI":"10.7812\/TPP\/17-138","article-title":"Sentiment, contents, and retweets: A study of two vaccine-related Twitter datasets","volume":"22","author":"Blankenship","year":"2018","journal-title":"Perm. J."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1111\/1468-5973.12193","article-title":"Sense\u2014Making in social media during extreme events","volume":"26","author":"Stieglitz","year":"2018","journal-title":"J. Contingencies Crisis Manag."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1277","DOI":"10.1007\/s13278-012-0079-3","article-title":"Social media and political communication: A social media analytics framework","volume":"3","author":"Stieglitz","year":"2013","journal-title":"Soc. Netw. Anal. Min."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3778","DOI":"10.1016\/j.vaccine.2011.11.112","article-title":"Anti-vaccine activists, Web 2.0, and the postmodern paradigm\u2013An overview of tactics and tropes used online by the anti-vaccination movement","volume":"30","author":"Kata","year":"2012","journal-title":"Vaccine"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e20550","DOI":"10.2196\/20550","article-title":"Twitter discussions and emotions about the COVID-19 pandemic: Machine learning approach","volume":"22","author":"Xue","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_25","unstructured":"(2021, October 26). The Anti-Vaxx Industry: How Big Tech Powers and Profits from Anti-Vaccine Misinformation.Centre for Countering Digital Hate. Available online: https:\/\/www.counterhate.com\/anti-vaxx-industry."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"101712","DOI":"10.1016\/j.tele.2021.101712","article-title":"Exploring public perceptions of the COVID-19 vaccine online from a cultural perspective: Semantic network analysis of two social media platforms in the United States and China","volume":"65","author":"Luo","year":"2021","journal-title":"Telemat. Inform."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e26627","DOI":"10.2196\/26627","article-title":"Artificial intelligence-enabled analysis of public attitudes on Facebook and Twitter toward COVID-19 vaccines in the United Kingdom and the United States: Observational study","volume":"23","author":"Hussain","year":"2021","journal-title":"J. Med. Internet Res."},{"key":"ref_28","first-page":"1","article-title":"Not just conspiracy theories: Vaccine opponents and proponents add to the COVID-19; \u2018Infodemic\u2019on Twitter","volume":"19","author":"Jamison","year":"2020","journal-title":"Harv. Kennedy Sch. Misinformation Rev."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"La, V.P., Pham, T.H., Ho, M.T., Nguyen, M.H., PNguyen, K.L., Vuong, T.T., Nguyen, H.-K.T., Tran, T., Khuc, Q., and Ho, M.-T. (2020). Policy response, social media and science journalism for the sustainability of the public health system amid the COVID-19 outbreak: The Vietnam lessons. Sustainability, 12.","DOI":"10.31235\/osf.io\/cfw8x"},{"key":"ref_30","first-page":"1","article-title":"The causes and consequences of COVID-19 misperceptions: Understanding the role of news and social media","volume":"21","author":"Bridgman","year":"2020","journal-title":"Harv. Kennedy Sch. Misinformation Rev."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1093\/pubmed\/fdaa049","article-title":"World leaders usage of Twitter in response to the COVID-19 pandemic: A content analysis","volume":"42","author":"Rufai","year":"2020","journal-title":"J. Public Health"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Abokhodair, N., Yoo, D., and McDonald, D.W. (2015, January 14\u201318). Dissecting a social botnet: Growth, content, and influence in Twitter. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, Vancouver, BC, Canada.","DOI":"10.1145\/2675133.2675208"},{"key":"ref_33","unstructured":"Stieglitz, S., Brachten, F., Ross, B., and Jung, A.K. (2017, January 4\u20136). Do social bots dream of electric sheep? A categorization of social media bot accounts. Proceedings of the Australasian Conference on Information Systems, Hobart, Australia."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-018-06930-7","article-title":"The spread of low-credibility content by social bots","volume":"9","author":"Shao","year":"2018","journal-title":"Nat. Commun."},{"key":"ref_35","first-page":"4882","article-title":"Automation, algorithms, and politics | political communication, computational propaganda, and autonomous agents\u2014Introduction","volume":"10","author":"Woolley","year":"2016","journal-title":"Int. J. Commun."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1177\/0894439317734157","article-title":"The Brexit botnet and user-generated hyperpartisan news","volume":"37","author":"Bastos","year":"2019","journal-title":"Soc. Sci. Comput. Rev."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1111\/eufm.12245","article-title":"Social media bots and stock markets","volume":"26","author":"Fan","year":"2020","journal-title":"Eur. Financ. Manag."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"357","DOI":"10.2105\/AJPH.2019.305461","article-title":"Cannabis surveillance with Twitter data: Emerging topics and social bots","volume":"110","author":"Allem","year":"2020","journal-title":"Am. J. Public Health"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Badawy, A., Lerman, K., and Ferrara, E. (2019, January 13\u201317). Who falls for online political manipulation?. Proceedings of the Companion Proceedings of The 2019 World Wide Web Conference, Association for Computing Machinery, San Francisco, CA, USA.","DOI":"10.1145\/3308560.3316494"},{"key":"ref_40","unstructured":"Statista (2021, October 13). The First Seven COVID-19 Vaccines to Have Been Approved Internationally Received Orders for over 15 Billion Doses Altogether. Statista. 15 September 2021. Available online: https:\/\/www-statista-com.ezproxy.gavilan.edu\/study\/86257\/covid19-vaccination-status-quo\/."},{"key":"ref_41","first-page":"1","article-title":"Evolution of bot and human behavior during elections","volume":"24","author":"Luceri","year":"2019","journal-title":"First Monday"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Natural Language Toolkit (2021, October 11). Team NLTK. Available online: https:\/\/www.nltk.org\/.","DOI":"10.17507\/tpls.1109.09"},{"key":"ref_43","first-page":"1","article-title":"Social bots distort the 2016 US presidential election online discussion","volume":"21","author":"Bessi","year":"2016","journal-title":"First Monday"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.vaccine.2019.06.086","article-title":"Temporal trends in anti-vaccine discourse on Twitter","volume":"37","author":"Gunaratne","year":"2019","journal-title":"Vaccine"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1281","DOI":"10.2105\/AJPH.2018.304661","article-title":"Health communication trolls and bots versus public health agencies\u2019 trusted voices","volume":"108","author":"Sutton","year":"2018","journal-title":"Am. J. Public Health"}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/3\/1651\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:12:10Z","timestamp":1760134330000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/3\/1651"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,31]]},"references-count":45,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["ijerph19031651"],"URL":"https:\/\/doi.org\/10.3390\/ijerph19031651","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,31]]}}}