{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T08:08:13Z","timestamp":1768723693633,"version":"3.49.0"},"reference-count":31,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,2,26]],"date-time":"2024-02-26T00:00:00Z","timestamp":1708905600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["BDCC"],"abstract":"<jats:p>This paper aims to test the hypothesis that the quality of social media bot detection systems based on supervised machine learning may not be as accurate as researchers claim, given that bots have become increasingly sophisticated, making it difficult for human annotators to detect them better than random selection. As a result, obtaining a ground-truth dataset with human annotation is not possible, which leads to supervised machine-learning models inheriting annotation errors. To test this hypothesis, we conducted an experiment where humans were tasked with recognizing malicious bots on the VKontakte social network. We then compared the \u201chuman\u201d answers with the \u201cground-truth\u201d bot labels (\u2018a bot\u2019\/\u2018not a bot\u2019). Based on the experiment, we evaluated the bot detection efficiency of annotators in three scenarios typical for cybersecurity but differing in their detection difficulty as follows: (1) detection among random accounts, (2) detection among accounts of a social network \u2018community\u2019, and (3) detection among verified accounts. The study showed that humans could only detect simple bots in all three scenarios but could not detect more sophisticated ones (p-value = 0.05). The study also evaluates the limits of hypothetical and existing bot detection systems that leverage non-expert-labelled datasets as follows: the balanced accuracy of such systems can drop to 0.5 and lower, depending on bot complexity and detection scenario. The paper also describes the experiment design, collected datasets, statistical evaluation, and machine learning accuracy measures applied to support the results. In the discussion, we raise the question of using human labelling in bot detection systems and its potential cybersecurity issues. We also provide open access to the datasets used, experiment results, and software code for evaluating statistical and machine learning accuracy metrics used in this paper on GitHub.<\/jats:p>","DOI":"10.3390\/bdcc8030024","type":"journal-article","created":{"date-parts":[[2024,2,26]],"date-time":"2024-02-26T06:50:23Z","timestamp":1708930223000},"page":"24","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Experimental Evaluation: Can Humans Recognise Social Media Bots?"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7873-2733","authenticated-orcid":false,"given":"Maxim","family":"Kolomeets","sequence":"first","affiliation":[{"name":"School of Computing, Newcastle University, Newcastle upon Tyne NE4 5TG, UK"},{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences, 14th Line of V.O. 39, St. Petersburg 199178, Russia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8394-0783","authenticated-orcid":false,"given":"Olga","family":"Tushkanova","sequence":"additional","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences, 14th Line of V.O. 39, St. Petersburg 199178, Russia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3748-5414","authenticated-orcid":false,"given":"Vasily","family":"Desnitsky","sequence":"additional","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences, 14th Line of V.O. 39, St. Petersburg 199178, Russia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4945-6151","authenticated-orcid":false,"given":"Lidia","family":"Vitkova","sequence":"additional","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences, 14th Line of V.O. 39, St. Petersburg 199178, Russia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7056-6972","authenticated-orcid":false,"given":"Andrey","family":"Chechulin","sequence":"additional","affiliation":[{"name":"St. Petersburg Federal Research Center of the Russian Academy of Sciences, 14th Line of V.O. 39, St. Petersburg 199178, Russia"}]}],"member":"1968","published-online":{"date-parts":[[2024,2,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Dialektakis, G., Dimitriadis, I., and Vakali, A. (2022). CALEB: A Conditional Adversarial Learning Framework to Enhance Bot Detection. arXiv.","DOI":"10.2139\/ssrn.4344054"},{"key":"ref_2","unstructured":"Cresci, S., Petrocchi, M., Spognardi, A., and Tognazzi, S. (July, January 30). Better Safe Than Sorry: An Adversarial Approach to Improve Social Bot Detection. Proceedings of the 10th ACM Conference on Web Science, Boston, MA, USA."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"28855","DOI":"10.1109\/ACCESS.2019.2901864","article-title":"Detecting Malicious Social Bots Based on Clickstream Sequences","volume":"7","author":"Shi","year":"2019","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1145\/3409116","article-title":"A decade of social bot detection","volume":"63","author":"Cresci","year":"2020","journal-title":"Commun. ACM"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kolomeets, M., and Chechulin, A. (2021, January 12\u201314). Analysis of the malicious bots market. Proceedings of the 2021 29th Conference of Open Innovations Association (FRUCT), Tampere, Finland.","DOI":"10.23919\/FRUCT52173.2021.9435421"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1145\/2818717","article-title":"The rise of social bots","volume":"59","author":"Ferrara","year":"2016","journal-title":"Commun. ACM"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., and Tesconi, M. (2017, January 3\u20137). The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia.","DOI":"10.1145\/3041021.3055135"},{"key":"ref_8","unstructured":"Beskow, D.M., and Carley, K.M. (2020). Bot-Match: Social Bot Detection with Recursive Nearest Neighbors Search. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Tan, Z., Feng, S., Sclar, M., Wan, H., Luo, M., Choi, Y., and Tsvetkov, Y. (2023). BotPercent: Estimating Twitter Bot Populations from Groups to Crowds. arXiv.","DOI":"10.18653\/v1\/2023.findings-emnlp.954"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1280","DOI":"10.1109\/TIFS.2013.2267732","article-title":"Empirical Evaluation and New Design for Fighting Evolving Twitter Spammers","volume":"8","author":"Yang","year":"2011","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Alsubaei, F.S. (2023). Detection of Inappropriate Tweets Linked to Fake Accounts on Twitter. Appl. Sci., 13.","DOI":"10.3390\/app13053013"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Dubasova, E., Berdashkevich, A., Kopanitsa, G., Kashlikov, P.P., and Metsker, O. (2022, January 9\u201311). Social Network Users Profiling Using Machine Learning for Information Security Tasks. Proceedings of the 2022 32nd Conference of Open Innovations Association (FRUCT), Tampere, Finland.","DOI":"10.23919\/FRUCT56874.2022.9953858"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tyagi, T., Sharma, P., Bansal, R., Jain, K., Bansal, P., and Malik, K. (2023, January 19\u201320). Twitter Bot Detection using Machine Learning Models. Proceedings of the 2023 13th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Noida, India.","DOI":"10.1109\/Confluence56041.2023.10048796"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hays, C., Schutzman, Z., Raghavan, M., Walk, E., and Zimmer, P. (2023). Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection. arXiv.","DOI":"10.1145\/3543507.3583214"},{"key":"ref_15","unstructured":"Cresci, S., Di Pietro, R., Spognardi, A., Tesconi, M., and Petrocchi, M. (2023). Demystifying Misconceptions in Social Bots Research. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"102250","DOI":"10.1016\/j.ipm.2020.102250","article-title":"Detection of bots in social media: A systematic review","volume":"57","author":"Orabi","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.ins.2015.10.039","article-title":"Account classification in online social networks with LBCA and wavelets","volume":"332","author":"Igawa","year":"2016","journal-title":"Inf. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3183506","article-title":"Detection of human, legitimate bot, and malicious bot in online social networks based on wavelets","volume":"14","author":"Jr","year":"2018","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl. (TOMM)"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Dickerson, J.P., Kagan, V., and Subrahmanian, V. (2014, January 17\u201320). Using sentiment to detect bots on twitter: Are humans more opinionated than bots?. Proceedings of the 2014 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014), Beijing, China.","DOI":"10.1109\/ASONAM.2014.6921650"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.dss.2015.09.003","article-title":"Fame for sale: Efficient detection of fake Twitter followers","volume":"80","author":"Cresci","year":"2015","journal-title":"Decis. Support Syst."},{"key":"ref_21","first-page":"141","article-title":"Bot detection by friends graph in social networks","volume":"12","author":"Kolomeets","year":"2021","journal-title":"J. Wirel. Mob. Netw. Ubiquitous Comput. Dependable Appl."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/MC.2016.183","article-title":"The DARPA Twitter bot challenge","volume":"49","author":"Subrahmanian","year":"2016","journal-title":"Computer"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Morstatter, F., Wu, L., Nazer, T.H., Carley, K.M., and Liu, H. (2016, January 18\u201321). A new approach to bot detection: Striking the balance between precision and recall. Proceedings of the 2016 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), San Francisco, CA, USA.","DOI":"10.1109\/ASONAM.2016.7752287"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Echeverria, J., and Zhou, S. (August, January 31). Discovery, retrieval, and analysis of the \u2018star wars\u2019 botnet in Twitter. Proceedings of the 2017 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, Sydney, Australia.","DOI":"10.1145\/3110025.3110074"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kantepe, M., and Ganiz, M.C. (2017, January 5\u20138). Preprocessing framework for Twitter bot detection. Proceedings of the 2017 International Conference on Computer Science and Engineering (UBMK), Antalya, Turkey.","DOI":"10.1109\/UBMK.2017.8093483"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Gallwitz, F., and Kreil, M. (2022, January 11\u201312). Investigating the Validity of Botometer-Based Social Bot Studies. Proceedings of the Disinformation in Open Online Media: 4th Multidisciplinary International Symposium, MISDOOM 2022, Boise, ID, USA.","DOI":"10.1007\/978-3-031-18253-2_5"},{"key":"ref_27","unstructured":"Kolomeets, M. (2023, November 26). MKVK2021 and MKVKTT2021 Security Datasets. Available online: https:\/\/github.com\/guardeec\/datasets."},{"key":"ref_28","unstructured":"(2023, November 26). Top Websites Ranking by Country (on 1 March 2023). Available online: https:\/\/www.similarweb.com\/top-websites\/computers-electronics-and-technology\/social-networks-and-online-communities."},{"key":"ref_29","unstructured":"(2023, November 26). VK Report for q1 2022 (in Russian). Available online: https:\/\/vk.com\/main.php?subdir=press&subsubdir=q1-2022-results."},{"key":"ref_30","first-page":"1","article-title":"Determination of Young Generation\u2019s Sensitivity to the Destructive Stimuli based on the Information in Social Networks","volume":"9","author":"Branitskiy","year":"2019","journal-title":"J. Internet Serv. Inf. Secur."},{"key":"ref_31","unstructured":"(2018). The Black Market for Social Media Manipulation, NATO Strategic Communications Centre of Excellence. Available online: https:\/\/stratcomcoe.org\/cuploads\/pfiles\/web_nato_report_-__the_black_market_of_malicious_use_of_social_media-1.pdf."}],"container-title":["Big Data and Cognitive Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-2289\/8\/3\/24\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:04:50Z","timestamp":1760105090000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-2289\/8\/3\/24"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,26]]},"references-count":31,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["bdcc8030024"],"URL":"https:\/\/doi.org\/10.3390\/bdcc8030024","relation":{},"ISSN":["2504-2289"],"issn-type":[{"value":"2504-2289","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,26]]}}}