{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:13:36Z","timestamp":1755839616195,"version":"3.44.0"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,9,28]]},"abstract":"<jats:p>Unlike most other forms of coordinated, inauthentic behavior occurring online, the goals of state-sponsored information operations, or SSIOs, are often complex and multifaceted. These goals range from flooding conversations with a certain narrative, to increasing the public's engagement with news sources of questionable quality, to stoking tensions between ideologically opposed groups to weaken public trust. The prevailing theoretical framework for understanding SSIOs is to treat them as a social botnet: a behaviorally homogeneous cluster of coordinated activity. However, the social bot framework is both at odds with some of the behaviors observed in early SSIOs and more broadly with the wide swathe of goals these operations set out to accomplish. To examine the fit of the social bot framework in the SSIO context, we develop a novel bag-of-words based method for clustering and describing user activity traces. Applying this method to a comprehensive repository of SSIOs conducted on Twitter over the last decade, we find that SSIOs violate both the core assumption of the social bot framework, and how it is operationalized in practical work. Instead, we find that SSIOs exhibit a clear division of labor and propose cooperative work with social roles as a more effective theoretical framework for understanding SSIOs. Through applying this framework, we find that the roles that SSIO agents take on have become more stable and simple over time, which holds substantial implications for developing methods for detection of these operations in the wild.<\/jats:p>","DOI":"10.1145\/3610033","type":"journal-article","created":{"date-parts":[[2023,10,4]],"date-time":"2023-10-04T15:54:10Z","timestamp":1696434850000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Working Together (to Undermine Democratic Institutions): Challenging the Social Bot Paradigm in SSIO Research"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7459-3716","authenticated-orcid":false,"given":"Cole","family":"Polychronis","sequence":"first","affiliation":[{"name":"The University of Utah, Salt Lake City, UT, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9200-6914","authenticated-orcid":false,"given":"Marina","family":"Kogan","sequence":"additional","affiliation":[{"name":"The University of Utah, Salt Lake City, UT, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,10,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675208"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comcom.2013.04.004"},{"key":"e_1_2_1_3_1","volume-title":"A comparative study of Bot Detection techniques methods with an application related to Covid-19 discourse on Twitter. arXiv:2102.01148 [cs] (Feb","author":"Antenore Marzia","year":"2021","unstructured":"Marzia Antenore, Jose M. Camacho-Rodriguez, and Emanuele Panizzi. 2021. A comparative study of Bot Detection techniques methods with an application related to Covid-19 discourse on Twitter. arXiv:2102.01148 [cs] (Feb. 2021). http:\/\/arxiv.org\/abs\/2102.01148 arXiv: 2102.01148."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274289"},{"key":"e_1_2_1_5_1","unstructured":"Dan Arnaudo. 2017. Computational Propaganda in Brazil: Social Bots during Elections. (2017) 39."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","unstructured":"Adam Badawy Emilio Ferrara and Kristina Lerman. 2018. Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign. In 2018 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 258--265. https:\/\/doi.org\/10.1109\/ASONAM.2018.8508646 ISSN: 2473--991X.","DOI":"10.1109\/ASONAM.2018.8508646"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.23919\/CYCON.2019.8756832"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/0894439317734157"},{"key":"e_1_2_1_9_1","volume-title":"The KGB and Soviet Disinformation: An Insider's View","author":"Bittman Ladislav","unstructured":"Ladislav Bittman. 1985. The KGB and Soviet Disinformation: An Insider's View 1st edition ed.). Pergamon Pr, Washington.","edition":"1"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Samantha Bradshaw and Philip N Howard. 2019. The Global Disinformation Order: 2019 Global Inventory of Organised Social Media Manipulation. (2019) 27.","DOI":"10.1021\/cen-09705-feature5"},{"key":"e_1_2_1_11_1","unstructured":"Ryan Broderick. 2019. Here's What The Mueller Report Says About Russian Trolls Using Social Media To Tamper With The 2016 Election. https:\/\/www.buzzfeednews.com\/article\/ryanhatesthis\/mueller-report-internet-research-agency-detailed-2016 Library Catalog: www.buzzfeednews.com."},{"key":"e_1_2_1_12_1","unstructured":"Vidhi Bubna. 2022. How TikTok is being used as propaganda machinery during war time. https:\/\/moderndiplomacy.eu\/2022\/04\/13\/how-tiktok-is-being-used-as-propaganda-machinery-during-war-time\/"},{"key":"e_1_2_1_13_1","unstructured":"Oliver Carroll. 2017. St Petersburg 'troll farm' had 90 dedicated staff working to influence US election campaign textbar The Independent textbar The Independent. https:\/\/www.independent.co.uk\/news\/world\/europe\/russia-us-election-donald-trump-st-petersburg-troll-farm-hillary-clinton-a8005276.html"},{"key":"e_1_2_1_14_1","unstructured":"Endalkachew Chala. 2018. Leaked Documents Show That Ethiopia's Ruling Elites Are Hiring Social Media Trolls (And Watching Porn) \u00b7 Global Voices. https:\/\/globalvoices.org\/2018\/01\/20\/leaked-documents-show-that-ethiopias-ruling-elites-are-hiring-social-media-trolls-and-watching-porn\/"},{"key":"e_1_2_1_15_1","volume-title":"Debot: Twitter bot detection via warped correlation.. In Icdm. 817--822.","author":"Chavoshi Nikan","year":"2016","unstructured":"Nikan Chavoshi, Hossein Hamooni, and Abdullah Mueen. 2016. Debot: Twitter bot detection via warped correlation.. In Icdm. 817--822."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3-030--61255--9_31"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3409116"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2016.29"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3055135"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2017.2681672"},{"key":"e_1_2_1_21_1","volume-title":"THE INFORMATION WAR ON UKRAINE NEW CHALLENGES","author":"Darczewska Dr JOLANTA","year":"2014","unstructured":"Dr JOLANTA Darczewska. 2014. THE INFORMATION WAR ON UKRAINE NEW CHALLENGES. Cicero Foundation Great Debate (2014), 19."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872518.2889302"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10190"},{"key":"e_1_2_1_24_1","volume-title":"The Tactics & Tropes of the Internet Research Agency. U.S. Senate Documents (Oct","author":"DiResta Renee","year":"2019","unstructured":"Renee DiResta, Kris Shaffer, Becky Ruppel, David Sullivan, Robert Matney, Ryan Fox, Jonathan Albright, and Ben Johnson. 2019. The Tactics & Tropes of the Internet Research Agency. U.S. Senate Documents (Oct. 2019). https:\/\/digitalcommons.unl.edu\/senatedocs\/2"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274694.3274738"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3217804.3217929"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v22i8.8005"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818717"},{"key":"e_1_2_1_29_1","volume-title":"Russian propaganda on social media during the 2022 invasion of Ukraine. arXiv preprint arXiv:2211.04154","author":"Geissler Dominique","year":"2022","unstructured":"Dominique Geissler, Dominik B\"ar, Nicolas Pr\u00f6llochs, and Stefan Feuerriegel. 2022. Russian propaganda on social media during the 2022 invasion of Ukraine. arXiv preprint arXiv:2211.04154 (2022)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00762ED1V01Y201703HLT037"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/poi3.184"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1089\/big.2017.0044"},{"key":"e_1_2_1_33_1","unstructured":"Simon Hegelich and D. Janetzko. 2016. Are Social Bots on Twitter Political Actors? Empirical Evidence from a Ukrainian Social Botnet. In ICWSM."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394231.3397889"},{"key":"e_1_2_1_35_1","unstructured":"Kai Kaspar Lars Gr\u00e4\u00dfer and Aycha Riffi (Eds.). 2017. Online Hate Speech: Perspektiven auf eine neue Form des Hasses. Number Band 4 in Schriftenreihe zur digitalen Gesellschaft NRW. kopaed D\u00fcsseldorf M\u00fcnchen."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819962"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00453-019-00548-x"},{"key":"e_1_2_1_38_1","volume-title":"Bots and automation over Twitter during the first US presidential debate. Comprop data memo","author":"Kollanyi Bence","year":"2016","unstructured":"Bence Kollanyi, Philip N Howard, and Samuel C Woolley. 2016. Bots and automation over Twitter during the first US presidential debate. Comprop data memo, Vol. 1 (2016), 1--4."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1086\/230270"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113383"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0134270"},{"key":"e_1_2_1_42_1","unstructured":"Nathalie Marechal. 2016. When Bots Tweet: Toward a Normative Framework for Bots on Social Networking Sites. (2016) 10."},{"key":"e_1_2_1_43_1","unstructured":"C. A. Martins. 2003. Reducing the Dimensionality of Bag-of-Words Text Representation Used by Learning Algorithms. https:\/\/www.semanticscholar.org\/paper\/Reducing-the-Dimensionality-of-Bag-of-Words-Text-by-Martins\/a90fca4c78b66fb0e28f4fe8086d28f3720c7999"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292522.3326015"},{"key":"e_1_2_1_45_1","unstructured":"Paul Mozur. 2018. A Genocide Incited on Facebook With Posts From Myanmar's Military - The New York Times. https:\/\/www.nytimes.com\/2018\/10\/15\/technology\/myanmar-facebook-genocide.html"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2020.102250"},{"key":"e_1_2_1_47_1","volume-title":"Alessandro Flammini, and Filippo Menczer.","author":"Pacheco Diogo","year":"2020","unstructured":"Diogo Pacheco, Pik-Mai Hui, Christopher Torres-Lugo, Bao Tran Truong, Alessandro Flammini, and Filippo Menczer. 2020. Uncovering Coordinated Networks on Social Media. arXiv:2001.05658 [physics] (Jan. 2020). http:\/\/arxiv.org\/abs\/2001.05658 arXiv: 2001.05658."},{"key":"e_1_2_1_48_1","volume-title":"Discriminating bot accounts based solely on temporal features of microblog behavior. Physica A: Statistical Mechanics and its Applications","author":"Pan Junshan","year":"2016","unstructured":"Junshan Pan, Ying Liu, Xiang Liu, and Hanping Hu. 2016. Discriminating bot accounts based solely on temporal features of microblog behavior. Physica A: Statistical Mechanics and its Applications, Vol. 450 (2016), 193--204."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14--1162"},{"key":"e_1_2_1_50_1","volume-title":"Flamers","author":"Phadke Shruti","year":"2021","unstructured":"Shruti Phadke and Tanushree Mitra. 2021. Educators, Solicitors, Flamers, Motivators, Sympathizers: Characterizing Roles in Online Extremist Movements. arXiv:2105.08827 [cs] (May 2021). http:\/\/arxiv.org\/abs\/2105.08827 arXiv: 2105.08827."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/IEC47844.2019.8950616"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2020.101715"},{"key":"e_1_2_1_53_1","unstructured":"Twitter Safety. 2018. Information Operations - Twitter Transparency Center. https:\/\/transparency.twitter.com\/en\/reports\/information-operations.html"},{"key":"e_1_2_1_54_1","unstructured":"Twitter Safety. 2019. Disclosing new data to our archive of information operations. https:\/\/blog.twitter.com\/en_us\/topics\/company\/2019\/info-ops-disclosure-data-september-2019"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412698"},{"key":"e_1_2_1_56_1","volume-title":"Analysis of Cooperative Work. A Conceptual Framework. (Jan","author":"Schmidt Kjeld","year":"1990","unstructured":"Kjeld Schmidt. 1990. Analysis of Cooperative Work. A Conceptual Framework. (Jan. 1990)."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1002\/hfm.4530010402"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00752449"},{"key":"e_1_2_1_59_1","unstructured":"Max Seddon. 2014. Documents Show How Russia's Troll Army Hit America. https:\/\/www.buzzfeednews.com\/article\/maxseddon\/documents-show-how-russias-troll-army-hit-america Library Catalog: www.buzzfeednews.com."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-06930--7"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19097-1_3"},{"key":"e_1_2_1_62_1","unstructured":"Freeman Spogli Institute for International Studies Stanford. 2020. Four Years Later How is Russian Interference Different in the 2020 Election? https:\/\/medium.com\/freeman-spogli-institute-for-international-studies\/four-years-later-how-is-russian-interference-different-in-the-2020-election-3d159c56c846"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14878"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359229"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v12i1.15009"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb022630"},{"key":"e_1_2_1_67_1","volume-title":"Defense Primer: Information Operations. Technical Report. LIBRARY OF CONGRESS WASHINGTON DC. https:\/\/apps.dtic.mil\/sti\/citations\/AD1169650 Section: Technical Reports.","author":"Theohary Catherine A.","year":"2020","unstructured":"Catherine A. Theohary. 2020. Defense Primer: Information Operations. Technical Report. LIBRARY OF CONGRESS WASHINGTON DC. https:\/\/apps.dtic.mil\/sti\/citations\/AD1169650 Section: Technical Reports."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1007\/s42001-020-00087--4"},{"key":"e_1_2_1_69_1","volume-title":"2010 international conference on security and cryptography (SECRYPT). IEEE, 1--10","author":"Wang Alex Hai","year":"2010","unstructured":"Alex Hai Wang. 2010. Don't follow me: Spam detection in twitter. In 2010 international conference on security and cryptography (SECRYPT). IEEE, 1--10."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.2307\/2393339"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v21i4.6161"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.32604\/cmc.2019.05155"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2013.2267732"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i01.5460"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/TFUZZ.2017.2690222"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610033","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610033","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T04:24:13Z","timestamp":1755750253000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610033"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"references-count":75,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2023,9,28]]}},"alternative-id":["10.1145\/3610033"],"URL":"https:\/\/doi.org\/10.1145\/3610033","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2023,9,28]]},"assertion":[{"value":"2023-10-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}