{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T23:05:28Z","timestamp":1783465528225,"version":"3.55.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2021,4,13]],"date-time":"2021-04-13T00:00:00Z","timestamp":1618272000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"US National Science Foundation","doi-asserted-by":"crossref","award":["CNS 1528138"],"award-info":[{"award-number":["CNS 1528138"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,4,13]]},"abstract":"<jats:p>Social media platforms have been exploited to conduct election interference in recent years. In particular, the Russian-backed Internet Research Agency (IRA) has been identified as a key source of misinformation spread on Twitter prior to the 2016 U.S. presidential election. The goal of this research is to understand whether general Twitter users changed their behavior in the year following first contact from an IRA account. We compare the before and after behavior of contacted users to determine whether there were differences in their mean tweet count, the sentiment of their tweets, and the frequency and sentiment of tweets mentioning @realDonaldTrump or @HillaryClinton. Our results indicate that users overall exhibited statistically significant changes in behavior across most of these metrics, and that those users that engaged with the IRA generally showed greater changes in behavior.<\/jats:p>","DOI":"10.1145\/3449164","type":"journal-article","created":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T17:51:09Z","timestamp":1619113869000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Analyzing Twitter Users' Behavior Before and After Contact by the Russia's Internet Research Agency"],"prefix":"10.1145","volume":"5","author":[{"given":"Upasana","family":"Dutta","sequence":"first","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rhett","family":"Hanscom","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason Shuo","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Han","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tamara","family":"Lehman","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qin","family":"Lv","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shivakant","family":"Mishra","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,4,22]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Barnes and Nicholas Fandos","author":"Adam Goldman Maggie Haberman","year":"2020","unstructured":"Maggie Haberman Adam Goldman, Julian E. Barnes and Nicholas Fandos. 2020. Lawmakers Are Warned That Russia Is Meddling to Re-elect Trump. https:\/\/www.nytimes.com\/2020\/02\/20\/us\/politics\/russian-interference-trump-democrats.html [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW","author":"Albadi Nuha","year":"2019","unstructured":"Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2019. Hateful People or Hateful Bots? Detection and Characterization of Bots Spreading Religious Hatred in Arabic Social Media. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--25."},{"key":"e_1_2_1_3_1","unstructured":"Dan Arnaudo. 2017. Computational propaganda in Brazil: Social bots during elections. (2017)."},{"key":"e_1_2_1_4_1","volume-title":"Assessing the Russian Internet Research Agency's impact on the political attitudes and behaviors of American Twitter users in late","author":"Bail Christopher A","year":"2017","unstructured":"Christopher A Bail, Brian Guay, Emily Maloney, Aidan Combs, D Sunshine Hillygus, Friedolin Merhout, Deen Freelon, and Alexander Volfovsky. 2020. Assessing the Russian Internet Research Agency's impact on the political attitudes and behaviors of American Twitter users in late 2017. Proceedings of the National Academy of Sciences 117, 1 (2020)."},{"key":"e_1_2_1_5_1","unstructured":"Shelly Banjo. 2019. Facebook Twitter and the Digital Disinformation Mess. https:\/\/www.washingtonpost.com\/ business\/facebook-twitter-and-the-digital-disinformation-mess\/2019\/10\/01\/53334c08-e4b4-11e9-b0a6-3d03721b85ef_story.html"},{"key":"e_1_2_1_6_1","volume-title":"Online extremism and the communities that sustain it: Detecting the ISIS supporting community on Twitter. PloS one 12, 12","author":"Benigni Matthew C","year":"2017","unstructured":"Matthew C Benigni, Kenneth Joseph, and Kathleen M Carley. 2017. Online extremism and the communities that sustain it: Detecting the ISIS supporting community on Twitter. PloS one 12, 12 (2017)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v21i11.7090"},{"key":"e_1_2_1_8_1","volume-title":"Troll factories: The internet research agency and state-sponsored agenda building","author":"Boatwright Brandon C","year":"2018","unstructured":"Brandon C Boatwright, Darren L Linvill, and Patrick L Warren. 2018. Troll factories: The internet research agency and state-sponsored agenda building. Resource Centre on Media Freedom in Europe (2018)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Ryan L Boyd Alexander Spangher Adam Fourney Besmira Nushi Gireeja Ranade James Pennebaker and Eric Horvitz. 2018. Characterizing the Internet Research Agency's Social Media Operations During the 2016 US Presidential Election using Linguistic Analyses. (2018).","DOI":"10.31234\/osf.io\/ajh2q"},{"key":"e_1_2_1_10_1","volume-title":"Combating fake news in the digital age","author":"Burkhardt Joanna M","unstructured":"Joanna M Burkhardt. 2017. Combating fake news in the digital age. Vol. 53. American Library Association."},{"key":"e_1_2_1_11_1","volume-title":"Katie Polglase and Tim Lister","author":"Gianluca Mezzofiore Clarissa Ward Sebastian Shukla","year":"2020","unstructured":"Sebastian Shukla Gianluca Mezzofiore Clarissa Ward, Katie Polglase and Tim Lister. 2020. Russian election meddling is back -- via Ghana and Nigeria -- and in your feeds. https:\/\/www.cnn.com\/2020\/03\/12\/world\/russia-ghana-troll-farms-2020-ward\/index.html [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_12_1","volume-title":"Fake news and alternative facts: Information literacy in a post-truth era","author":"Cooke Nicole A","unstructured":"Nicole A Cooke. 2018. Fake news and alternative facts: Information literacy in a post-truth era. American Library Association."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3055135"},{"key":"e_1_2_1_14_1","volume-title":"langdetect 1.0.8. https:\/\/pypi.org\/project\/langdetect\/ [Online","author":"Danilak Michal Mimino","year":"2020","unstructured":"Michal Mimino Danilak. 2020. langdetect 1.0.8. https:\/\/pypi.org\/project\/langdetect\/ [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_15_1","unstructured":"Renee DiResta Kris Shaffer Becky Ruppel David Sullivan Robert Matney Ryan Fox Jonathan Albright and Ben Johnson. 2018. The tactics and tropes of the Internet Research Agency. (2018)."},{"key":"e_1_2_1_16_1","volume-title":"Disinformation and social bot operations in the run up to the 2017 French presidential election. arXiv preprint arXiv:1707.00086","author":"Ferrara Emilio","year":"2017","unstructured":"Emilio Ferrara. 2017. Disinformation and social bot operations in the run up to the 2017 French presidential election. arXiv preprint arXiv:1707.00086 (2017)."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818717"},{"key":"e_1_2_1_18_1","unstructured":"Vijaya Gadde and Kayvon Beykpour. 2018. Setting the record straight on shadow banning. https:\/\/blog.twitter. com\/en_us\/topics\/company\/2018\/Setting-the-record-straight-on-shadow-banning.html"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177705148"},{"key":"e_1_2_1_20_1","unstructured":"Garrett M. Graff. 2018. Russian Trolls Are Still Playing Both Sides-Even With the Mueller Probe. https:\/\/www.wired. com\/story\/russia-indictment-twitter-facebook-play-both-sides\/"},{"key":"e_1_2_1_21_1","volume-title":"The IRA, Social Media and Political Polarization in the United States","author":"Howard Philip N","year":"2012","unstructured":"Philip N Howard, Bharath Ganesh, Dimitra Liotsiou, John Kelly, and Camille Fran\u00e7ois. 2019. The IRA, Social Media and Political Polarization in the United States, 2012--2018. U.S. Senate Documents (2019)."},{"key":"e_1_2_1_22_1","volume-title":"computational propaganda during the UK-EU referendum. Available at SSRN 2798311","author":"Howard Philip N","year":"2016","unstructured":"Philip N Howard and Bence Kollanyi. 2016. Bots,# StrongerIn, and# Brexit: computational propaganda during the UK-EU referendum. Available at SSRN 2798311 (2016)."},{"key":"e_1_2_1_23_1","volume-title":"VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media. https:\/\/pypi.org\/project\/vaderSentiment\/.","author":"Hutto C.J.","unstructured":"C.J. Hutto and E.E. Gilbert. 2014. VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media. https:\/\/pypi.org\/project\/vaderSentiment\/."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1080\/10584609.2019.1661888"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","first-page":"273","DOI":"10.69554\/BVCW5365","article-title":"Social media celebrities as influencers in brand communication: An empirical study on influencer content, its advertising relevance and audience expectations","volume":"6","author":"Kolo Castulus","year":"2018","unstructured":"Castulus Kolo and Florian Haumer. 2018. Social media celebrities as influencers in brand communication: An empirical study on influencer content, its advertising relevance and audience expectations. Journal of Digital & Social Media Marketing 6, 3 (2018), 273--282.","journal-title":"Journal of Digital & Social Media Marketing"},{"key":"e_1_2_1_26_1","volume-title":"Reverse Engineering Russian Internet Research Agency Tactics through Network Analysis. Defence Strategic Communication","author":"Kriel Charles","year":"2019","unstructured":"Charles Kriel and Alexa Pavliuc. 2019. Reverse Engineering Russian Internet Research Agency Tactics through Network Analysis. Defence Strategic Communication (2019), 199--227."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1967.10482916"},{"key":"e_1_2_1_28_1","volume-title":"Instagram says growth hackers are behind spate of fake Stories. https:\/\/techcrunch.com\/2019\/ 08\/15\/instagram-says-growth-hackers-are-behind-spate-of-fake-stories-views\/ [Online","author":"Lomas Natasha","year":"2020","unstructured":"Natasha Lomas. 2020. Instagram says growth hackers are behind spate of fake Stories. https:\/\/techcrunch.com\/2019\/ 08\/15\/instagram-says-growth-hackers-are-behind-spate-of-fake-stories-views\/ [Online; accessed 15- May-2020]."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/15252019.2018.1533501"},{"key":"e_1_2_1_30_1","volume-title":"Russia's 'troll factory' is alive and well in Africa. https: \/\/www.cnn.com\/2019\/10\/31\/europe\/russia-africa-propaganda-intl\/index.html [Online","author":"Mary Ilyushina Gianluca Mezzofiore","year":"2020","unstructured":"Gianluca Mezzofiore Mary Ilyushina and Tim Lister. 2019. Russia's 'troll factory' is alive and well in Africa. https: \/\/www.cnn.com\/2019\/10\/31\/europe\/russia-africa-propaganda-intl\/index.html [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1951.10500769"},{"key":"e_1_2_1_32_1","volume-title":"Report on the investigation into Russian interference in the 2016 presidential election","author":"Mueller Robert S","unstructured":"Robert S Mueller and Man With A. Cat. 2019. Report on the investigation into Russian interference in the 2016 presidential election. Vol. 1. US Department of Justice Washington, DC."},{"key":"e_1_2_1_33_1","unstructured":"US House of Representatives Permanent Select Committee on Intelligence. 2018. Exposing Russia's Effort to Sow Discord Online: The Internet Research Agency and Advertisements. (2018)."},{"key":"e_1_2_1_34_1","volume-title":"Facebook: Russian trolls are back. And they're here to meddle with","author":"O'Sullivan Donie","year":"2020","unstructured":"Donie O'Sullivan. 2020. Facebook: Russian trolls are back. And they're here to meddle with 2020. https:\/\/www.cnn.com\/ 2019\/10\/21\/tech\/russia-instagram-accounts-2020-election\/index.html [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_35_1","volume-title":"Update on Twitter's review of the 2016 US election. https:\/\/blog.twitter.com\/en_ us\/topics\/company\/2018\/2016-election-update.html. [Online","author":"Policy Twitter Public","year":"2020","unstructured":"Twitter Public Policy. 2018. Update on Twitter's review of the 2016 US election. https:\/\/blog.twitter.com\/en_ us\/topics\/company\/2018\/2016-election-update.html. [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_36_1","volume-title":"http:\/\/docs.tweepy.org\/en\/latest\/ [Online","author":"Roesslein Joshua","year":"2020","unstructured":"Joshua Roesslein. 2019. Tweepy Documentation. http:\/\/docs.tweepy.org\/en\/latest\/ [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_37_1","volume-title":"Internet Research Agency Twitter activity predicted 2016 US election polls. First Monday 24, 7","author":"Ruck Damian J","year":"2019","unstructured":"Damian J Ruck, Natalie M Rice, Joshua Borycz, and R Alexander Bentley. 2019. Internet Research Agency Twitter activity predicted 2016 US election polls. First Monday 24, 7 (2019)."},{"key":"e_1_2_1_38_1","volume-title":"Onur Varol, Alessandro Flammini, and Filippo Menczer.","author":"Shao Chengcheng","year":"2017","unstructured":"Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Alessandro Flammini, and Filippo Menczer. 2017. The spread of fake news by social bots. arXiv preprint arXiv:1707.07592 96 (2017), 104."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0142209"},{"key":"e_1_2_1_40_1","volume-title":"How to Conduct the Wilcoxon Sign Test. https:\/\/www.statisticssolutions.com\/how-to-conduct-the-wilcox-sign-test\/ [Online","author":"Solutions Statistics","year":"2020","unstructured":"Statistics Solutions. 2020. How to Conduct the Wilcoxon Sign Test. https:\/\/www.statisticssolutions.com\/how-to-conduct-the-wilcox-sign-test\/ [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_41_1","unstructured":"Liam Stack. 2018. What Is a 'Shadow Ban ' and Is Twitter Doing It to Republican Accounts? https:\/\/www.nytimes. com\/2018\/07\/26\/us\/politics\/twitter-shadowbanning.html"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359229"},{"key":"e_1_2_1_43_1","volume-title":"TWINT - Twitter Intelligence Tool. https:\/\/github.com\/twintproject\/twint [Online","author":"OSINT","year":"2020","unstructured":"OSINT team. 2020. TWINT - Twitter Intelligence Tool. https:\/\/github.com\/twintproject\/twint [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_44_1","volume-title":"How Russian Trolls Used Meme Warfare to Divide America. https:\/\/www.wired.com\/ story\/russia-ira-propaganda-senate-report\/ [Online","author":"Thompson Nicholas","year":"2020","unstructured":"Nicholas Thompson. 2018. How Russian Trolls Used Meme Warfare to Divide America. https:\/\/www.wired.com\/ story\/russia-ira-propaganda-senate-report\/ [Online; accessed 15-May-2020]."},{"key":"e_1_2_1_45_1","article-title":"Cross-platform disinformation campaigns: lessons learned and next steps","volume":"1","author":"Wilson Tom","year":"2020","unstructured":"Tom Wilson and Kate Starbird. 2020. Cross-platform disinformation campaigns: lessons learned and next steps. Harvard Kennedy School Misinformation Review 1, 1 (2020).","journal-title":"Harvard Kennedy School Misinformation Review"},{"key":"e_1_2_1_46_1","volume-title":"Bots in the Twittersphere","author":"Wojcik Stefan","year":"2018","unstructured":"Stefan Wojcik, Solomon Messing, Aaron Smith, Lee Rainie, and Paul Hitlin. 2018. Bots in the Twittersphere. Pew Research Center 18 (2018)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Samuel C Woolley and Philip Howard. 2017. Computational propaganda worldwide: Executive summary. (2017).","DOI":"10.1093\/oso\/9780190931407.001.0001"},{"key":"e_1_2_1_48_1","unstructured":"Kaicheng Yang. 2020. Botometer Python API. https:\/\/github.com\/yangkcatiu\/botometer-python"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/hbe2.115"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449164","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449164","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:17Z","timestamp":1750191497000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449164"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,13]]},"references-count":49,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2021,4,13]]}},"alternative-id":["10.1145\/3449164"],"URL":"https:\/\/doi.org\/10.1145\/3449164","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,13]]},"assertion":[{"value":"2021-04-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}