{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T16:17:00Z","timestamp":1759940220314,"version":"3.37.3"},"reference-count":67,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,10,9]],"date-time":"2023-10-09T00:00:00Z","timestamp":1696809600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,9]],"date-time":"2023-10-09T00:00:00Z","timestamp":1696809600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["CRSK-2_195707"],"award-info":[{"award-number":["CRSK-2_195707"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["EPJ Data Sci."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The detection of state-sponsored trolls operating in influence campaigns on social media is a critical and unsolved challenge for the research community, which has significant implications beyond the online realm. To address this challenge, we propose a new AI-based solution that identifies troll accounts solely through behavioral cues associated with their sequences of sharing activity, encompassing both their actions and the feedback they receive from others. Our approach does not incorporate any textual content shared and consists of two steps: First, we leverage an LSTM-based classifier to determine whether account sequences belong to a state-sponsored troll or an organic, legitimate user. Second, we employ the classified sequences to calculate a metric named the \u201cTroll Score\u201d, quantifying the degree to which an account exhibits troll-like behavior. To assess the effectiveness of our method, we examine its performance in the context of the 2016 Russian interference campaign during the U.S. Presidential election. Our experiments yield compelling results, demonstrating that our approach can identify account sequences with an AUC close to 99% and accurately differentiate between Russian trolls and organic users with an AUC of 91%. Notably, our behavioral-based approach holds a significant advantage in the ever-evolving landscape, where textual and linguistic properties can be easily mimicked by Large Language Models (LLMs): In contrast to existing language-based techniques, it relies on more challenging-to-replicate behavioral cues, ensuring greater resilience in identifying influence campaigns, especially given the potential increase in the usage of LLMs for generating inauthentic content. Finally, we assessed the generalizability of our solution to various entities driving different information operations and found promising results that will guide future research.<\/jats:p>","DOI":"10.1140\/epjds\/s13688-023-00423-4","type":"journal-article","created":{"date-parts":[[2023,10,9]],"date-time":"2023-10-09T10:05:44Z","timestamp":1696845944000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Exposing influence campaigns in the age of LLMs: a behavioral-based AI approach to detecting state-sponsored trolls"],"prefix":"10.1140","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5128-5814","authenticated-orcid":false,"given":"Fatima","family":"Ezzeddine","sequence":"first","affiliation":[]},{"given":"Omran","family":"Ayoub","sequence":"additional","affiliation":[]},{"given":"Silvia","family":"Giordano","sequence":"additional","affiliation":[]},{"given":"Gianluca","family":"Nogara","sequence":"additional","affiliation":[]},{"given":"Ihab","family":"Sbeity","sequence":"additional","affiliation":[]},{"given":"Emilio","family":"Ferrara","sequence":"additional","affiliation":[]},{"given":"Luca","family":"Luceri","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,10,9]]},"reference":[{"key":"423_CR1","unstructured":"Luceri L, Cresci S, Giordano S (2021) Social media against society. The Internet and the 2020 Campaign, 1"},{"issue":"1","key":"423_CR2","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1007\/s12290-016-0395-5","volume":"15","author":"J Aro","year":"2016","unstructured":"Aro J (2016) The cyberspace war: propaganda and trolling as warfare tools. Eur View 15(1):121\u2013132","journal-title":"Eur View"},{"issue":"9","key":"423_CR3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0138740","volume":"10","author":"F Zollo","year":"2015","unstructured":"Zollo F, Novak PK, Del Vicario M, Bessi A, Mozeti\u010d I, Scala A, Caldarelli G, Quattrociocchi W (2015) Emotional dynamics in the age of misinformation. PLoS ONE 10(9):0138740","journal-title":"PLoS ONE"},{"key":"423_CR4","volume-title":"The filter bubble: what the Internet is hiding from you","author":"E Pariser","year":"2011","unstructured":"Pariser E (2011) The filter bubble: what the Internet is hiding from you. Penguin, New York"},{"key":"423_CR5","doi-asserted-by":"crossref","unstructured":"Luceri L, Cardoso F, Giordano S (2021) Down the bot hole: actionable insights from a one-year analysis of bot activity on Twitter. First Monday","DOI":"10.5210\/fm.v26i3.11441"},{"issue":"1","key":"423_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-022-10070-w","volume":"12","author":"F Pierri","year":"2022","unstructured":"Pierri F, Perry BL, DeVerna MR, Yang K-C, Flammini A, Menczer F, Bryden J (2022) Online misinformation is linked to early COVID-19 vaccination hesitancy and refusal. Sci Rep 12(1):1\u20137","journal-title":"Sci Rep"},{"key":"423_CR7","doi-asserted-by":"publisher","DOI":"10.1145\/2749279.2749283","volume":"2015","author":"E Ferrara","year":"2015","unstructured":"Ferrara E (2015) \u201cManipulation and abuse on social media\u201d by Emilio Ferrara with Ching-Man Au Yeung as coordinator. ACM SIGWEB Newsl 2015:4","journal-title":"ACM SIGWEB Newsl"},{"key":"423_CR8","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1145\/3578503.3583597","volume-title":"Proceedings of the 15th ACM web science conference 2023","author":"F Pierri","year":"2023","unstructured":"Pierri F, Luceri L, Jindal N, Ferrara E (2023) Propaganda and misinformation on Facebook and Twitter during the Russian invasion of Ukraine. In: Proceedings of the 15th ACM web science conference 2023, pp\u00a065\u201374"},{"issue":"2","key":"423_CR9","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1007\/s42001-020-00094-5","volume":"3","author":"E Ferrara","year":"2020","unstructured":"Ferrara E, Cresci S, Luceri L (2020) Misinformation, manipulation, and abuse on social media in the era of COVID-19. J Comput Soc Sci 3(2):271\u2013277","journal-title":"J Comput Soc Sci"},{"issue":"8","key":"423_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(20)30565-X","volume":"20","author":"TLI Diseases","year":"2020","unstructured":"Diseases TLI (2020) The COVID-19 infodemic. Lancet Infect Dis 20(8):875","journal-title":"Lancet Infect Dis"},{"issue":"11","key":"423_CR11","doi-asserted-by":"publisher","DOI":"10.2196\/22639","volume":"22","author":"Z Hu","year":"2020","unstructured":"Hu Z, Yang Z, Li Q, Zhang A (2020) The COVID-19 infodemic: infodemiology study analyzing stigmatizing search terms. J Med Internet Res 22(11):22639","journal-title":"J Med Internet Res"},{"key":"423_CR12","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1145\/3501247.3531573","volume-title":"14th ACM web science conference 2022","author":"G Nogara","year":"2022","unstructured":"Nogara G, Vishnuprasad PS, Cardoso F, Ayoub O, Giordano S, Luceri L (2022) The disinformation dozen: an exploratory analysis of COVID-19 disinformation proliferation on Twitter. In: 14th ACM web science conference 2022, pp\u00a0348\u2013358"},{"key":"423_CR13","doi-asserted-by":"crossref","unstructured":"Pierri F, DeVerna MR, Yang K-C, Axelrod D, Bryden J, Menczer F (2022) One year of COVID-19 vaccine misinformation on Twitter. arXiv preprint. arXiv:2209.01675","DOI":"10.2196\/preprints.42227"},{"key":"423_CR14","doi-asserted-by":"crossref","unstructured":"Wang EL, Luceri L, Pierri F, Ferrara E (2022) Identifying and characterizing behavioral classes of radicalization within the qanon conspiracy on Twitter. arXiv preprint. arXiv:2209.09339","DOI":"10.1609\/icwsm.v17i1.22197"},{"key":"423_CR15","volume-title":"Proceedings of the international AAAI conference on web and social media","author":"VP Suresh","year":"2024","unstructured":"Suresh VP, Nogara G, Cardoso F, Cresci S, Giordano S, Luceri L (2024) Tracking fringe and coordinated activity on Twitter leading up to the us capitol attack. In: Proceedings of the international AAAI conference on web and social media"},{"key":"423_CR16","first-page":"770","volume-title":"Proceedings of the international AAAI conference on web and social media","author":"S Phadke","year":"2022","unstructured":"Phadke S, Samory M, Mitra T (2022) Pathways through conspiracy: the evolution of conspiracy radicalization through engagement in online conspiracy discussions. In: Proceedings of the international AAAI conference on web and social media, vol\u00a016, pp\u00a0770\u2013781"},{"issue":"4","key":"423_CR17","doi-asserted-by":"publisher","DOI":"10.2196\/publichealth.8641","volume":"3","author":"J-P Allem","year":"2017","unstructured":"Allem J-P, Ferrara E, Uppu SP, Cruz TB, Unger JB (2017) E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR Public Health Surveill 3(4):8641","journal-title":"JMIR Public Health Surveill"},{"issue":"1","key":"423_CR18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/srep37825","volume":"6","author":"M Del Vicario","year":"2016","unstructured":"Del Vicario M, Vivaldo G, Bessi A, Zollo F, Scala A, Caldarelli G, Quattrociocchi W (2016) Echo chambers: emotional contagion and group polarization on Facebook. Sci Rep 6(1):1\u201312","journal-title":"Sci Rep"},{"issue":"5","key":"423_CR19","doi-asserted-by":"publisher","first-page":"1480","DOI":"10.1007\/s10618-017-0527-9","volume":"31","author":"A Matakos","year":"2017","unstructured":"Matakos A, Terzi E, Tsaparas P (2017) Measuring and moderating opinion polarization in social networks. Data Min Knowl Discov 31(5):1480\u20131505","journal-title":"Data Min Knowl Discov"},{"issue":"6380","key":"423_CR20","doi-asserted-by":"publisher","first-page":"1146","DOI":"10.1126\/science.aap9559","volume":"359","author":"S Vosoughi","year":"2018","unstructured":"Vosoughi S, Roy D, Aral S (2018) The spread of true and false news online. Science 359(6380):1146\u20131151","journal-title":"Science"},{"issue":"6106","key":"423_CR21","doi-asserted-by":"publisher","first-page":"472","DOI":"10.1126\/science.1230456","volume":"338","author":"PT Metaxas","year":"2012","unstructured":"Metaxas PT, Mustafaraj E (2012) Social media and the elections. Science 338(6106):472\u2013473","journal-title":"Science"},{"key":"423_CR22","first-page":"1","volume-title":"Proceedings of the 34th ACM conference on hypertext and social media","author":"VL Gatta","year":"2023","unstructured":"Gatta VL, Luceri L, Fabbri F, Ferrara E (2023) The interconnected nature of online harm and moderation: investigating the cross-platform spread of harmful content between youtube and Twitter. In: Proceedings of the 34th ACM conference on hypertext and social media, pp\u00a01\u201310"},{"key":"423_CR23","unstructured":"Carroll O (2017) St. petersburg troll farm had 90 dedicated staff working to influence US election campaign. The Independent"},{"key":"423_CR24","unstructured":"Popken B (2018) Twitter deleted Russian troll tweets. So we published more than 200,000 of them. NBC News 14"},{"key":"423_CR25","unstructured":"Mueller RS (2019) The Mueller report: report on the investigation into Russian interference in the 2016 presidential election. WSBLD"},{"key":"423_CR26","doi-asserted-by":"crossref","unstructured":"Lopez J, Hillygus DS (2018) Why so serious?: survey trolls and misinformation. Why so serious","DOI":"10.2139\/ssrn.3131087"},{"key":"423_CR27","unstructured":"Gadde V, Beykpour K (2020) Additional steps we\u2019re taking ahead of the 2020 US election. Social Media Twitter"},{"issue":"30","key":"423_CR28","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abb5824","volume":"6","author":"M Alizadeh","year":"2020","unstructured":"Alizadeh M, Shapiro JN, Buntain C, Tucker JA (2020) Content-based features predict social media influence operations. Sci Adv 6(30):5824","journal-title":"Sci Adv"},{"issue":"1","key":"423_CR29","doi-asserted-by":"publisher","DOI":"10.1140\/epjds\/s13688-023-00410-9","volume":"12","author":"AC Nwala","year":"2023","unstructured":"Nwala AC, Flammini A, Menczer F (2023) A language framework for modeling social media account behavior. EPJ Data Sci 12(1):33","journal-title":"EPJ Data Sci"},{"key":"423_CR30","doi-asserted-by":"crossref","unstructured":"Pierri F, Luceri L, Ferrara E (2022) How does Twitter account moderation work? Dynamics of account creation and suspension during major geopolitical events. arXiv preprint. arXiv:2209.07614","DOI":"10.1140\/epjds\/s13688-023-00420-7"},{"key":"423_CR31","doi-asserted-by":"crossref","unstructured":"Luceri L, Deb A, Giordano S, Ferrara E (2019) Evolution of bot and human behavior during elections. First Monday","DOI":"10.5210\/fm.v24i9.10213"},{"key":"423_CR32","doi-asserted-by":"publisher","first-page":"312","DOI":"10.1016\/j.ins.2018.08.019","volume":"467","author":"S Kudugunta","year":"2018","unstructured":"Kudugunta S, Ferrara E (2018) Deep neural networks for bot detection. Inf Sci 467:312\u2013322","journal-title":"Inf Sci"},{"issue":"7","key":"423_CR33","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1145\/2818717","volume":"59","author":"E Ferrara","year":"2016","unstructured":"Ferrara E, Varol O, Davis C, Menczer F, Flammini A (2016) The rise of social bots. Commun ACM 59(7):96\u2013104","journal-title":"Commun ACM"},{"key":"423_CR34","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1145\/3292522.3326015","volume-title":"Proceedings of the 10th ACM conference on web science","author":"M Mazza","year":"2019","unstructured":"Mazza M, Cresci S, Avvenuti M, Quattrociocchi W, Tesconi M (2019) Rtbust: exploiting temporal patterns for botnet detection on Twitter. In: Proceedings of the 10th ACM conference on web science, pp\u00a0183\u2013192"},{"key":"423_CR35","first-page":"817","volume-title":"Icdm","author":"N Chavoshi","year":"2016","unstructured":"Chavoshi N, Hamooni H, Mueen A (2016) Debot: Twitter bot detection via warped correlation. In: Icdm, pp\u00a0817\u2013822"},{"key":"423_CR36","first-page":"144","volume-title":"2019 IFIP\/IEEE symposium on integrated network and service management (IM)","author":"A Abou Daya","year":"2019","unstructured":"Abou Daya A, Salahuddin MA, Limam N, Boutaba R (2019) A graph-based machine learning approach for bot detection. In: 2019 IFIP\/IEEE symposium on integrated network and service management (IM). IEEE, New York, pp\u00a0144\u2013152"},{"issue":"4","key":"423_CR37","first-page":"561","volume":"15","author":"S Cresci","year":"2017","unstructured":"Cresci S, Di Pietro R, Petrocchi M, Spognardi A, Tesconi M (2017) Social fingerprinting: detection of spambot groups through DNA-inspired behavioral modeling. IEEE Trans Dependable Secure Comput 15(4):561\u2013576","journal-title":"IEEE Trans Dependable Secure Comput"},{"key":"423_CR38","doi-asserted-by":"crossref","unstructured":"Ferrara E (2022) Twitter spam and false accounts prevalence, detection and characterization: a survey. arXiv preprint. arXiv:2211.05913","DOI":"10.5210\/fm.v27i12.12872"},{"key":"423_CR39","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1145\/3292522.3326016","volume-title":"Proceedings of the 10th ACM conference on web science","author":"S Zannettou","year":"2019","unstructured":"Zannettou S, Caulfield T, Setzer W, Sirivianos M, Stringhini G, Blackburn J (2019) Who let the trolls out? Towards understanding state-sponsored trolls. In: Proceedings of the 10th ACM conference on web science, pp\u00a0353\u2013362"},{"key":"423_CR40","first-page":"1","volume-title":"12th ACM conference on web science","author":"J Im","year":"2020","unstructured":"Im J, Chandrasekharan E, Sargent J, Lighthammer P, Denby T, Bhargava A, Hemphill L, Jurgens D, Gilbert E (2020) Still out there: modeling and identifying Russian troll accounts on Twitter. In: 12th ACM conference on web science, pp\u00a01\u201310"},{"issue":"1","key":"423_CR41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13278-019-0578-6","volume":"9","author":"A Badawy","year":"2019","unstructured":"Badawy A, Addawood A, Lerman K, Ferrara E (2019) Characterizing the 2016 Russian IRA influence campaign. Soc Netw Anal Min 9(1):1\u201311","journal-title":"Soc Netw Anal Min"},{"key":"423_CR42","doi-asserted-by":"publisher","first-page":"195132","DOI":"10.1109\/ACCESS.2020.3033666","volume":"8","author":"S Alhazbi","year":"2020","unstructured":"Alhazbi S (2020) Behavior-based machine learning approaches to identify state-sponsored trolls on Twitter. IEEE Access 8:195132\u2013195141","journal-title":"IEEE Access"},{"key":"423_CR43","doi-asserted-by":"crossref","unstructured":"Saeed MH, Ali S, Blackburn J, De Cristofaro E, Zannettou S, Stringhini G (2021) Trollmagnifier: detecting state-sponsored troll accounts on Reddit. arXiv preprint. arXiv:2112.00443","DOI":"10.1109\/SP46214.2022.9833706"},{"key":"423_CR44","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1016\/j.comcom.2022.09.022","volume":"196","author":"M Mazza","year":"2022","unstructured":"Mazza M, Avvenuti M, Cresci S, Tesconi M (2022) Investigating the difference between trolls, social bots, and humans on Twitter. Comput Commun 196:23\u201336","journal-title":"Comput Commun"},{"key":"423_CR45","first-page":"15","volume-title":"Proceedings of the international AAAI conference on web and social media","author":"A Addawood","year":"2019","unstructured":"Addawood A, Badawy A, Lerman K, Ferrara E (2019) Linguistic cues to deception: identifying political trolls on social media. In: Proceedings of the international AAAI conference on web and social media, vol\u00a013, pp\u00a015\u201325"},{"key":"423_CR46","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1145\/3442167.3442169","volume-title":"New security paradigms workshop 2020","author":"P Jachim","year":"2020","unstructured":"Jachim P, Sharevski F, Treebridge P (2020) Trollhunter [evader]: automated detection [evasion] of Twitter trolls during the COVID-19 pandemic. In: New security paradigms workshop 2020, pp\u00a059\u201375"},{"key":"423_CR47","unstructured":"Yang K-C, Menczer F (2023) Anatomy of an AI-powered malicious social botnet. arXiv preprint. arXiv:2307.16336"},{"key":"423_CR48","doi-asserted-by":"crossref","unstructured":"Ferrara E (2023) Social bot detection in the age of ChatGPT: challenges and opportunities. First Monday","DOI":"10.5210\/fm.v28i6.13185"},{"key":"423_CR49","first-page":"417","volume-title":"Proceedings of the international AAAI conference on web and social media","author":"L Luceri","year":"2020","unstructured":"Luceri L, Giordano S, Ferrara E (2020) Detecting troll behavior via inverse reinforcement learning: a case study of Russian trolls in the 2016 US election. In: Proceedings of the international AAAI conference on web and social media, vol\u00a014, pp\u00a0417\u2013427"},{"key":"423_CR50","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1038\/s42256-023-00690-w","volume":"5","author":"F Menczer","year":"2023","unstructured":"Menczer F, Crandall D, Ahn Y-Y, Kapadia A (2023) Addressing the harms of AI-generated inauthentic content. Nat Mach Intell 5:679\u2013680","journal-title":"Nat Mach Intell"},{"key":"423_CR51","unstructured":"Mitrovi\u0107 S, Andreoletti D, Ayoub O (2023) Chatgpt or human? Detect and explain. Explaining decisions of machine learning model for detecting short ChatGPT-generated text. arXiv preprint. arXiv:2301.13852"},{"key":"423_CR52","unstructured":"Frommer D (2019) Twitter\u2019s list of 2,752 Russian trolls"},{"key":"423_CR53","unstructured":"Weller H, Woo J (2019) Identifying Russian trolls on Reddit with deep learning and bert word embeddings"},{"key":"423_CR54","first-page":"1195","volume-title":"2013 IFIP\/IEEE international symposium on integrated network management (IM 2013)","author":"T Vanhove","year":"2013","unstructured":"Vanhove T, Leroux P, Wauters T, De Turck F (2013) Towards the design of a platform for abuse detection in OSNs using multimedial data analysis. In: 2013 IFIP\/IEEE international symposium on integrated network management (IM 2013). IEEE, New York, pp\u00a01195\u20131198"},{"key":"423_CR55","volume-title":"Swedish symposium on deep learning","author":"E Valldor","year":"2018","unstructured":"Valldor E, Stenborg K, Gustavsson D (2018) Firearm detection in social media images. In: Swedish symposium on deep learning"},{"issue":"2","key":"423_CR56","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1007\/s42001-019-00051-x","volume":"2","author":"D Kim","year":"2019","unstructured":"Kim D, Graham T, Wan Z, Rizoiu M-A (2019) Analysing user identity via time-sensitive semantic edit distance (t-sed): a\u00a0case study of Russian trolls on Twitter. J Comput Soc Sci 2(2):331\u2013351","journal-title":"J Comput Soc Sci"},{"key":"423_CR57","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1145\/2858036.2858107","volume-title":"Proceedings of the 2016 CHI conference on human factors in computing systems","author":"G Wang","year":"2016","unstructured":"Wang G, Zhang X, Tang S, Zheng H, Zhao BY (2016) Unsupervised clickstream clustering for user behavior analysis. In: Proceedings of the 2016 CHI conference on human factors in computing systems, pp\u00a0225\u2013236"},{"key":"423_CR58","volume-title":"Proceedings of the international AAAI conference on web and social media","author":"P Metaxas","year":"2015","unstructured":"Metaxas P, Mustafaraj E, Wong K, Zeng L, O\u2019Keefe M, Finn S (2015) What do retweets indicate? Results from user survey and meta-review of research. In: Proceedings of the international AAAI conference on web and social media, vol\u00a09"},{"issue":"49","key":"423_CR59","doi-asserted-by":"publisher","first-page":"12435","DOI":"10.1073\/pnas.1803470115","volume":"115","author":"M Stella","year":"2018","unstructured":"Stella M, Ferrara E, De Domenico M (2018) Bots increase exposure to negative and inflammatory content in online social systems. Proc Natl Acad Sci 115(49):12435\u201312440","journal-title":"Proc Natl Acad Sci"},{"key":"423_CR60","doi-asserted-by":"crossref","unstructured":"Bessi A, Ferrara E (2016) Social bots distort the 2016 US presidential election online discussion. First monday 21(11-7)","DOI":"10.5210\/fm.v21i11.7090"},{"issue":"1","key":"423_CR61","first-page":"1929","volume":"15","author":"N Srivastava","year":"2014","unstructured":"Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929\u20131958","journal-title":"J Mach Learn Res"},{"key":"423_CR62","unstructured":"Nielsen MA (2015) Neural networks and deep learning"},{"issue":"2","key":"423_CR63","first-page":"539","volume":"39","author":"X-Y Liu","year":"2008","unstructured":"Liu X-Y, Wu J, Zhou Z-H (2008) Exploratory undersampling for class-imbalance learning. IEEE Trans Syst Man Cybern, Part B, Cybern 39(2):539\u2013550","journal-title":"IEEE Trans Syst Man Cybern, Part B, Cybern"},{"key":"423_CR64","volume-title":"ICML-KDD 2003 workshop: learning from imbalanced datasets","author":"C Drumnond","year":"2003","unstructured":"Drumnond C (2003) Class imbalance and cost sensitivity: why undersampling beats oversampling. In: ICML-KDD 2003 workshop: learning from imbalanced datasets, vol\u00a03"},{"issue":"4","key":"423_CR65","doi-asserted-by":"publisher","first-page":"554","DOI":"10.1016\/j.lingua.2006.11.012","volume":"118","author":"T Ionin","year":"2008","unstructured":"Ionin T, Zubizarreta ML, Maldonado SB (2008) Sources of linguistic knowledge in the second language acquisition of English articles. Lingua 118(4):554\u2013576","journal-title":"Lingua"},{"key":"423_CR66","doi-asserted-by":"publisher","first-page":"854","DOI":"10.3115\/v1\/P14-2138","volume-title":"Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers)","author":"G Nicolai","year":"2014","unstructured":"Nicolai G, Kondrak G (2014) Does the phonology of l1 show up in l2 texts? In: Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers), pp\u00a0854\u2013859"},{"issue":"1","key":"423_CR67","first-page":"1","volume":"5","author":"E Chen","year":"2021","unstructured":"Chen E, Deb A, Ferrara E (2021) #Election2020: the first public Twitter dataset on the 2020 US presidential election. J\u00a0Comput Soc Sci 5(1):1\u201318","journal-title":"J\u00a0Comput Soc Sci"}],"container-title":["EPJ Data Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1140\/epjds\/s13688-023-00423-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1140\/epjds\/s13688-023-00423-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1140\/epjds\/s13688-023-00423-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,19]],"date-time":"2023-11-19T09:03:19Z","timestamp":1700384599000},"score":1,"resource":{"primary":{"URL":"https:\/\/epjdatascience.springeropen.com\/articles\/10.1140\/epjds\/s13688-023-00423-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,9]]},"references-count":67,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["423"],"URL":"https:\/\/doi.org\/10.1140\/epjds\/s13688-023-00423-4","relation":{},"ISSN":["2193-1127"],"issn-type":[{"type":"electronic","value":"2193-1127"}],"subject":[],"published":{"date-parts":[[2023,10,9]]},"assertion":[{"value":"11 November 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 October 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"46"}}