{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T22:11:56Z","timestamp":1777587116913,"version":"3.51.4"},"reference-count":96,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2019,10,26]],"date-time":"2019-10-26T00:00:00Z","timestamp":1572048000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"government of Spain","award":["MTM2015-65803-R"],"award-info":[{"award-number":["MTM2015-65803-R"]}]},{"name":"European Union's Horizon 2020 research","award":["No 691161 (GEOSAFE)"],"award-info":[{"award-number":["No 691161 (GEOSAFE)"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Social Media are sensors in the real world that can be used to measure the pulse of societies. However, the massive and unfiltered feed of messages posted in social media is a phenomenon that nowadays raises social alarms, especially when these messages contain hate speech targeted to a specific individual or group. In this context, governments and non-governmental organizations (NGOs) are concerned about the possible negative impact that these messages can have on individuals or on the society. In this paper, we present HaterNet, an intelligent system currently being used by the Spanish National Office Against Hate Crimes of the Spanish State Secretariat for Security that identifies and monitors the evolution of hate speech in Twitter. The contributions of this research are many-fold: (1) It introduces the first intelligent system that monitors and visualizes, using social network analysis techniques, hate speech in Social Media. (2) It introduces a novel public dataset on hate speech in Spanish consisting of 6000 expert-labeled tweets. (3) It compares several classification approaches based on different document representation strategies and text classification models. (4) The best approach consists of a combination of a LTSM+MLP neural network that takes as input the tweet\u2019s word, emoji, and expression tokens\u2019 embeddings enriched by the tf-idf, and obtains an area under the curve (AUC) of 0.828 on our dataset, outperforming previous methods presented in the literature.<\/jats:p>","DOI":"10.3390\/s19214654","type":"journal-article","created":{"date-parts":[[2019,10,28]],"date-time":"2019-10-28T04:44:31Z","timestamp":1572237871000},"page":"4654","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":159,"title":["Detecting and Monitoring Hate Speech in Twitter"],"prefix":"10.3390","volume":"19","author":[{"given":"Juan Carlos","family":"Pereira-Kohatsu","sequence":"first","affiliation":[{"name":"Engineering School, Autonomous University of Madrid, 28049 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9389-2380","authenticated-orcid":false,"given":"Lara","family":"Quijano-S\u00e1nchez","sequence":"additional","affiliation":[{"name":"Engineering School, Autonomous University of Madrid, 28049 Madrid, Spain"},{"name":"UC3M-BS Institute of Financial Big Data, Charles III University of Madrid, 28903 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Federico","family":"Liberatore","sequence":"additional","affiliation":[{"name":"UC3M-BS Institute of Financial Big Data, Charles III University of Madrid, 28903 Madrid, Spain"},{"name":"School of Computer Science &amp; Informatics, Cardiff University, Cardiff CF24 3AA, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miguel","family":"Camacho-Collados","sequence":"additional","affiliation":[{"name":"Artificial Intelligence Area Coordinator, Cabinet of the Secretary of State for Digital Advancement, 28020 Madrid, Spain"},{"name":"State Secretariat for Security, Interior Ministry, 28010 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,26]]},"reference":[{"key":"ref_1","unstructured":"Office of the United Nations High Commissioner for Human Rights (2013). Report of the United Nations High Commissioner for Human Rights on the Expert Workshops on the Prohibition of Incitement to National, Racial or Religious Hatred, Office of the United Nations High Commissioner for Human Rights."},{"key":"ref_2","unstructured":"Peretz, P. (1996). 2.1. Up and Down with Ecology: The\u201d Issue-Attention Cycle. The Politics of American Economic Policy Making, M.E. Shape, Inc."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Sui, X., Chen, Z., Wu, K., Ren, P., Ma, J., and Zhou, F. (2014). Social media as sensor in real world: Geolocate user with microblog. Natural Language Processing and Chinese Computing, Springer.","DOI":"10.1007\/978-3-662-45924-9_21"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2461","DOI":"10.1109\/TIFS.2015.2464775","article-title":"Forecasting violent extremist cyber recruitment","volume":"10","author":"Scanlon","year":"2015","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Waseem, Z., and Hovy, D. (2016, January 13\u201315). Hateful symbols or hateful people? Predictive features for hate speech detection on twitter. Proceedings of the NAACL Student Research Workshop, San Diego, CA, USA.","DOI":"10.18653\/v1\/N16-2013"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Waseem, Z. (2016, January 5). Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. Proceedings of the First Workshop on NLP and Computational Social Science, Austin, TX, USA.","DOI":"10.18653\/v1\/W16-5618"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017, January 15\u201318). Automated hate speech detection and the problem of offensive language. Proceedings of the Eleventh International AAAI Conference on Web and Social Media, Montreal, QC, Canada.","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"ref_8","unstructured":"Gamb\u00e4ck, B., and Sikdar, U.K. (August, January 30). Using convolutional neural networks to classify hate-speech. Proceedings of the First Workshop on Abusive Language Online, Vancouver, BC, Canada."},{"key":"ref_9","unstructured":"Park, J.H., and Fung, P. (August, January 30). One-step and Two-step Classification for Abusive Language Detection on Twitter. Proceedings of the First Workshop on Abusive Language Online, Vancouver, BC, Canada."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Robinson, D., and Tepper, J. (2018). Detecting hate speech on twitter using a convolution-gru based deep neural network. European Semantic Web Conference, Springer.","DOI":"10.1007\/978-3-319-93417-4_48"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Badjatiya, P., Gupta, S., Gupta, M., and Varma, V. (2017, January 3\u20137). Deep learning for hate speech detection in tweets. Proceedings of the 26th International Conference on World Wide Web Companion, International World Wide Web Conferences Steering Committee, Perth, Australia.","DOI":"10.1145\/3041021.3054223"},{"key":"ref_12","first-page":"85","article-title":"A survey on automatic detection of hate speech in text","volume":"51","author":"Fortuna","year":"2018","journal-title":"ACM Comput. Surv. CSUR"},{"key":"ref_13","first-page":"189","article-title":"The right to explanation, explained","volume":"34","author":"Kaminski","year":"2019","journal-title":"Berkeley Tech. LJ"},{"key":"ref_14","unstructured":"Sap, M., Card, D., Gabriel, S., Choi, Y., and Smith, N.A. (August, January July). The risk of racial bias in hate speech detection. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1177\/0002716215569197","article-title":"Data-driven content analysis of social media: A systematic overview of automated methods","volume":"659","author":"Schwartz","year":"2015","journal-title":"ANNALS Am. Acad. Political Soc. Sci."},{"key":"ref_16","first-page":"634","article-title":"Opinion polarity detection in Twitter data combining shrinkage regression and topic modeling","volume":"10","author":"Yoon","year":"2016","journal-title":"J. Inf."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1080\/19331681.2012.705080","article-title":"(Wisdom of the Crowds) 2: 2010 UK election prediction with social media","volume":"10","author":"Franch","year":"2013","journal-title":"J. Inf. Technol. Politics"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"74","DOI":"10.4018\/JOEUC.2016040105","article-title":"Social media-based forecasting: A case study of tweets and stock prices in the financial services industry","volume":"28","author":"He","year":"2016","journal-title":"J. Organ. End User Comput. JOEUC"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Perry, W.L. (2013). Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations, Rand Corporation.","DOI":"10.7249\/RR233"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1111\/j.1538-4632.2006.00697.x","article-title":"Leading indicators and spatial interactions: A crime-forecasting model for proactive police deployment","volume":"39","author":"Cohen","year":"2007","journal-title":"Geogr. Anal."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yu, C.H., Ward, M.W., Morabito, M., and Ding, W. (2011, January 11). Crime forecasting using data mining techniques. Proceedings of the International Conference on Data Mining, ICDM\u201911, Vancouver, BC, Canada.","DOI":"10.1109\/ICDMW.2011.56"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kang, H.W., and Kang, H.B. (2017). Prediction of crime occurrence from multi-modal data using deep learning. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0176244"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.knosys.2018.03.010","article-title":"Applying automatic text-based detection of deceptive language to police reports: Extracting behavioral patterns from a multi-step classification model to understand how we lie to the police","volume":"149","author":"Liberatore","year":"2018","journal-title":"Knowl.-Based Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1057\/palgrave.sj.8350066","article-title":"The utility of hotspot mapping for predicting spatial patterns of crime","volume":"21","author":"Chainey","year":"2008","journal-title":"Secur. J."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.apgeog.2018.08.001","article-title":"A spatio-temporal kernel density estimation framework for predictive crime hotspot mapping and evaluation","volume":"99","author":"Hu","year":"2018","journal-title":"Appl. Geogr."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.dss.2015.04.012","article-title":"A decision support system for predictive police patrolling","volume":"75","author":"Liberatore","year":"2015","journal-title":"Decis. Support Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1198\/jasa.2011.ap09546","article-title":"Self-exciting point process modeling of crime","volume":"106","author":"Mohler","year":"2011","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_28","unstructured":"Bendler, J., Ratku, A., and Neumann, D. (2014, January 14\u201317). Crime Mapping through Geo-Spatial Social Media Activity. Proceedings of the Thirty FifthInternational Conference on Information Systems, Auckland, New Zwaland."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"De Choudhury, M., Sundaram, H., John, A., and Seligmann, D.D. (2010). Analyzing the dynamics of communication in online social networks. Handbook of Social Network Technologies and Applications, Springer.","DOI":"10.1007\/978-1-4419-7142-5_4"},{"key":"ref_30","unstructured":"Bermingham, A., and Smeaton, A. (2011, January 13). On using Twitter to monitor political sentiment and predict election results. Proceedings of the Workshop on Sentiment Analysis Where AI meets Psychology (SAAIP 2011), Chiang Mai, Thailand."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"e2353","DOI":"10.1136\/bmj.e2353","article-title":"Can Twitter predict disease outbreaks?","volume":"344","author":"Zorlu","year":"2012","journal-title":"BMJ"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1108\/IntR-06-2012-0114","article-title":"Understanding the predictive power of social media","volume":"23","author":"Kalampokis","year":"2013","journal-title":"Internet Res."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Culotta, A. (2010, January 25\u201328). Towards detecting influenza epidemics by analyzing Twitter messages. Proceedings of the First Workshop on Social Media Analytics, Washington, DC, Canada.","DOI":"10.1145\/1964858.1964874"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, X., Brown, D.E., and Gerber, M.S. (2012, January 11\u201314). Spatio-temporal modeling of criminal incidents using geographic, demographic, and Twitter-derived information. Proceedings of the 2012 IEEE International Conference on Intelligence and Security Informatics (ISI), Arlington, VA, USA.","DOI":"10.1109\/ISI.2012.6284088"},{"key":"ref_35","first-page":"22","article-title":"Designing a social-broadcasting-based business intelligence system","volume":"2","author":"Rui","year":"2011","journal-title":"ACM Trans. Manag. Inf. Syst. TMIS"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Earle, P.S., Bowden, D.C., and Guy, M. (2012). Twitter earthquake detection: earthquake monitoring in a social world. Ann. Geophys., 54.","DOI":"10.4401\/ag-5364"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1111\/j.1475-4932.2012.00809.x","article-title":"Predicting the present with Google Trends","volume":"88","author":"Choi","year":"2012","journal-title":"Econ. Rec."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jocs.2010.12.007","article-title":"Twitter mood predicts the stock market","volume":"2","author":"Bollen","year":"2011","journal-title":"J. Comput. Sci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1016\/j.dss.2014.02.003","article-title":"Predicting crime using Twitter and kernel density estimation","volume":"61","author":"Gerber","year":"2014","journal-title":"Decis. Support Syst."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, X., Cho, Y., and Jang, S.Y. (2015, January 24). Crime prediction using twitter sentiment and weather. Proceedings of the Systems and Information Engineering Design Symposium (SIEDS), Charlottesville, VA, USA.","DOI":"10.1109\/SIEDS.2015.7117012"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Liu, B. (2015). Sentiment Analysis: Mining Opinions, Sentiments, and Emotions, Cambridge University Press.","DOI":"10.1017\/CBO9781139084789"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Metaxas, P.T., Mustafaraj, E., and Gayo-Avello, D. (2011, January 9\u201311). How (not) to predict elections. Proceedings of the 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, Boston, MA, USA.","DOI":"10.1109\/PASSAT\/SocialCom.2011.98"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1016\/j.eswa.2017.02.002","article-title":"Enhancing deep learning sentiment analysis with ensemble techniques in social applications","volume":"77","author":"Araque","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Sah\u0131n, E., Aydos, M., and Orhan, F. (2018, January 2\u20135). Spam\/ham e-mail classification using machine learning methods based on bag of words technique. Proceedings of the 2018 26th Signal Processing and Communications Applications Conference (SIU), Izmir, Turkey.","DOI":"10.1109\/SIU.2018.8404347"},{"key":"ref_45","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_46","unstructured":"Ott, M., Choi, Y., Cardie, C., and Hancock, J.T. (2011, January 19\u201324). Finding deceptive opinion spam by any stretch of the imagination. Proceedings of the Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, Portland, OR, USA."},{"key":"ref_47","unstructured":"Ott, M., Cardie, C., and Hancock, J.T. (2013, January 9\u201314). Negative Deceptive Opinion Spam. Proceedings of the HLT-NAACL."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Hern\u00e1ndez Fusilier, D., Montes-y G\u00f3mez, M., Rosso, P., and Guzm\u00e1n Cabrera, R. (2015). Detection of opinion spam with character n-grams. International Conference on Intelligent Text Processing and Computational Linguistics, Springer.","DOI":"10.1007\/978-3-319-18117-2_21"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1142\/S0218488517400165","article-title":"Detecting Deceptive Opinions: Intra and Cross-Domain Classification Using an Efficient Representation","volume":"25","author":"Cagnina","year":"2017","journal-title":"Int. J. Uncertainty Fuzziness Knowl.-Based Syst."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Mihalcea, R., and Strapparava, C. (2009, January 4). The Lie Detector: Explorations in the Automatic Recognition of Deceptive Language. Proceedings of the International Joint Conference on Natural Language Processing, AFNLP\u201909, Singapore.","DOI":"10.3115\/1667583.1667679"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Li, J., Ott, M., Cardie, C., and Hovy, E. (2014, January 22\u201327). Towards a general rule for identifying deceptive opinion spam. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, MD, USA.","DOI":"10.3115\/v1\/P14-1147"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., and Bhamidipati, N. (2015, January 18\u201322). Hate speech detection with comment embeddings. Proceedings of the 24th International Conference on World Wide Web, Florence, Italy.","DOI":"10.1145\/2740908.2742760"},{"key":"ref_53","unstructured":"Zia, T., Akram, M., Nawaz, M., Shahzad, B., Abdullatif, A., Mustafa, R., and Lali, M. (2016, January 13). Identification of hatred speeches on Twitter. Proceedings of the 52nd The IRES International Conference."},{"key":"ref_54","unstructured":"Silva, L., Mondal, M., Correa, D., Benevenuto, F., and Weber, I. (2016, January 17\u201320). Analyzing the targets of hate in online social media. Proceedings of the Tenth International AAAI Conference on Web and Social Media, Cologne, Germany."},{"key":"ref_55","unstructured":"Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., and Wojatzki, M. (2017). Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv."},{"key":"ref_56","unstructured":"Del Vigna, F., Cimino, A., Dell\u2019Orletta, F., Petrocchi, M., and Tesconi, M. (2017, January 17\u201320). Hate me, hate me not: Hate speech detection on facebook. Proceedings of the First Italian Conference on Cybersecurity, Venice, Italy."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Salminen, J., Almerekhi, H., Milenkovi\u0107, M., Jung, S.G., An, J., Kwak, H., and Jansen, B.J. (2018, January 25\u201328). Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online news media. Proceedings of the Twelfth International AAAI Conference on Web and Social Media, Palo Alto, CA, USA.","DOI":"10.1609\/icwsm.v12i1.15028"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.eswa.2018.03.058","article-title":"A recent overview of the state-of-the-art elements of text classification","volume":"106","author":"Protasiewicz","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_59","unstructured":"Schimid, H. (2019, October 26). Probabilistic Part-of-Speech Tagging Using Decision Trees. Available online: https:\/\/www.cis.uni-muenchen.de\/~schmid\/tools\/TreeTagger\/data\/tree-tagger1.pdf."},{"key":"ref_60","unstructured":"Schmid, H. (2019, October 18). Spanish Tagset Documentation. Available online: https:\/\/www.cis.uni-muenchen.de\/~schmid\/tools\/TreeTagger\/data\/spanish-tagset.txt."},{"key":"ref_61","unstructured":"Kusner, M., Sun, Y., Kolkin, N., and Weinberger, K. (2015, January 6\u201311). From word embeddings to document distances. Proceedings of the International Conference on Machine Learning, Lille, France."},{"key":"ref_62","unstructured":"Liu, H., and Motoda, H. (2012). Feature Selection for Knowledge Discovery and Data Mining, Springer Science & Business Media."},{"key":"ref_63","unstructured":"Liu, H., and Setiono, R. (1996, January 4\u20137). Feature selection and classification-a probabilistic wrapper approach. Proceedings of the 9th International Conference on Industrial and Engineering Applications of AI and ES, Fukuoka, Japan."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.knosys.2012.06.005","article-title":"A novel probabilistic feature selection method for text classification","volume":"36","author":"Uysal","year":"2012","journal-title":"Knowl.-Based Syst."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression Shrinkage and Selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. R. Stat. Soc. Ser. B Methodol."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1126\/science.aaa8415","article-title":"Machine learning: Trends, perspectives, and prospects","volume":"349","author":"Jordan","year":"2015","journal-title":"Science"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/4235.585893","article-title":"No free lunch theorems for optimization","volume":"1","author":"Wolpert","year":"1997","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Lin, Y.L., Yen, M.F., and Yu, L.C. (2018). Grid-based crime prediction using geographical features. ISPRS Int. J. Geo-Inf., 7.","DOI":"10.3390\/ijgi7080298"},{"key":"ref_70","unstructured":"Haykin, S.S. (2009). Neural Networks and Learning Machines\/Simon Haykin, Prentice Hall."},{"key":"ref_71","first-page":"37","article-title":"Evaluation: From precision, recall and f-measure to roc., informedness, markedness & correlation","volume":"2","author":"Powers","year":"2011","journal-title":"J. Mach. Learn. Technol."},{"key":"ref_72","unstructured":"Twitter Inc. (2019, October 18). Twitter Developers. Available online: https:\/\/developer.twitter.com\/en.html."},{"key":"ref_73","unstructured":"Real Academia Espa\u00f1ola (2014). Diccionario de la lengua espa\u00f1ola [Dictionary of the Spanish Language], Espasa. [23rd ed.]."},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Lilleberg, J., Zhu, Y., and Zhang, Y. (2015, January 6\u20138). Support vector machines and word2vec for text classification with semantic features. Proceedings of the 2015 IEEE 14th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC), Beijing, China.","DOI":"10.1109\/ICCI-CC.2015.7259377"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Graves, A. (2012). Supervised Sequence Labelling with Recurrent Neural Networks, Springer.","DOI":"10.1007\/978-3-642-24797-2"},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: The kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem. Med. Biochem. Med."},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit","volume":"70","author":"Cohen","year":"1968","journal-title":"Psychol. Bull."},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychol. Bull."},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_80","first-page":"2009","article-title":"Twitter sentiment classification using distant supervision","volume":"1","author":"Go","year":"2009","journal-title":"CS224N Proj. Rep. Stanf."},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., and McClosky, D. (2014, January 23-24). The Stanford CoreNLP natural language processing toolkit. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Baltimore, MD, USA.","DOI":"10.3115\/v1\/P14-5010"},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Narayanan, V., Arora, I., and Bhatia, A. (2013). Fast and accurate sentiment classification using an enhanced Naive Bayes model. International Conference on Intelligent Data Engineering and Automated Learning, Springer.","DOI":"10.1007\/978-3-642-41278-3_24"},{"key":"ref_83","first-page":"2063","article-title":"Pattern for python","volume":"13","author":"Smedt","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_84","unstructured":"Loria, S. (2019, October 26). Textblob Documentation. Available online: https:\/\/buildmedia.readthedocs.org\/media\/pdf\/textblob\/dev\/textblob.pdf."},{"key":"ref_85","unstructured":"Kathuria, P. (2019, October 26). Sentiment wsd github repository. Available online: https:\/\/github.com\/kevincobain2000\/sentiment$_$classifier."},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Hutto, C.J., and Gilbert, E. (2014, January 1\u20134). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media, Ann Arbor, MI, USA.","DOI":"10.1609\/icwsm.v8i1.14550"},{"key":"ref_87","unstructured":"Saif, H., Fernandez, M., He, Y., and Alani, H. (2013, January 3). Evaluation datasets for Twitter sentiment analysis: A survey and a new dataset, the STS-Gold. Proceedings of the 1st Interantional Workshop on Emotion and Sentiment in Social and Expressive Media: Approaches and Perspectives from AI (ESSEM 2013), Turin, Italy."},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1177\/016555150202800601","article-title":"Social network analysis: A powerful strategy, also for the information sciences","volume":"28","author":"Otte","year":"2002","journal-title":"J. Inf. Sci."},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"3866","DOI":"10.1016\/j.comnet.2012.08.009","article-title":"Online social networks: A survey of a global phenomenon","volume":"56","author":"Heidemann","year":"2012","journal-title":"Comput. Netw."},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1016\/j.future.2016.04.009","article-title":"Clustering of trending topics in microblogging posts: A graph-based approach","volume":"67","author":"Hachaj","year":"2017","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1016\/j.tele.2015.04.008","article-title":"The anatomy of tweet overload: How number of tweets received, number of friends, and egocentric network density affect perceived information overload","volume":"32","author":"Sasaki","year":"2015","journal-title":"Telemat. Inform."},{"key":"ref_92","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"Xing, W., and Ghorbani, A. (2004, January 21). Weighted pagerank algorithm. Proceedings of the Second Annual Conference on Communication Networks and Services Research, Fredericton, NB, Canada.","DOI":"10.1109\/DNSR.2004.1344743"},{"key":"ref_94","first-page":"6","article-title":"Community detection in social networks: An overview","volume":"2","author":"Choudhury","year":"2013","journal-title":"Int. J. Res. Eng. Technol."},{"key":"ref_95","unstructured":"P\u00e9rez Colom\u00e9, J. (2019, October 18). This Is How Hate Spreads When There Are Elections in Spain (As\u00ed se reparte el odio cuando hay elecciones en Espa\u00f1a). Available online: https:\/\/elpais.com\/tecnologia\/2019\/05\/28\/actualidad\/1559039892_332196.html."},{"key":"ref_96","unstructured":"La Moncloa (Spanish Presidency) (2019, October 18). The Secretary of State for the Digital Advancement Organizes a Conference on Protection of LGTBI Rights on the Internet (La Secretar\u00eda de Estado para el Avance Digital organiza una jornada sobre protecci\u00f3n de derechos LGTBI en Internet). Available online: http:\/\/www.mineco.gob.es\/portal\/site\/mineco\/menuitem.ac30f9268750bd56a0b0240e026041a0\/?vgnextoid=b1ad544a1929b610VgnVCM1000001d04140aRCRD&vgnextchannel=864e154527515310VgnVCM1000001d04140aRCRD."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/21\/4654\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:29:35Z","timestamp":1760189375000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/21\/4654"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,26]]},"references-count":96,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2019,11]]}},"alternative-id":["s19214654"],"URL":"https:\/\/doi.org\/10.3390\/s19214654","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,26]]}}}