{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T19:33:16Z","timestamp":1769283196371,"version":"3.49.0"},"reference-count":73,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,1,13]],"date-time":"2023-01-13T00:00:00Z","timestamp":1673568000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>The correct detection of negations is essential to the performance of sentiment analysis tools. The evaluation of such tools is currently conducted through the use of corpora as an opportunistic approach. In this paper, we advocate using a different evaluation approach based on a set of intentionally built sentences that include negations, which aim to highlight those tools\u2019 vulnerabilities. To demonstrate the effectiveness of this approach, we propose a basic testset of such sentences. We employ that testset to evaluate six popular sentiment analysis tools (with eight lexicons) available as packages in the R language distribution. By adopting a supervised classification approach, we show that the performance of most of these tools is largely unsatisfactory.<\/jats:p>","DOI":"10.3390\/computers12010018","type":"journal-article","created":{"date-parts":[[2023,1,13]],"date-time":"2023-01-13T02:05:52Z","timestamp":1673575552000},"page":"18","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["A Testset-Based Method to Analyse the Negation-Detection Performance of Lexicon-Based Sentiment Analysis Tools"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0903-398X","authenticated-orcid":false,"given":"Maurizio","family":"Naldi","sequence":"first","affiliation":[{"name":"Department of Law, Economics, Politics and Modern Languages, LUMSA University, Via Marcantonio Colonna 19, 00192 Rome, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6856-947X","authenticated-orcid":false,"given":"Sandra","family":"Petroni","sequence":"additional","affiliation":[{"name":"Department of Literary, Philosophical and Art History Studies, School of Humanities, Tor Vergata University, Via Columbia 1, 00133 Rome, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1145\/2436256.2436274","article-title":"Techniques and applications for sentiment analysis","volume":"56","author":"Feldman","year":"2013","journal-title":"Commun. ACM"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1093","DOI":"10.1016\/j.asej.2014.04.011","article-title":"Sentiment analysis algorithms and applications: A survey","volume":"5","author":"Medhat","year":"2014","journal-title":"Ain Shams Eng. J."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Joachims, T. (1999). Making large-scale support vector machine learning practical. Advances in Kernel Methods: Support Vector Learning, MIT Press.","DOI":"10.7551\/mitpress\/1130.003.0015"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Turney, P.D. (2002). Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. arXiv.","DOI":"10.3115\/1073083.1073153"},{"key":"ref_5","first-page":"143","article-title":"Sentiment analysis: A combined approach","volume":"3","author":"Prabowo","year":"2009","journal-title":"J. Inf."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e1253","DOI":"10.1002\/widm.1253","article-title":"Deep learning for sentiment analysis: A survey","volume":"8","author":"Zhang","year":"2018","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wang, J., Xu, B., and Zu, Y. (2021, January 9\u201311). Deep learning for aspect-based sentiment analysis. Proceedings of the 2021 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), Chongqing, China.","DOI":"10.1109\/MLISE54096.2021.00056"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.dss.2018.09.002","article-title":"Deep learning for affective computing: Text-based emotion recognition in decision support","volume":"115","author":"Kratzwald","year":"2018","journal-title":"Decis. Support Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1109\/MIS.2016.31","article-title":"Affective Computing and Sentiment Analysis","volume":"31","author":"Cambria","year":"2016","journal-title":"IEEE Intell. Syst."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Cambria, E., Das, D., Bandyopadhyay, S., and Feraco, A. (2017). Affective computing and sentiment analysis. A Practical Guide to Sentiment Analysis, Springer.","DOI":"10.1007\/978-3-319-55394-8"},{"key":"ref_11","first-page":"801","article-title":"Bibliometric analysis of affective computing researches during 1999\u223c 2018","volume":"36","author":"Guo","year":"2020","journal-title":"Int. J. Hum.- Interact."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.future.2020.08.005","article-title":"ABCDM: An attention-based bidirectional CNN-RNN deep model for sentiment analysis","volume":"115","author":"Basiri","year":"2021","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1016\/j.future.2020.07.022","article-title":"Attention-based sentiment analysis using convolutional and recurrent neural network","volume":"113","author":"Usama","year":"2020","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Cambria, E., Li, Y., Xing, F.Z., Poria, S., and Kwok, K. (2020, January 19\u201323). SenticNet 6: Ensemble application of symbolic and subsymbolic AI for sentiment analysis. Proceedings of the 29th ACM International Conference on Information & knowledge Management, Virtual Event, Ireland.","DOI":"10.1145\/3340531.3412003"},{"key":"ref_15","first-page":"330","article-title":"A survey on sentiment analysis challenges","volume":"30","author":"Hussein","year":"2018","journal-title":"J. King Saud Univ.-Eng. Sci."},{"key":"ref_16","unstructured":"Heerschop, B., van Iterson, P., Hogenboom, A., Frasincar, F., and Kaymak, U. (2011, January 4). Accounting for negation in sentiment analysis. Proceedings of the 11th Dutch-Belgian Information Retrieval Workshop (DIR 2011). Citeseer, Amsterdam, The Netherlands."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Hogenboom, A., Van Iterson, P., Heerschop, B., Frasincar, F., and Kaymak, U. (2011, January 9\u201312). Determining negation scope and strength in sentiment analysis. Proceedings of the 2011 IEEE International Conference on Systems, Man and Cybernetics, Anchorage, AK, USA.","DOI":"10.1109\/ICSMC.2011.6084066"},{"key":"ref_18","unstructured":"Asmi, A., and Ishaya, T. (2012, January 21\u201326). Negation identification and calculation in sentiment analysis. Proceedings of the second international conference on advances in information mining and management, Venice, Italy."},{"key":"ref_19","unstructured":"Dadvar, M., Hauff, C., and de Jong, F. (2011, January 27\u201328). Scope of negation detection in sentiment analysis. Proceedings of the Dutch-Belgian Information Retrieval Workshop (DIR 2011). Citeseer, Amsterdam, The Netherlands."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jia, L., Yu, C., and Meng, W. (2009, January 2\u20136). The effect of negation on sentiment analysis and retrieval effectiveness. Proceedings of the 18th ACM Conference on Information and Knowledge Management, Hong Kong, China.","DOI":"10.1145\/1645953.1646241"},{"key":"ref_21","unstructured":"Remus, R. (2013, January 3). Modeling and Representing Negation in Data-driven Machine Learning-based Sentiment Analysis. Proceedings of the ESSEM@ AI*IA, Turin, Italy."},{"key":"ref_22","unstructured":"Wiegand, M., Balahur, A., Roth, B., Klakow, D., and Montoyo, A. (2010, January 10). A survey on the role of negation in sentiment analysis. Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, Uppsala, Sweden."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lapponi, E., Read, J., and \u00d8vrelid, L. (2012, January 10\u201313). Representing and resolving negation for sentiment analysis. Proceedings of the 2012 IEEE 12th International Conference on Data Mining Workshops, Brussels, Belgium.","DOI":"10.1109\/ICDMW.2012.23"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"470","DOI":"10.17706\/jcp.12.5.470-478","article-title":"Negation Handling in Sentiment Analysis at Sentence Level","volume":"12","author":"Farooq","year":"2017","journal-title":"JCP"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Barnes, J., Velldal, E., and \u00d8vrelid, L. (2019). Improving sentiment analysis with multi-task learning of negation. arXiv.","DOI":"10.1017\/S1351324920000510"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/j.ins.2020.05.022","article-title":"Negation scope detection for sentiment analysis: A reinforcement learning framework for replicating human interpretations","volume":"536","author":"Feuerriegel","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_27","unstructured":"D\u00edaz, N.P.C., and L\u00f3pez, M.J.M. (2019). Negation and speculation detection, John Benjamins Publishing Company."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/coli_a_00371","article-title":"Corpora Annotated with Negation: An Overview","volume":"46","author":"Morante","year":"2020","journal-title":"Comput. Linguist."},{"key":"ref_29","unstructured":"Councill, I.G., McDonald, R., and Velikovich, L. (2010, January 10). What\u2019s great and what\u2019s not: Learning to classify the scope of negation for improved sentiment analysis. Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, Uppsala, Sweden."},{"key":"ref_30","unstructured":"Jespersen, O. (1917). Negation in English and Other Languages, Host."},{"key":"ref_31","unstructured":"Horn, L. (1989). A Natural History of Negation, University of Chicago Press."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1162\/COLI_a_00095","article-title":"Modality and negation: An introduction to the special issue","volume":"38","author":"Morante","year":"2012","journal-title":"Comput. Linguist."},{"key":"ref_33","first-page":"65","article-title":"Negation in syntax: On the nature of functional categories and projections","volume":"25","author":"Laka","year":"1990","journal-title":"Anu. Semin. Filol. Vasca Julio Urquijo"},{"key":"ref_34","unstructured":"Tottie, G. (1991). Negation in English Speech and Writing: A Study in Variation, Academic Press."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Huddleston, R.D., and Pullum, G.K. (2002). The Cambridge Grammar of the English Language, Cambridge University Press.","DOI":"10.1017\/9781316423530"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Liu, B. (2015). Sentiment Analysis: Mining Opinions, Sentiments and Emotions, Cambridge University Press.","DOI":"10.1017\/CBO9781139084789"},{"key":"ref_37","unstructured":"De Albornoz, J.C., Plaza, L., D\u00edaz, A., and Ballesteros, M. (2012, January 7\u20138). Ucm-i: A rule-based syntactic approach for resolving the scope of negation. Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the Main Conference and the Shared Task and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, Montr\u00e9al, QC, Canada."},{"key":"ref_38","unstructured":"Ballesteros, M., Francisco, V., D\u00edaz, A., Herrera, J., and Gerv\u00e1s, P. Inferring the scope of negation in biomedical documents. Proceedings of the International Conference on Intelligent Text Processing and Computational Linguistics."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2118","DOI":"10.1002\/asi.23533","article-title":"A machine learning approach to negation and speculation detection for sentiment analysis","volume":"67","author":"Cruz","year":"2016","journal-title":"J. Assoc. Inf. Sci. Technol."},{"key":"ref_40","unstructured":"Skeppstedt, M., Paradis, C., and Kerren, A. (2016, January 4\u20135). Marker Words for Negation and Speculation in Health Records and Consumer Reviews. Proceedings of the 7th International Symposium on Semantic Mining in Biomedicine (SMBM), Potsdam, Germany."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Qian, Z., Li, P., Zhu, Q., Zhou, G., Luo, Z., and Luo, W. (2016, January 1\u20135). Speculation and negation scope detection via convolutional neural networks. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1078"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1504\/IJHPCN.2019.097501","article-title":"Negation scope detection with recurrent neural networks models in review texts","volume":"13","author":"Lazib","year":"2019","journal-title":"Int. J. High Perform. Comput. Netw."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","year":"2001","journal-title":"J. Biomed. Inform."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"839","DOI":"10.1016\/j.jbi.2009.05.002","article-title":"ConText: An algorithm for determining negation, experiencer and temporal status from clinical reports","volume":"42","author":"Harkema","year":"2009","journal-title":"J. Biomed. Inform."},{"key":"ref_45","first-page":"187","article-title":"Syntactical negation detection in clinical practice guidelines","volume":"136","author":"Gindl","year":"2008","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_46","unstructured":"Konstantinova, N., De Sousa, S.C., D\u00edaz, N.P.C., L\u00f3pez, M.J.M., Taboada, M., and Mitkov, R. (2012, January 21\u201327). A review corpus annotated for negation, speculation and their scope. Proceedings of the LREC, Istanbul, Turkey."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Reitan, J., Faret, J., Gamb\u00e4ck, B., and Bungum, L. (2015, January 17). Negation scope detection for Twitter sentiment analysis. Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Lisbon, Portugal.","DOI":"10.18653\/v1\/W15-2914"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s41701-019-00065-w","article-title":"The SFU opinion and comments corpus: A corpus for the analysis of online news comments","volume":"4","author":"Kolhatkar","year":"2019","journal-title":"Corpus Pragmat."},{"key":"ref_49","first-page":"14","article-title":"Negminer: An automated tool for mining negations from electronic narrative medical documents","volume":"9","author":"Elazhary","year":"2017","journal-title":"Int. J. Intell. Syst. Appl."},{"key":"ref_50","first-page":"273","article-title":"Building a Large Annotated Corpus of English: The Penn Treebank","volume":"19","author":"Marcus","year":"1994","journal-title":"Comput. Linguist."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Taylor, A., Marcus, M., and Santorini, B. (2003). The Penn Treebank: An overview. Treebanks, Springer.","DOI":"10.1007\/978-94-010-0201-1_1"},{"key":"ref_52","unstructured":"Santorini, B. (1990). Part-of-Speech Tagging Guidelines for the Penn Treebank Project (3rd Revision), University of Pennsylvania. MS-CIS-90-47."},{"key":"ref_53","unstructured":"Misuraca, M., Forciniti, A., Scepi, G., and Spano, M. (2020). Sentiment Analysis for Education with R: Packages, methods and practical applications. arXiv."},{"key":"ref_54","unstructured":"Naldi, M. (2019). A review of sentiment computation methods with R packages. arXiv."},{"key":"ref_55","unstructured":"Bose, S., Saha, U., Kar, D., Goswami, S., Nayak, A.K., and Chakrabarti, S. RSentiment: A tool to extract meaningful insights from textual reviews. Proceedings of the 5th International Conference on Frontiers in Intelligent Computing: Theory and Applications."},{"key":"ref_56","unstructured":"Jockers, M. (2023, January 10). Package \u2018Syuzhet\u2019. Available online: https:\/\/cran.r-project.org\/web\/packages\/syuzhet\/index.html."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MIS.2017.3121555","article-title":"Sentiment analysis in Tripadvisor","volume":"32","author":"Valdivia","year":"2017","journal-title":"IEEE Intell. Syst."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/j.dcm.2018.05.001","article-title":"The appeal to political sentiment: An analysis of Donald Trump\u2019s and Hillary Clinton\u2019s speech themes and discourse strategies in the 2016 US presidential election","volume":"25","author":"Liu","year":"2018","journal-title":"Discourse Context Media"},{"key":"ref_59","first-page":"1292","article-title":"Comparison of different algorithms for sentiment analysis: Psychological stress notes","volume":"245","author":"Yoon","year":"2017","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_60","unstructured":"Rinker, T. (2023, January 10). Package \u2018Sentimentr\u2019. Available online: https:\/\/cran.r-project.org\/web\/packages\/sentimentr\/index.html."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Ikoro, V., Sharmina, M., Malik, K., and Batista-Navarro, R. (2018, January 15\u201318). Analyzing sentiments expressed on Twitter by UK energy company consumers. Proceedings of the 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), Valencia, Spain.","DOI":"10.1109\/SNAMS.2018.8554619"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.jbi.2018.12.001","article-title":"Construct validity of six sentiment analysis methods in the text of encounter notes of patients with critical illness","volume":"89","author":"Weissman","year":"2019","journal-title":"J. Biomed. Inform."},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Pr\u00f6llochs, N., Feuerriegel, S., and Neumann, D. (2015, January 26\u201329). Generating Domain-Specific Dictionaries using Bayesian Learning. Proceedings of the European Conference on Information Systems (ECIS), M\u00fcnster, Germany.","DOI":"10.2139\/ssrn.2522884"},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Kassraie, P., Modirshanechi, A., and Aghajan, H.K. (2017, January 24\u201326). Election Vote Share Prediction using a Sentiment-based Fusion of Twitter Data with Google Trends and Online Polls. Proceedings of the 6th International Conference on Data Science, Technology and Applications (DATA 2017), Madrid, Spain.","DOI":"10.5220\/0006484303630370"},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1007\/s12469-018-0184-4","article-title":"Using Twitter data for transit performance assessment: A framework for evaluating transit riders\u2019 opinions about quality of service","volume":"10","author":"Haghighi","year":"2018","journal-title":"Public Transp."},{"key":"ref_66","unstructured":"Hu, M., and Liu, B. (2004, January 25\u201329). Mining opinion features in customer reviews. Proceedings of the AAAI, San Jose, CA, USA."},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Hutto, C., and Gilbert, E. (2014, January 1\u20134). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Weblogs and Social Media, Ann Arbor, MI, USA.","DOI":"10.1609\/icwsm.v8i1.14550"},{"key":"ref_68","unstructured":"Newman, H., and Joyner, D. Sentiment analysis of student evaluations of teaching. Proceedings of the International Conference on Artificial Intelligence in Education."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"113746","DOI":"10.1016\/j.eswa.2020.113746","article-title":"Using VADER sentiment and SVM for predicting customer response sentiment","volume":"162","author":"Borg","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1142\/S0218488520500294","article-title":"Multi-level fine-scaled sentiment sensing with ambivalence handling","volume":"28","author":"Wang","year":"2020","journal-title":"Int. J. Uncertain. Fuzziness Knowl.-Based Syst."},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Seliya, N., Khoshgoftaar, T.M., and Hulse, J.V. (2009, January 2\u20135). A Study on the Relationships of Classifier Performance Metrics. Proceedings of the 2009 21st IEEE International Conference on Tools with Artificial Intelligence, Newark, NJ, USA.","DOI":"10.1109\/ICTAI.2009.25"},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Zaki, M.J., and Meira Jr, W. (2019). Data Mining and Machine Learning: Fundamental Concepts and Algorithms, Cambridge University Press.","DOI":"10.1017\/9781108564175"},{"key":"ref_73","unstructured":"Murphy, K.P. (2012). Machine Learning: A Probabilistic Perspective, MIT Press."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/1\/18\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:04:50Z","timestamp":1760119490000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/1\/18"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,13]]},"references-count":73,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["computers12010018"],"URL":"https:\/\/doi.org\/10.3390\/computers12010018","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,13]]}}}