{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T13:16:50Z","timestamp":1775135810961,"version":"3.50.1"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,4,9]],"date-time":"2022-04-09T00:00:00Z","timestamp":1649462400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,7,31]]},"abstract":"<jats:p>\n            Sentiment analysis in\n            <jats:bold>software engineering (SE)<\/jats:bold>\n            has shown promise to analyze and support diverse development activities. Recently, several tools are proposed to detect sentiments in software artifacts. While the tools improve accuracy over off-the-shelf tools, recent research shows that their performance could still be unsatisfactory. A more accurate sentiment detector for SE can help reduce noise in analysis of software scenarios where sentiment analysis is required. Recently, combinations, i.e., hybrids of stand-alone classifiers are found to offer better performance than the stand-alone classifiers for fault detection. However, we are aware of no such approach for sentiment detection for software artifacts. We report the results of an empirical study that we conducted to determine the feasibility of developing an ensemble engine by combining the polarity labels of stand-alone SE-specific sentiment detectors. Our study has two phases. In the first phase, we pick five SE-specific sentiment detection tools from two recently published papers by Lin et\u00a0al.\u00a0[\n            <jats:xref ref-type=\"bibr\">29<\/jats:xref>\n            ,\n            <jats:xref ref-type=\"bibr\">30<\/jats:xref>\n            ], who first reported negative results with stand alone sentiment detectors and then proposed an improved SE-specific sentiment detector, POME\u00a0[\n            <jats:xref ref-type=\"bibr\">29<\/jats:xref>\n            ]. We report the study results on 17,581 units (sentences\/documents) coming from six currently available sentiment benchmarks for software engineering. We find that the existing tools can be complementary to each other in 85-95% of the cases, i.e., one is wrong but another is right. However, a majority voting-based ensemble of those tools fails to improve the accuracy of sentiment detection. We develop Sentisead, a supervised tool by combining the polarity labels and bag of words as features. Sentisead improves the performance (F1-score) of the individual tools by 4% (over Senti4SD\u00a0[\n            <jats:xref ref-type=\"bibr\">5<\/jats:xref>\n            ]) \u2013 100% (over POME\u00a0[\n            <jats:xref ref-type=\"bibr\">29<\/jats:xref>\n            ]). The initial development of Sentisead occurred before we observed the use of deep learning models for SE-specific sentiment detection. In particular, recent papers show the superiority of advanced language-based\n            <jats:bold>pre-trained transformer models (PTM)<\/jats:bold>\n            over rule-based and shallow learning models. Consequently, in a second phase, we compare and improve Sentisead infrastructure using the PTMs. We find that a Sentisead infrastructure with RoBERTa as the ensemble of the five stand-alone rule-based and shallow learning SE-specific tools from Lin et\u00a0al.\u00a0[\n            <jats:xref ref-type=\"bibr\">29<\/jats:xref>\n            ,\n            <jats:xref ref-type=\"bibr\">30<\/jats:xref>\n            ] offers the best F1-score of 0.805 across the six datasets, while a stand-alone RoBERTa shows an F1-score of 0.801.\n          <\/jats:p>","DOI":"10.1145\/3491211","type":"journal-article","created":{"date-parts":[[2022,1,31]],"date-time":"2022-01-31T17:28:25Z","timestamp":1643650105000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1376-095X","authenticated-orcid":false,"given":"Gias","family":"Uddin","sequence":"first","affiliation":[{"name":"University of Calgary, Calgary, Canada"}]},{"given":"Yann-Ga\u00ebl","family":"Gu\u00e9h\u00e9nuc","sequence":"additional","affiliation":[{"name":"Concordia University, Montr\u00e9al, Canada"}]},{"given":"Foutse","family":"Khomh","sequence":"additional","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montr\u00e9al, Canada"}]},{"given":"Chanchal K.","family":"Roy","sequence":"additional","affiliation":[{"name":"University of Saskatchewan, Canada"}]}],"member":"320","published-online":{"date-parts":[[2022,4,9]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155579"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.06.005"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00025"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00020"},{"key":"e_1_3_3_6_2","first-page":"2543","article-title":"Sentiment polarity detection for software development","author":"Calefato Fabio","year":"2017","unstructured":"Fabio Calefato, Filippo Lanubile, Federico Maiorano, and Nicole Novielli. 2017. Sentiment polarity detection for software development. Journal Empirical Software Engineering (2017), 2543\u20132584.","journal-title":"Journal Empirical Software Engineering"},{"key":"e_1_3_3_7_2","first-page":"2","volume-title":"Proc. 7th Affective Computing and Intelligent Interaction","author":"Calefato Fabio","year":"2017","unstructured":"Fabio Calefato, Filippo Lanubile, and Nicole Novielli. 2017. EmoTxt: A toolkit for emotion recognition from text. In Proc. 7th Affective Computing and Intelligent Interaction. 2."},{"key":"e_1_3_3_8_2","first-page":"235","volume-title":"IEEE\/ACM 13th Working Conference on Mining Software Repositories","author":"Castaldi C\u00e1ssio","year":"2016","unstructured":"C\u00e1ssio Castaldi, Araujo Blaz, and Karin Becker. 2016. Sentiment analysis in tickets for IT support. In IEEE\/ACM 13th Working Conference on Mining Software Repositories. 235\u2013246."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338977"},{"key":"e_1_3_3_11_2","volume-title":"Applied Multiple Regression\/Correlation Analysis for the Behavioral Sciences (3rd ed.)","author":"Cohen Jacob","year":"2002","unstructured":"Jacob Cohen, Stephen G. West, Leona Aiken, and Patricia Cohen. 2002. Applied Multiple Regression\/Correlation Analysis for the Behavioral Sciences (3rd ed.). Lawrence Erlbaum Associates."},{"key":"e_1_3_3_12_2","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Technical Report. https:\/\/arxiv.org\/abs\/1810.04805."},{"key":"e_1_3_3_13_2","first-page":"428","volume-title":"Proceedings of the 8th International Workshop on Semantic Evaluation","author":"Filho Pedro P. Balage","year":"2014","unstructured":"Pedro P. Balage Filho, Lucas Avanco, Thiago A. S. Pardo, and Maria G. V. Nunes. 2014. NILC_USP: An improved hybrid system for sentiment analysis in Twitter messages. In Proceedings of the 8th International Workshop on Semantic Evaluation. 428\u2013432."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2017.18"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.91"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2512938.2512951"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2016.67"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597073.2597118"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2494578"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070510"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014073"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2017.9"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","first-page":"1536","DOI":"10.1145\/3167132.3167296","volume-title":"33rd Annual ACM Symposium on Applied Computing","author":"Islam Md Rakibul","year":"2018","unstructured":"Md Rakibul Islam and Minhaz F. Zibran. 2018. DEVA: Sensing emotions in the valence arousal space in software engineering text. In 33rd Annual ACM Symposium on Applied Computing. 1536\u20131543."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2015.7332508"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9493-x"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2011.39"},{"key":"e_1_3_3_27_2","first-page":"127","volume-title":"2005 IEEE Symposium on Visual Languages and Human-Centric Computing","author":"Ko Andrew J.","year":"2005","unstructured":"Andrew J. Ko, Brad A. Myers, and Duen Horng Chau. 2005. A linguistic analysis of how people describe software problems. In 2005 IEEE Symposium on Visual Languages and Human-Centric Computing. 127\u2013134."},{"key":"e_1_3_3_28_2","volume-title":"An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets (Online Appendix)","author":"Lab. DISA","unstructured":"DISA Lab. 21 April 2021 (last accessed). An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets (Online Appendix). https:\/\/github.com\/disa-lab\/HybridSESentimentTOSEM."},{"key":"e_1_3_3_29_2","volume-title":"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations","author":"Lan Zhenzhong","year":"2020","unstructured":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Technical Report. https:\/\/arxiv.org\/abs\/1909.11942."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00066"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180195"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.5555\/3019323"},{"key":"e_1_3_3_33_2","volume-title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Technical Report. https:\/\/arxiv.org\/abs\/1907.11692."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-016-0251-9"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2019.2919573"},{"key":"e_1_3_3_36_2","volume-title":"An Introduction to Information Retrieval","author":"Manning Christopher D.","year":"2009","unstructured":"Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch\u00fctze. 2009. An Introduction to Information Retrieval. Cambridge Uni Press."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2901752"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-010-9128-1"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597073.2597086"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.11.016"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387446"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/2804381.2804387"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196453"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196403"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2019.2924013"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/TETCI.2017.2699224"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2015.35"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2903505"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.3115\/1118693.1118704"},{"key":"e_1_3_3_50_2","first-page":"281","volume-title":"IEEE International Conf. on Software Maintenance and Evolution","author":"Panichella Sebastiano","year":"2015","unstructured":"Sebastiano Panichella, Andrea Di Sorbo, Emitza Guzman, Corrado A. Visaggio, Gerardo Canfora, and Harald C. Gall. 2015. How can I improve my app? Classifying user reviews for software maintenance and evolution. In IEEE International Conf. on Software Maintenance and Evolution. 281\u2013290."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962610"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597073.2597117"},{"key":"e_1_3_3_53_2","volume-title":"Machine Learning in Python","author":"learn scikit","year":"2017","unstructured":"scikit learn. 2017. Machine Learning in Python. http:\/\/scikit-learn.org\/stable\/index.html#."},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb01338.x"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2903501"},{"key":"e_1_3_3_57_2","first-page":"12","volume-title":"Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Socher Richard","year":"2013","unstructured":"Richard Socher, Alex Perelygin, Jean Wu, Christopher Manning, Andrew Ng, and Jason Chuang. 2013. Recursive models for semantic compositionality over a sentiment treebank. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP). 12."},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.5555\/1890706.1890713"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9452-6"},{"key":"e_1_3_3_60_2","first-page":"40","article-title":"Understanding how and why developers seek and analyze API related opinions","author":"Uddin Gias","year":"2019","unstructured":"Gias Uddin, Olga Baysal, Latifa Guerroj, and Foutse Khomh. 2019. Understanding how and why developers seek and analyze API related opinions. IEEE Transactions on Software Engineering (2019), 40.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155586"},{"key":"e_1_3_3_62_2","volume-title":"Mining API Aspects in API Reviews","author":"Uddin Gias","year":"2017","unstructured":"Gias Uddin and Foutse Khomh. 2017. Mining API Aspects in API Reviews. Technical Report. https:\/\/swat.polymtl.ca\/data\/opinionvalue-technical-report.pdf."},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155690"},{"key":"e_1_3_3_64_2","first-page":"35","article-title":"Automatic opinion mining from API reviews from stack overflow","author":"Uddin Gias","year":"2019","unstructured":"Gias Uddin and Foutse Khomh. 2019. Automatic opinion mining from API reviews from stack overflow. IEEE Transactions on Software Engineering (2019), 35.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_65_2","first-page":"43","article-title":"Automatic API usage scenario documentation from technical Q&A sites","author":"Uddin Gias","year":"2020","unstructured":"Gias Uddin, Foutse Khomh, and Chanchal K. Roy. 2020. Automatic API usage scenario documentation from technical Q&A sites. ACM Transactions on Software Engineering and Methodology (2020), 43.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_3_66_2","first-page":"16","article-title":"Automatic mining of API usage scenarios from stack overflow","author":"Uddin Gias","year":"2020","unstructured":"Gias Uddin, Foutse Khomh, and Chanchal K. Roy. 2020. Automatic mining of API usage scenarios from stack overflow. Information and Software Technology (IST) (2020), 16.","journal-title":"Information and Software Technology (IST)"},{"key":"e_1_3_3_67_2","volume-title":"Resolving API Mentions in Forum Texts","author":"Uddin Gias","year":"2015","unstructured":"Gias Uddin and Martin P. Robillard. 2015. Resolving API Mentions in Forum Texts. Technical Report. McGill University."},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236062"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884818"},{"key":"e_1_3_3_70_2","first-page":"90","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics","author":"Wang Sida","year":"2012","unstructured":"Sida Wang and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. 90\u201394."},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-012-0314-x"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-4625-2"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.03.007"},{"key":"e_1_3_3_74_2","volume-title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","author":"Yang Zhilin","year":"2020","unstructured":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. XLNet: Generalized Autoregressive Pretraining for Language Understanding. Technical Report. https:\/\/arxiv.org\/abs\/1906.08237."},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00017"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-017-6015-y"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491211","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3491211","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:18Z","timestamp":1750183758000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491211"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,9]]},"references-count":75,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,7,31]]}},"alternative-id":["10.1145\/3491211"],"URL":"https:\/\/doi.org\/10.1145\/3491211","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,9]]},"assertion":[{"value":"2020-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}