{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T14:24:16Z","timestamp":1775571856415,"version":"3.50.1"},"reference-count":93,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2017,1,10]],"date-time":"2017-01-10T00:00:00Z","timestamp":1484006400000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003005","name":"Eindhoven University of Technology","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003005","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2017,10]]},"DOI":"10.1007\/s10664-016-9493-x","type":"journal-article","created":{"date-parts":[[2017,1,10]],"date-time":"2017-01-10T04:04:57Z","timestamp":1484021097000},"page":"2543-2584","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":145,"title":["On negative results when using sentiment analysis tools for software engineering research"],"prefix":"10.1007","volume":"22","author":[{"given":"Robbert","family":"Jongeling","sequence":"first","affiliation":[]},{"given":"Proshanta","family":"Sarkar","sequence":"additional","affiliation":[]},{"given":"Subhajit","family":"Datta","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1418-0095","authenticated-orcid":false,"given":"Alexander","family":"Serebrenik","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2017,1,10]]},"reference":[{"key":"9493_CR1","unstructured":"Abbasi A, Hassan A, Dhar M (2014) Benchmarking Twitter sentiment analysis tools. In: International Conference on Language Resources and Evaluation. ELRA, Reykjavik, Iceland, pp 823\u2013829"},{"key":"9493_CR2","unstructured":"Agarwal A, Xie B, Vovsha I, Rambow O, Passonneau R (2011) Sentiment Analysis of Twitter Data. In: Proceedings of the Workshop on Languages in Social Media, LSM \u201911, pp 30\u201338. Association for Computational Linguistics, Stroudsburg, PA, USA. http:\/\/dl.acm.org\/citation.cfm?id=2021109.2021114"},{"key":"9493_CR3","doi-asserted-by":"publisher","unstructured":"Asaduzzaman M, Bullock MC, Roy CK, Schneider KA Bug introducing changes: A case study with android. In: Lanza et al. [43], pp 116\u2013119. doi: 10.1109\/MSR.2012.6224267","DOI":"10.1109\/MSR.2012.6224267"},{"key":"9493_CR4","unstructured":"Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC\u201910). European Language Resources Association (ELRA), Valletta, Malta. http:\/\/www.lrec-conf.org\/proceedings\/lrec2010\/pdf\/769_Paper.pdf"},{"key":"9493_CR5","doi-asserted-by":"crossref","unstructured":"Bakeman R, Gottman JM (1997) Observing interaction: an introduction to sequential analysis. Cambridge University Press. https:\/\/books.google.nl\/books?id=CMj2SmcijhEC","DOI":"10.1017\/CBO9780511527685"},{"key":"9493_CR6","doi-asserted-by":"crossref","unstructured":"Barkmann H, Lincke R, L\u00f6we W (2009) Quantitative evaluation of software quality metrics in open-source projects. In: IEEE International Workshop on Quantitative Evaluation of large-scale Systems and Technologies, pp 1067\u20131072","DOI":"10.1109\/WAINA.2009.190"},{"key":"9493_CR7","doi-asserted-by":"crossref","unstructured":"Batista GEAPA, Carvalho ACPLF, Monard MC (2000) Applying one-sided selection to unbalanced datasets. In: Mexican International Conference on Artificial Intelligence: Advances in Artificial Intelligence. Springer-Verlag, London, UK, UK, pp 315\u2013325","DOI":"10.1007\/10720076_29"},{"key":"9493_CR8","unstructured":"Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, Grenning J, Highsmith J, Hunt A, Jeffries R, Kern J, Marick B, Martin RC, Mellor S, Schwaber K, Sutherland J, Thomas D (2001) Manifesto for agile software development http:\/\/agilemanifesto.org\/principles.html Last accessed: October 14, 2015"},{"key":"9493_CR9","unstructured":"Bird S, Loper E, Klein E (2009) Natural language processing with Python. O\u2019Reilly Media Inc"},{"issue":"1","key":"9493_CR10","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1002\/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U","volume":"42","author":"E Brunner","year":"2000","unstructured":"Brunner E, Munzel U (2000) The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample Approximation. Biometrical Journal 42(1):17\u201325","journal-title":"Biometrical Journal"},{"key":"9493_CR11","doi-asserted-by":"publisher","unstructured":"Capiluppi A, Serebrenik A, Singer L (2013) Assessing technical candidates on the social web. Software. IEEE 30(1):45\u201351. doi: 10.1109\/MS.2012.169","DOI":"10.1109\/MS.2012.169"},{"key":"9493_CR12","doi-asserted-by":"publisher","unstructured":"Cataldo M, Herbsleb JD (2008) Communication networks in geographically distributed software development. In: Proceedings of the 2008 ACM conference on Computer supported cooperative work, CSCW \u201908, pp 579\u2013588. ACM, New York, NY, USA. doi: 10.1145\/1460563.1460654","DOI":"10.1145\/1460563.1460654"},{"issue":"4","key":"9493_CR13","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","volume":"70","author":"J Cohen","year":"1968","unstructured":"Cohen J (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 70(4):213\u2013220","journal-title":"Psychol Bull"},{"key":"9493_CR14","doi-asserted-by":"publisher","unstructured":"Costa JM, Cataldo M, de Souza CR (2011) The scale and evolution of coordination needs in large-scale distributed projects: implications for the future generation of collaborative tools. In: Proceedings of the 2011 annual conference on Human factors in computing systems, CHI \u201911, pp 3151\u20133160. ACM, New York, NY, USA. doi: 10.1145\/1978942.1979409","DOI":"10.1145\/1978942.1979409"},{"key":"9493_CR15","doi-asserted-by":"publisher","unstructured":"Dajsuren Y, van den Brand MGJ, Serebrenik A, Roubtsov S (2013) Simulink models are also software: Modularity assessment. In: Proceedings of the 9th International ACM Sigsoft Conference on Quality of Software Architectures, QoSA \u201913, pp 99\u2013106. ACM, New York, NY, USA. doi: 10.1145\/2465478.2465482","DOI":"10.1145\/2465478.2465482"},{"key":"9493_CR16","unstructured":"Danescu-Niculescu-Mizil C, Sudhof M, Jurafsky D, Leskovec J, Potts C (2013) A computational approach to politeness with application to social factors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pp 250\u2013259. The Association for Computer Linguistics. http:\/\/aclweb.org\/anthology\/P\/P13\/P13-1025.pdf"},{"key":"9493_CR17","doi-asserted-by":"publisher","unstructured":"Datta S, Sindhgatta R, Sengupta B (2012) Talk versus work: characteristics of developer collaboration on the Jazz platform. In: Proceedings of the ACM international conference on Object oriented programming systems languages and applications, OOPSLA \u201912, pp 655\u2013668. ACM, New York, NY, USA. doi: 10.1145\/2384616.2384664","DOI":"10.1145\/2384616.2384664"},{"key":"9493_CR18","unstructured":"Davidov D, Tsur O, Rappoport A (2010) Semi-supervised recognition of sarcastic sentences in Twitter and Amazon. In: Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL \u201910, pp. 107\u2013116. Association for Computational Linguistics, Stroudsburg, PA, USA. http:\/\/dl.acm.org\/citation.cfm?id=1870568.1870582"},{"key":"9493_CR49","doi-asserted-by":"publisher","unstructured":"de Magalh\u00e3es CVC, da Silva FQB, Santos RES (2014) Investigations about replication of empirical studies in software engineering: Preliminary findings from a mapping study. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE \u201914, pp 37:1\u201337:10. ACM, New York, NY, USA. doi: 10.1145\/2601248.2601289","DOI":"10.1145\/2601248.2601289"},{"issue":"e73","key":"9493_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.7717\/peerj-cs.73","volume":"2","author":"G Destefanis","year":"2016","unstructured":"Destefanis G, Ortu M, Counsell S, Swift S, Marchesi M, Tonelli R (2016). Peer J Comput Sci 2(e73):1\u201335. doi: 10.7717\/peerj-cs.73","journal-title":"Peer J Comput Sci"},{"key":"9493_CR20","doi-asserted-by":"publisher","unstructured":"Dewan P (2015) Towards Emotion-Based Collaborative Software Engineering. In: 2015 IEEE\/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp 109\u2013112. doi: 10.1109\/CHASE.2015.32","DOI":"10.1109\/CHASE.2015.32"},{"issue":"293","key":"9493_CR21","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","volume":"56","author":"OJ Dunn","year":"1961","unstructured":"Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52\u201364","journal-title":"J Am Stat Assoc"},{"key":"9493_CR22","doi-asserted-by":"crossref","unstructured":"Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE \u201913, pp 422\u2013431. IEEE Press, Piscataway, NJ, USA. http:\/\/dl.acm.org\/citation.cfm?id=2486788.2486844","DOI":"10.1109\/ICSE.2013.6606588"},{"key":"9493_CR23","doi-asserted-by":"crossref","DOI":"10.1002\/0471445428","volume-title":"Statistical methods for rates and proportions","author":"JL Fleiss","year":"2003","unstructured":"Fleiss JL, Levin B, Paik MC (2003) Statistical methods for rates and proportions, 3rd edn. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ","edition":"3rd edn"},{"key":"9493_CR24","doi-asserted-by":"crossref","unstructured":"Fontana FA, Mariani E, Morniroli A, Sormani R, Tonello A (2011) An experience report on using code smells detection tools. In: ICST Workshops, pp 450\u2013457. IEEE","DOI":"10.1109\/ICSTW.2011.12"},{"key":"9493_CR25","doi-asserted-by":"publisher","unstructured":"Fuhr N, Muller P (1987) Probabilistic search term weighting - some negative results. In: Proceedings of the 10th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR \u201987, pp 13\u201318. ACM, New York, NY, USA. doi: 10.1145\/42005.42007","DOI":"10.1145\/42005.42007"},{"issue":"1","key":"9493_CR26","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1214\/aoms\/1177697819","volume":"40","author":"KR Gabriel","year":"1969","unstructured":"Gabriel KR (1969) Simultaneous test procedures\u2014some theory of multiple comparisons. Ann Math Stat 40(1):224\u2013250","journal-title":"Ann Math Stat"},{"key":"9493_CR27","doi-asserted-by":"publisher","unstructured":"Gamon M, Aue A, Corston-Oliver S, Ringger E (2005) Pulse: Mining customer opinions from free text. In: Proceedings of the 6th International Conference on Advances in Intelligent Data Analysis, IDA\u201905. Springer-Verlag, Berlin, Heidelberg, pp 121\u2013132. doi: 10.1007\/11552253_12","DOI":"10.1007\/11552253_12"},{"key":"9493_CR28","doi-asserted-by":"crossref","unstructured":"Garcia D, Zanetti MS, Schweitzer F (2013) The role of emotions in contributors activity: A case study on the Gentoo community. In: International Conference on Cloud and Green Computing, pp 410\u2013417","DOI":"10.1109\/CGC.2013.71"},{"issue":"2","key":"9493_CR29","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/1964897.1964899","volume":"12","author":"C Giraud-Carrier","year":"2011","unstructured":"Giraud-Carrier C, Dunham MH (2011) On the importance of sharing negative results. Sigkdd Explor. Newsl. 12(2):3\u20134. doi: 10.1145\/1964897.1964899","journal-title":"Sigkdd Explor. Newsl."},{"key":"9493_CR30","doi-asserted-by":"publisher","unstructured":"Goul M, Marjanovic O, Baxley S, Vizecky K (2012) Managing the Enterprise Business Intelligence App Store: Sentiment Analysis Supported Requirements Engineering. In: 2012 45th Hawaii International Conference on System Science (HICSS). doi: 10.1109\/HICSS.2012.421 , pp 4168\u20134177","DOI":"10.1109\/HICSS.2012.421"},{"key":"9493_CR31","unstructured":"Gousios G (2013) The GHTorrent dataset and tool suite. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR\u201913, pp 233\u2013236. http:\/\/dl.acm.org\/citation.cfm?id=2487085.2487132"},{"key":"9493_CR32","doi-asserted-by":"crossref","unstructured":"Greiler M, Herzig K, Czerwonka J (2015) Code ownership and software quality: A replication study. In: Proceedings of the 12th Working Conference on Mining Software Repositories, MSR \u201915, pp. 2\u201312. IEEE Press, Piscataway, NJ, USA. http:\/\/dl.acm.org.library.sutd.edu.sg:2048\/citation.cfm?id=2820518.2820522","DOI":"10.1109\/MSR.2015.8"},{"key":"9493_CR33","doi-asserted-by":"crossref","unstructured":"Guzman E, Az\u00f3car D, Li Y (2014) Sentiment analysis of commit comments in GitHub: An empirical study. In: MSR, pp 352\u2013355, ACM, New York, NY, USA","DOI":"10.1145\/2597073.2597118"},{"key":"9493_CR34","doi-asserted-by":"crossref","unstructured":"Guzman E, Bruegge B (2013) Towards emotional awareness in software development teams. In: Joint Meeting on Foundations of Software Engineering, pp 671\u2013674, ACM, New York, NY, USA","DOI":"10.1145\/2491411.2494578"},{"issue":"1","key":"9493_CR35","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/1656274.1656278","volume":"11","author":"M Hall","year":"2009","unstructured":"Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The Weka data mining software: An upyear. SIGKDD Explor Newsl 11(1):10\u201318","journal-title":"SIGKDD Explor Newsl"},{"key":"9493_CR36","doi-asserted-by":"crossref","unstructured":"Honkela T, Izzatdust Z, Lagus K (2012) Text mining for wellbeing: Selecting stories using semantic and pragmatic features. In: Artificial Neural Networks and Machine Learning, Part II, LNCS, vol 7553. Springer, pp 467\u2013474","DOI":"10.1007\/978-3-642-33266-1_58"},{"key":"9493_CR37","doi-asserted-by":"crossref","unstructured":"Howard MJ, Gupta S, Pollock LL, Vijay-Shanker K (2013) Automatically mining software-based, semantically-similar words from comment-code mappings. In: Zimmermann T, Penta MD, Kim S (eds) MSR, pp 377\u2013386. IEEE Computer Society","DOI":"10.1109\/MSR.2013.6624052"},{"issue":"1","key":"9493_CR38","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/BF01908075","volume":"2","author":"L Hubert","year":"1985","unstructured":"Hubert L, Arabie P (1985) Comparing partitions. J Classif 2(1):193\u2013218. doi: 10.1007\/BF01908075","journal-title":"J Classif"},{"key":"9493_CR39","doi-asserted-by":"publisher","unstructured":"Islam MR, Zibran MF (2016) Towards understanding and exploiting developers\u2019 emotional variations in software engineering. In: 2016 IEEE 14th International Conference on Software Engineering Research, Management and Applications (SERA), pp 185\u2013192. doi: 10.1109\/SERA.2016.7516145","DOI":"10.1109\/SERA.2016.7516145"},{"key":"9493_CR40","doi-asserted-by":"publisher","unstructured":"Jongeling R, Datta S, Serebrenik A (2015) Choosing your weapons: On sentiment analysis tools for software engineering research. In: ICSME, pp 531\u2013535. IEEE. doi: 10.1109\/ICSM.2015.7332508","DOI":"10.1109\/ICSM.2015.7332508"},{"key":"9493_CR41","doi-asserted-by":"crossref","first-page":"738","DOI":"10.1214\/12-EJS691","volume":"6","author":"F Konietschke","year":"2012","unstructured":"Konietschke F, Hothorn LA, Brunner E (2012) Rank-based multiple test procedures and simultaneous confidence intervals. Electronic Journal of Statistics 6:738\u2013759","journal-title":"Electronic Journal of Statistics"},{"key":"9493_CR42","doi-asserted-by":"publisher","unstructured":"Kucuktunc O, Cambazoglu BB, Weber I, Ferhatosmanoglu H (2012) A Large-scale Sentiment Analysis for Yahoo! Answers. In: Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM \u201912, pp 633\u2013642. ACM, New York, NY, USA. doi: 10.1145\/2124295.2124371","DOI":"10.1145\/2124295.2124371"},{"key":"9493_CR43","unstructured":"Lanza M, Di Penta M, Xie T (2012) (eds.): 9th IEEE Working Conference of Mining Software Repositories, MSR 2012, June 2-3, 2012, Zurich, Switzerland. IEEE Computer Society. http:\/\/ieeexplore.ieee.org\/xpl\/mostRecentIssue.jsp?punumber=6220358"},{"key":"9493_CR44","doi-asserted-by":"publisher","unstructured":"Leopairote W, Surarerks A, Prompoon N (2013) Evaluating software quality in use using user reviews mining. In: 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE), pp 257\u2013262. doi: 10.1109\/JCSSE.2013.6567355","DOI":"10.1109\/JCSSE.2013.6567355"},{"key":"9493_CR45","unstructured":"Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR \u201994, pp. 3\u201312. Springer-Verlag New York, Inc., New York, NY, USA. http:\/\/dl.acm.org.dianus.libr.tue.nl\/citation.cfm?id=188490.188495"},{"key":"9493_CR46","doi-asserted-by":"publisher","unstructured":"Li TH, Liu R, Sukaviriya N, Li Y, Yang J, Sandin M, Lee J (2014) Incident ticket analytics for it application management services. In: 2014 IEEE International Conference on Services Computing (SCC), pp 568\u2013574. doi: 10.1109\/SCC.2014.80","DOI":"10.1109\/SCC.2014.80"},{"key":"9493_CR47","doi-asserted-by":"publisher","unstructured":"Lindsey MR (2011) What went wrong?: Negative results from VoIP service providers. In: Proceedings of the 5th International Conference on Principles, Systems and Applications of IP Telecommunications, IPTcomm \u201911, pp 13:1\u201313:3. ACM, New York, NY, USA. doi: 10.1145\/2124436.2124453","DOI":"10.1145\/2124436.2124453"},{"key":"9493_CR48","doi-asserted-by":"publisher","unstructured":"Linstead E, Baldi P (2009) Mining the coherence of GNOME bug reports with statistical topic models. In: Godfrey MW, Whitehead J (eds) Proceedings of the 6th International Working Conference on Mining Software Repositories, MSR 2009 (Co-located with ICSE), Vancouver, BC, Canada, May 16\u201317, 2009, Proceedings, pp 99\u2013102. IEEE Computer Society. doi: 10.1109\/MSR.2009.5069486","DOI":"10.1109\/MSR.2009.5069486"},{"key":"9493_CR50","doi-asserted-by":"publisher","unstructured":"Martie L, Palepu VK, Sajnani H, Lopes CV Trendy bugs: Topic trends in the android bug reports. In: Lanza et al. [43], pp 120\u2013123. doi: 10.1109\/MSR.2012.6224268","DOI":"10.1109\/MSR.2012.6224268"},{"key":"9493_CR51","doi-asserted-by":"publisher","unstructured":"Mende T (2010) Replication of defect prediction studies: Problems, pitfalls and recommendations. In: Proceedings of the 6th International Conference on Predictive Models in Software Engineering, PROMISE \u201910, pp 5:1\u20135:10. ACM, New York, NY, USA. doi: 10.1145\/1868328.1868336","DOI":"10.1145\/1868328.1868336"},{"key":"9493_CR52","unstructured":"Mishne G, Glance NS (2006) Predicting movie sales from blogger sentiment. In: AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, pp 155\u2013158"},{"key":"9493_CR53","unstructured":"Mohammad SM, Kiritchenko S, Zhu X (2013) NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets. arXiv: 1308.6242 [cs]"},{"key":"9493_CR54","doi-asserted-by":"crossref","unstructured":"Murgia A, Tourani P, Adams B, Ortu M (2014) Do developers feel emotions? an exploratory analysis of emotions in software artifacts. In: MSR, pp 262-271, ACM, New York, NY, USA","DOI":"10.1145\/2597073.2597086"},{"key":"9493_CR55","doi-asserted-by":"publisher","unstructured":"Novielli N, Calefato F, Lanubile F (2015) The challenges of sentiment detection in the social programmer ecosystem. In: Proceedings of the 7th International Workshop on Social Software Engineering, SSE 2015, pp 33\u201340. ACM, New York, NY, USA. doi: 10.1145\/2804381.2804387","DOI":"10.1145\/2804381.2804387"},{"key":"9493_CR56","doi-asserted-by":"crossref","unstructured":"Ortu M, Adams B, Destefanis G, Tourani P, Marchesi M, Tonelli R (2015) Are bullies more productive? empirical study of affectiveness vs. issue fixing time. In: MSR","DOI":"10.1109\/MSR.2015.35"},{"key":"9493_CR57","doi-asserted-by":"publisher","unstructured":"Ortu M, Destefanis G, Adams B, Murgia A, Marchesi M, Tonelli R (2015) The JIRA repository dataset: Understanding social aspects of software development. In: Proceedings of the 11th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE \u201915, pp 1:1\u20131:4. ACM, New York, NY, USA. doi: 10.1145\/2810146.2810147","DOI":"10.1145\/2810146.2810147"},{"key":"9493_CR58","doi-asserted-by":"publisher","unstructured":"Ortu M, Destefanis G, Kassab M, Counsell S, Marchesi M, Tonelli R (2015) Would you mind fixing this issue? - an empirical analysis of politeness and attractiveness in software developed using agile boards. In: Lassenius C, Dings\u00f8yr T, Paasivaara M (eds) Agile Processes, in Software Engineering, and Extreme Programming - 16th International Conference, XP 2015, Helsinki, Finland, May 25\u201329, 2015, Proceedings, Lecture Notes in Business Information Processing, vol 212. Springer, pp 129\u2013140. doi: 10.1007\/978-3-319-18612-2_11","DOI":"10.1007\/978-3-319-18612-2_11"},{"key":"9493_CR59","unstructured":"Pak A, Paroubek P (2010) Twitter Based System: Using Twitter for Disambiguating Sentiment Ambiguous Adjectives. In: Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval \u201910, pp. 436\u2013439. Association for Computational Linguistics, Stroudsburg, PA, USA. http:\/\/dl.acm.org\/citation.cfm?id=1859664.1859761"},{"issue":"1-2","key":"9493_CR60","first-page":"1","volume":"2","author":"B Pang","year":"2007","unstructured":"Pang B, Lee L (2007) Opinion mining and sentiment analysis. Found Trends Inf Retr 2(1-2):1\u2013135","journal-title":"Found Trends Inf Retr"},{"key":"9493_CR61","doi-asserted-by":"publisher","unstructured":"Pang B, Lee L, Vaithyanathan S (2002) Thumbs Up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP \u201902, pp 79\u201386. Association for Computational Linguistics, Stroudsburg, PA, USA. doi: 10.3115\/1118693.1118704","DOI":"10.3115\/1118693.1118704"},{"key":"9493_CR62","doi-asserted-by":"crossref","unstructured":"Panichella S, Sorbo AD, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can I improve my app? classifying user reviews for software maintenance and evolution. In: ICSME. IEEE, pp 281\u2013290","DOI":"10.1109\/ICSM.2015.7332474"},{"key":"9493_CR63","doi-asserted-by":"publisher","unstructured":"Pletea D, Vasilescu B, Serebrenik A (2014) Security and emotion: Sentiment analysis of security discussions on GitHub. In: MSR. ACM, New York, NY, USA, pp 348\u2013351. doi: 10.1145\/2597073.2597117","DOI":"10.1145\/2597073.2597117"},{"issue":"1","key":"9493_CR64","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1145\/69605.357970","volume":"27","author":"P Pritchard","year":"1984","unstructured":"Pritchard P (1984) Some negative results concerning prime number generators. Commun ACM 27(1):53\u201357. doi: 10.1145\/69605.357970","journal-title":"Commun ACM"},{"issue":"336","key":"9493_CR65","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1080\/01621459.1971.10482356","volume":"66","author":"WM Rand","year":"1971","unstructured":"Rand WM (1971) Objective criteria for the evaluation of clustering methods. J Am Stat Assoc 66(336):846\u2013850","journal-title":"J Am Stat Assoc"},{"issue":"2","key":"9493_CR67","doi-asserted-by":"crossref","first-page":"6:1","DOI":"10.5329\/RESI.2014.1302006","volume":"13","author":"AI Rousinopoulos","year":"2014","unstructured":"Rousinopoulos AI, Robles G, Gonz\u00e1lez-Barahona JM (2014) Sentiment analysis of Free\/Open Source developers: preliminary findings from a case study. Revista Eletr\u00f4nica de Sistemas de Informa\u00e7\u00e3o 13(2):6:1\u20136:21","journal-title":"Revista Eletr\u00f4nica de Sistemas de Informa\u00e7\u00e3o"},{"key":"9493_CR68","doi-asserted-by":"crossref","unstructured":"Santos JM, Embrechts M (2009) On the use of the adjusted rand index as a metric for evaluating supervised classification. In: International Conference on Artificial Neural Networks, LNCS, vol 5769. Springer, pp 175\u2013184","DOI":"10.1007\/978-3-642-04277-5_18"},{"key":"9493_CR69","doi-asserted-by":"publisher","unstructured":"Schr\u00f6ter A, Aranda J, Damian D, Kwan I (2012) To talk or not to talk: factors that influence communication around changesets. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, CSCW \u201912, pp 1317\u20131326. ACM, New York, NY, USA. doi: 10.1145\/2145204.2145401","DOI":"10.1145\/2145204.2145401"},{"key":"9493_CR70","doi-asserted-by":"publisher","unstructured":"Sfetsos P, Adamidis P, Angelis L, Stamelos I, Deligiannis I (2012) Investigating the impact of personality and temperament traits on pair programming: a controlled experiment replication. In: 2012 Eighth International Conference on the Quality of Information and Communications Technology (QUATIC), pp 57\u201365. doi: 10.1109\/QUATIC.2012.36","DOI":"10.1109\/QUATIC.2012.36"},{"key":"9493_CR71","unstructured":"Sheskin DJ (2007) Handbook of parametric and nonparametric statistical procedures, 4 edn. Chapman & Hall"},{"key":"9493_CR72","doi-asserted-by":"crossref","unstructured":"Shihab E, Kamei Y, Bhattacharya P (2012) Mining challenge 2012: the Android platform. In: MSR, pp 112\u2013115","DOI":"10.1109\/MSR.2012.6224307"},{"issue":"2","key":"9493_CR73","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s10664-008-9060-1","volume":"13","author":"FJ Shull","year":"2008","unstructured":"Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211\u2013218. doi: 10.1007\/s10664-008-9060-1","journal-title":"Empir Softw Eng"},{"key":"9493_CR74","doi-asserted-by":"crossref","unstructured":"Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng A, Potts C (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Empirical Methods in Natural Language Processing, pp 1631\u20131642. Ass. for Comp. Linguistics","DOI":"10.18653\/v1\/D13-1170"},{"key":"9493_CR75","doi-asserted-by":"publisher","unstructured":"Sun X, Li B, Leung H, Li B, Li Y (2015) MSR4SM: Using topic models to effectively mining software repositories for software maintenance tasks. Inf Softw Technol 66:1\u201312. doi: 10.1016\/j.infsof.2015.05.003 . http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584915001007","DOI":"10.1016\/j.infsof.2015.05.003"},{"key":"9493_CR76","doi-asserted-by":"publisher","unstructured":"T\u00e4ht D (2014) The value of repeatable experiments and negative results: - a journey through the history and future of aqm and fair queuing algorithms. In: Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing Workshop, CSWS \u201914, pp 1\u20132. ACM, New York, NY, USA. doi: 10.1145\/2630088.2652480","DOI":"10.1145\/2630088.2652480"},{"issue":"1","key":"9493_CR77","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1002\/asi.21662","volume":"63","author":"M Thelwall","year":"2012","unstructured":"Thelwall M, Buckley K, Paltoglou G (2012) Sentiment strength detection for the social web. J Am Soc Inf Sci Technol 63(1):163\u2013173","journal-title":"J Am Soc Inf Sci Technol"},{"issue":"12","key":"9493_CR78","doi-asserted-by":"crossref","first-page":"2544","DOI":"10.1002\/asi.21416","volume":"61","author":"M Thelwall","year":"2010","unstructured":"Thelwall M, Buckley K, Paltoglou G, Cai D, Kappas A (2010) Sentiment in short strength detection informal text. J Am Soc Inf Sci Technol 61(12):2544\u20132558","journal-title":"J Am Soc Inf Sci Technol"},{"issue":"5","key":"9493_CR79","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1007\/s10664-007-9037-5","volume":"12","author":"P Tonella","year":"2007","unstructured":"Tonella P, Torchiano M, Du Bois B, Syst\u00e4 T (2007) Empirical studies in reverse engineering: State of the art and future trends. Empir Softw Eng 12(5):551\u2013571. doi: 10.1007\/s10664-007-9037-5","journal-title":"Empir Softw Eng"},{"key":"9493_CR80","unstructured":"Tourani P, Jiang Y, Adams B (2014) Monitoring sentiment in open source mailing lists: exploratory study on the apache ecosystem. In: Proceedings of 24th Annual International Conference on Computer Science and Software Engineering, CASCON \u201914, pp 34\u201344. IBM Corp., Riverton, NJ, USA. http:\/\/dl.acm.org\/citation.cfm?id=2735522.2735528"},{"key":"9493_CR81","unstructured":"Tukey JW (1951) Quick and dirty methods in statistics, part II, Simple analysis for standard designs. In: American Society for Quality Control, pp 189\u2013197"},{"key":"9493_CR82","doi-asserted-by":"crossref","unstructured":"Tumasjan A, Sprenger TO, Sandner PG, Welpe IM (2010) Predicting elections with twitter: What 140 characters reveal about political sentiment. In: International AAAI Conference on Weblogs and Social Media, pp 178\u2013185","DOI":"10.1609\/icwsm.v4i1.14009"},{"key":"9493_CR66","unstructured":"van Rijsbergen CJ (1979) Information Retrieval, 2nd edn. Butterworth-Heinemann, Newton, MA, USA"},{"key":"9493_CR83","doi-asserted-by":"publisher","unstructured":"Vasilescu B, Filkov V, Serebrenik A (2013) StackOverflow and github: associations between software development and crowdsourced knowledge. In: 2013 International Conference on Social Computing (SocialCom), pp 188\u2013195. doi: 10.1109\/SocialCom.2013.35","DOI":"10.1109\/SocialCom.2013.35"},{"key":"9493_CR84","doi-asserted-by":"publisher","unstructured":"Vasilescu B, Serebrenik A, van den Brand MGJ (2011) By no means: a study on aggregating software metrics. In: Concas G, Tempero ED, Zhang H, Penta MD (eds) Proceedings of the 2nd International Workshop on Emerging Trends in Software Metrics, WETSoM 2011, Waikiki, Honolulu, HI, USA, May 24, 2011. ACM, pp 23\u201326. doi: 10.1145\/1985374.1985381","DOI":"10.1145\/1985374.1985381"},{"issue":"4","key":"9493_CR85","doi-asserted-by":"publisher","first-page":"955","DOI":"10.1007\/s10664-013-9244-1","volume":"19","author":"B Vasilescu","year":"2013","unstructured":"Vasilescu B, Serebrenik A, Goeminne M, Mens T (2013) On the variation and specialisation of workload \u2013 a case study of the Gnome ecosystem community. Empir Softw Eng 19(4):955\u20131008. doi: 10.1007\/s10664-013-9244-1","journal-title":"Empir Softw Eng"},{"issue":"5","key":"9493_CR86","first-page":"360","volume":"37","author":"AJ Viera","year":"2005","unstructured":"Viera AJ, Garrett JM (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360\u2013363","journal-title":"Fam Med"},{"key":"9493_CR87","doi-asserted-by":"crossref","unstructured":"Vivian R, Tarmazdi H, Falkner K, Falkner N, Szabo C (2015) The development of a dashboard tool for visualising online teamwork discussions. In: Proceedings of the 37th International Conference on Software Engineering - Volume 2, ICSE \u201915, pp 380\u2013388. IEEE Press, Piscataway, NJ, USA. http:\/\/dl.acm.org\/citation.cfm?id=2819009.2819070","DOI":"10.1109\/ICSE.2015.170"},{"key":"9493_CR88","doi-asserted-by":"crossref","unstructured":"Wang S, Lo D, Vasilescu B, Serebrenik A (2014) EnTagRec: An enhanced tag recommendation system for software information sites. In: ICSME. IEEE, pp 291\u2013300","DOI":"10.1109\/ICSME.2014.51"},{"issue":"6","key":"9493_CR89","doi-asserted-by":"crossref","first-page":"80","DOI":"10.2307\/3001968","volume":"1","author":"F Wilcoxon","year":"1945","unstructured":"Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1 (6):80\u201383","journal-title":"Biom Bull"},{"key":"9493_CR90","doi-asserted-by":"crossref","unstructured":"Wilson T, Wiebe J, Hoffmann P (2005) Recognizing contextual polarity in phrase-level sentiment analysis. In: Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Stroudsburg, PA, USA, pp 347\u2013354","DOI":"10.3115\/1220575.1220619"},{"key":"9493_CR91","unstructured":"Yu HF, Ho CH, Juan YC, Lin CJ (2013) Libshorttext: A library for short-text classification and analysis. Tech. rep., Technical Report. http:\/\/www.csie.ntu.edu.tw\/~cjlin\/papers\/libshorttext.pdf"},{"key":"9493_CR92","doi-asserted-by":"publisher","unstructured":"Yu Y, Wang H, Yin G, Wang T (2016) Reviewer recommendation for pull-requests in github: What can we learn from code review and bug assignment?. Inf Softw Technol 74:204\u2013218. doi: 10.1016\/j.infsof.2016.01.004 . http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584916000069","DOI":"10.1016\/j.infsof.2016.01.004"},{"issue":"31","key":"9493_CR93","doi-asserted-by":"crossref","first-page":"835","DOI":"10.2466\/pms.1992.74.3.835","volume":"74","author":"DW Zimmerman","year":"1992","unstructured":"Zimmerman DW, Zumbo BD (1992) Parametric alternatives to the Student t test under violation of normality and homogeneity of variance. Percept Mot Skills 74(31):835\u2013844","journal-title":"Percept Mot Skills"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10664-016-9493-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-016-9493-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-016-9493-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,14]],"date-time":"2025-06-14T04:46:52Z","timestamp":1749876412000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10664-016-9493-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,1,10]]},"references-count":93,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2017,10]]}},"alternative-id":["9493"],"URL":"https:\/\/doi.org\/10.1007\/s10664-016-9493-x","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,1,10]]}}}