{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,20]],"date-time":"2026-06-20T23:43:04Z","timestamp":1781998984358,"version":"3.54.5"},"reference-count":78,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2020,8,19]],"date-time":"2020-08-19T00:00:00Z","timestamp":1597795200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,19]],"date-time":"2020-08-19T00:00:00Z","timestamp":1597795200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2020,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Software defect prediction models are classifiers often built by setting a threshold <jats:italic>t<\/jats:italic> on a defect proneness model, i.e., a scoring function. For instance, they classify a software module non-faulty if its defect proneness is below <jats:italic>t<\/jats:italic> and positive otherwise. Different values of <jats:italic>t<\/jats:italic> may lead to different defect prediction models, possibly with very different performance levels. Receiver Operating Characteristic (ROC) curves provide an overall assessment of a defect proneness model, by taking into account all possible values of <jats:italic>t<\/jats:italic> and thus all defect prediction models that can be built based on it. However, using a defect proneness model with a value of <jats:italic>t<\/jats:italic> is sensible only if the resulting defect prediction model has a performance that is at least as good as some minimal performance level that depends on practitioners\u2019 and researchers\u2019 goals and needs. We introduce a new approach and a new performance metric (the Ratio of Relevant Areas) for assessing a defect proneness model by taking into account only the parts of a ROC curve corresponding to values of <jats:italic>t<\/jats:italic> for which defect proneness models have higher performance than some reference value. We provide the practical motivations and theoretical underpinnings for our approach, by: 1) showing how it addresses the shortcomings of existing performance metrics like the Area Under the Curve and Gini\u2019s coefficient; 2) deriving reference values based on random defect prediction policies, in addition to deterministic ones; 3) showing how the approach works with several performance metrics (e.g., <jats:italic>Precision<\/jats:italic> and <jats:italic>Recall<\/jats:italic>) and their combinations; 4) studying misclassification costs and providing a general upper bound for the cost related to the use of any defect proneness model; 5) showing the relationships between misclassification costs and performance metrics. We also carried out a comprehensive empirical study on real-life data from the SEACRAFT repository, to show the differences between our metric and the existing ones and how more reliable and less misleading our metric can be.<\/jats:p>","DOI":"10.1007\/s10664-020-09861-4","type":"journal-article","created":{"date-parts":[[2020,8,19]],"date-time":"2020-08-19T13:03:19Z","timestamp":1597842199000},"page":"3977-4019","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["On the assessment of software defect prediction models via ROC curves"],"prefix":"10.1007","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4598-7024","authenticated-orcid":false,"given":"Sandro","family":"Morasca","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luigi","family":"Lavazza","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,8,19]]},"reference":[{"key":"9861_CR1","unstructured":"The SEACRAFT repository of empirical software engineering data, https:\/\/zenodo.org\/communities\/seacraft (2017)"},{"key":"9861_CR2","doi-asserted-by":"publisher","unstructured":"Alves TL, Ypma C, Visser J (2010) Deriving metric thresholds from benchmark data. In: 26Th IEEE international conference on software maintenance (ICSM 2010), september 12-18, 2010, timisoara, romania, pp 1\u201310. https:\/\/doi.org\/10.1109\/ICSM.2010.5609747","DOI":"10.1109\/ICSM.2010.5609747"},{"key":"9861_CR3","doi-asserted-by":"crossref","unstructured":"Arisholm E, Briand LC, Fuglerud M (2007) Data mining techniques for building fault-proneness models in telecom java software. In: The 18th IEEE international symposium on Software reliability, 2007. ISSRE\u201907. , IEEE, pp 215\u2013224","DOI":"10.1109\/ISSRE.2007.22"},{"issue":"1","key":"9861_CR4","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1016\/j.jss.2009.06.055","volume":"83","author":"E Arisholm","year":"2010","unstructured":"Arisholm E, Briand LC, Johannessen EB (2010) A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. J Syst Softw 83(1):2\u201317. https:\/\/doi.org\/10.1016\/j.jss.2009.06.055","journal-title":"J Syst Softw"},{"issue":"454","key":"9861_CR5","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1198\/016214501753168136","volume":"96","author":"SG Baker","year":"2001","unstructured":"Baker SG, Pinsky PF (2001) A proposed design and analysis for comparing digital and analog mammography: Special receiver operating characteristic methods for cancer screening. Journal of the American Statistical Association 96(454):421\u2013428. http:\/\/www.jstor.org\/stable\/2670280","journal-title":"Journal of the American Statistical Association"},{"key":"9861_CR6","unstructured":"Beecham S, Hall T, Bowes D, Gray D, Counsell S, Black S (2010) A systematic review of fault prediction approaches used in software engineering. Lero Technical Report Lero-TR-S20P1L0\u20132004"},{"key":"9861_CR7","unstructured":"Beecham S, Hall T, Bowes D, Gray D, Counsell S, Black S (2010) A systematic review of fault prediction approaches used in software engineering. Tech. rep., Technical Report lero-TR-2010-04 Lero"},{"issue":"2","key":"9861_CR8","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1007\/s11219-016-9353-3","volume":"26","author":"D Bowes","year":"2018","unstructured":"Bowes D, Hall T, Petri\u0107 J (2018) Software defect prediction: do different classifiers find the same defects? Softw Qual J 26(2):525\u2013552","journal-title":"Softw Qual J"},{"issue":"4","key":"9861_CR9","first-page":"193","volume":"9","author":"C Catal","year":"2012","unstructured":"Catal C (2012) Performance evaluation metrics for software fault prediction studies. Acta Polytechnica Hungarica 9(4):193\u2013206","journal-title":"Acta Polytechnica Hungarica"},{"issue":"4","key":"9861_CR10","doi-asserted-by":"crossref","first-page":"7346","DOI":"10.1016\/j.eswa.2008.10.027","volume":"36","author":"C Catal","year":"2009","unstructured":"Catal C, Diri B (2009) A systematic review of software fault prediction studies. Expert Syst Appl 36(4):7346\u20137354","journal-title":"Expert Syst Appl"},{"issue":"9","key":"9861_CR11","doi-asserted-by":"crossref","first-page":"934","DOI":"10.1016\/j.infsof.2010.03.011","volume":"52","author":"N Cerpa","year":"2010","unstructured":"Cerpa N, Bardeen M, Kitchenham B, Verner J (2010) Evaluating logistic regression models to estimate software project outcomes. Inf Softw Technol 52(9):934\u2013944","journal-title":"Inf Softw Technol"},{"key":"9861_CR12","unstructured":"Cohen J (1988) Statistical power analysis for the behavioral sciences Lawrence Earlbaum Associates. routledge, New York, NY USA"},{"issue":"1","key":"9861_CR13","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1007\/s10664-017-9520-6","volume":"23","author":"DA da Costa","year":"2018","unstructured":"da Costa DA, McIntosh S, Kulesza U, Hassan AE, Abebe SL (2018) An empirical study of the integration time of fixed issues. Empir Softw Eng 23(1):334\u2013383. https:\/\/doi.org\/10.1007\/s10664-017-9520-6","journal-title":"Empir Softw Eng"},{"key":"9861_CR14","unstructured":"Cremona L (2005) Elements of projective geometry: 3rd edn (dover phoenix editions) dover publications"},{"issue":"4","key":"9861_CR15","doi-asserted-by":"publisher","first-page":"775","DOI":"10.1007\/s10664-012-9239-3","volume":"19","author":"JA Dallal","year":"2014","unstructured":"Dallal JA, Morasca S (2014) Predicting object-oriented class reuse-proneness using internal quality attributes. Empir Softw Eng 19 (4):775\u2013821. https:\/\/doi.org\/10.1007\/s10664-012-9239-3","journal-title":"Empir Softw Eng"},{"issue":"3","key":"9861_CR16","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1111\/1541-0420.00071","volume":"59","author":"LE Dodd","year":"2003","unstructured":"Dodd LE, Pepe MS (2003) Partial auc estimation and regression. Biometrics 59(3):614\u2013623. http:\/\/www.jstor.org\/stable\/3695437","journal-title":"Biometrics"},{"issue":"1","key":"9861_CR17","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s10994-006-8199-5","volume":"65","author":"C Drummond","year":"2006","unstructured":"Drummond C, Holte RC (2006) Cost curves: an improved method for visualizing classifier performance. Mach Learn 65(1):95\u2013130","journal-title":"Mach Learn"},{"key":"9861_CR18","doi-asserted-by":"publisher","unstructured":"Erni K, Lewerentz C (1996) Applying design-metrics to object-oriented frameworks. In: 3Rd IEEE international software metrics symposium (METRICS 1996), from measurement to empirical results, march 25-26, 1996, Berlin, Germany, pp 64\u201374. https:\/\/doi.org\/10.1109\/METRIC.1996.492444","DOI":"10.1109\/METRIC.1996.492444"},{"issue":"8","key":"9861_CR19","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","volume":"27","author":"T Fawcett","year":"2006","unstructured":"Fawcett T (2006) An introduction to roc analysis. Patt Recogn Lett 27(8):861\u2013874. https:\/\/doi.org\/10.1016\/j.patrec.2005.10.010","journal-title":"Patt Recogn Lett"},{"key":"9861_CR20","doi-asserted-by":"crossref","unstructured":"Fenton NE, Bieman JM Software Metrics: A Rigorous and Practical Approach, Third Edition. Chapman & Hall\/CRC Innovations in Software Engineering and Software Development Series. Taylor & Francis (2014). https:\/\/books.google.es\/books?id=lx_OBQAAQBAJ","DOI":"10.1201\/b17461"},{"key":"9861_CR21","unstructured":"Flach PA (2003) The geometry of ROC space: Understanding machine learning metrics through ROC isometrics. In: Machine learning, proceedings of the twentieth international conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pp 194\u2013201. http:\/\/www.aaai.org\/Library\/ICML\/2003\/icml03-028.php"},{"key":"9861_CR22","unstructured":"Gini C (1912). In: Pizetti E, Salvemini T (eds) Variabilit\u00e0 e mutabilit\u00e0. Reprinted in Memorie di metodologica statistica. Libreria Eredi Virgilio Veschi, Rome"},{"issue":"6","key":"9861_CR23","doi-asserted-by":"crossref","first-page":"1276","DOI":"10.1109\/TSE.2011.103","volume":"38","author":"T Hall","year":"2012","unstructured":"Hall T, Beecham S, Bowes D, Gray D, Counsell S (2012) A systematic literature review on fault prediction performance in software engineering. IEEE Trans Software Eng 38(6):1276\u20131304","journal-title":"IEEE Trans Software Eng"},{"issue":"1","key":"9861_CR24","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/s10994-009-5119-5","volume":"77","author":"DJ Hand","year":"2009","unstructured":"Hand DJ (2009) Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach Learn 77(1):103\u2013123. https:\/\/doi.org\/10.1007\/s10994-009-5119-5","journal-title":"Mach Learn"},{"issue":"1","key":"9861_CR25","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","volume":"143","author":"JA Hanley","year":"1982","unstructured":"Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology 143(1):29\u201336","journal-title":"Radiology"},{"key":"9861_CR26","doi-asserted-by":"crossref","DOI":"10.1201\/9781420035285","volume-title":"Generalized estimating equations","author":"JW Hardin","year":"2002","unstructured":"Hardin JW, M HJ (2002) Generalized estimating equations. CRC Press, Abingdon"},{"key":"9861_CR27","doi-asserted-by":"publisher","unstructured":"Herbold S On the costs and profit of software defect prediction. Accepted for publication in IEEE Transactions on Software Engineering (01), 1\u20131. https:\/\/doi.org\/10.1109\/TSE.2019.2957794","DOI":"10.1109\/TSE.2019.2957794"},{"issue":"9","key":"9861_CR28","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1109\/TSE.2017.2724538","volume":"44","author":"S Herbold","year":"2018","unstructured":"Herbold S, Trautsch A, Grabowski J (2018) A comparative study to benchmark cross-project defect prediction approaches. IEEE Trans Software Eng 44 (9):811\u2013833. https:\/\/doi.org\/10.1109\/TSE.2017.2724538","journal-title":"IEEE Trans Software Eng"},{"key":"9861_CR29","doi-asserted-by":"crossref","unstructured":"Hosmer Jr, DW, Lemeshow S, Sturdivant RX (2013) Applied logistic regression. John Wiley & Sons","DOI":"10.1002\/9781118548387"},{"issue":"5","key":"9861_CR30","doi-asserted-by":"publisher","first-page":"2823","DOI":"10.1007\/s10664-018-9661-2","volume":"24","author":"Q Huang","year":"2019","unstructured":"Huang Q, Xia X, Lo D (2019) Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction. Empir Softw Eng 24 (5):2823\u20132862. https:\/\/doi.org\/10.1007\/s10664-018-9661-2","journal-title":"Empir Softw Eng"},{"key":"9861_CR31","doi-asserted-by":"publisher","unstructured":"Jiang Y, Cukic B (2009) Misclassification cost-sensitive fault prediction models. In: Proceedings of the 5th international workshop on predictive models in software engineering, PROMISE 2009, Vancouver, BC, Canada, May 18-19, 2009, pp 20. https:\/\/doi.org\/10.1145\/1540438.1540466","DOI":"10.1145\/1540438.1540466"},{"key":"9861_CR32","doi-asserted-by":"publisher","unstructured":"Jiang Y, Cukic B, Menzies T (2008) Cost curve evaluation of fault prediction models. In: 19Th international symposium on software reliability engineering (ISSRE 2008), 11-14 november 2008, seattle\/redmond, WA, USA, pp 197\u2013206. https:\/\/doi.org\/10.1109\/ISSRE.2008.54","DOI":"10.1109\/ISSRE.2008.54"},{"issue":"3","key":"9861_CR33","doi-asserted-by":"publisher","first-page":"745","DOI":"10.1148\/radiology.201.3.8939225","volume":"201","author":"Y Jiang","year":"1996","unstructured":"Jiang Y, Metz CE, Nishikawa RM (1996) A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 201 (3):745\u2013750. https:\/\/doi.org\/10.1148\/radiology.201.3.8939225. PMID: 8939225","journal-title":"Radiology"},{"key":"9861_CR34","unstructured":"Jones C (2008) A new business model for function point metrics. http:\/\/concepts.gilb.com\/dl185"},{"key":"9861_CR35","doi-asserted-by":"crossref","unstructured":"Jureczko M, Madeyski L (2010) Towards identifying software project clusters with regard to defect prediction. In: Proceedings of the 6th international conference on predictive models in software engineering, pp 1\u201310","DOI":"10.1145\/1868328.1868342"},{"issue":"1","key":"9861_CR36","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1007\/s10664-017-9518-0","volume":"23","author":"S Kabinna","year":"2018","unstructured":"Kabinna S, Bezemer C, Shang W, Syer MD, Hassan AE (2018) Examining the stability of logging statements. Empir Softw Eng 23(1):290\u2013333. https:\/\/doi.org\/10.1007\/s10664-017-9518-0","journal-title":"Empir Softw Eng"},{"issue":"3","key":"9861_CR37","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1023\/A:1009736205722","volume":"3","author":"TM Khoshgoftaar","year":"1998","unstructured":"Khoshgoftaar TM, Allen EB (1998) Classification of fault-prone software modules: Prior probabilities, costs, and model evaluation. Empir Softw Eng 3(3):275\u2013298. https:\/\/doi.org\/10.1023\/A:1009736205722","journal-title":"Empir Softw Eng"},{"issue":"1","key":"9861_CR38","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1023\/A:1016621219262","volume":"9","author":"TM Khoshgoftaar","year":"2001","unstructured":"Khoshgoftaar TM, Allen EB, Jones WD, Hudepohl JP (2001) Cost-benefit analysis of software quality models. Software Quality Journal 9(1):9\u201330. https:\/\/doi.org\/10.1023\/A:1016621219262","journal-title":"Software Quality Journal"},{"issue":"3","key":"9861_CR39","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1023\/B:EMSE.0000027781.18360.9b","volume":"9","author":"TM Khoshgoftaar","year":"2004","unstructured":"Khoshgoftaar TM, Seliya N (2004) Comparative assessment of software quality classification techniques: An empirical case study. Empirical Software Engineering 9(3):229\u2013257. http:\/\/www.springerlink.com\/index\/10.1023\/B:EMSE.0000027781.18360.9b","journal-title":"Empirical Software Engineering"},{"key":"9861_CR40","volume-title":"Tversky a.: Foundations of measurement, vol 1","author":"DH Krantz","year":"1971","unstructured":"Krantz DH, Luce RD, Suppes P (1971) Tversky a.: Foundations of measurement, vol 1. Academic Press, San Diego"},{"key":"9861_CR41","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.infsof.2016.01.003","volume":"73","author":"WB Langdon","year":"2016","unstructured":"Langdon WB, Dolado J, Sarro F, Harman M (2016) Exact mean absolute error of baseline predictor, MARP0. Inf Softw Technol 73:16\u201318","journal-title":"Inf Softw Technol"},{"key":"9861_CR42","doi-asserted-by":"crossref","unstructured":"Lavazza L, Morasca S (2017) On the evaluation of effort estimation models. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering, ACM, pp 41\u201350","DOI":"10.1145\/3084226.3084260"},{"issue":"1","key":"9861_CR43","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1016\/j.ejor.2015.05.030","volume":"247","author":"S Lessmann","year":"2015","unstructured":"Lessmann S, Baesens B, Seow HV, Thomas LC (2015) Benchmarking state-of-the-art classification algorithms for credit scoring: an update of research. Eur J Oper Res 247(1):124\u2013136","journal-title":"Eur J Oper Res"},{"key":"9861_CR44","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1016\/j.asoc.2014.11.023","volume":"27","author":"R Malhotra","year":"2015","unstructured":"Malhotra R (2015) A systematic review of machine learning techniques for software fault prediction. Appl Soft Comput 27:504\u2013518","journal-title":"Appl Soft Comput"},{"issue":"4","key":"9861_CR45","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/s13042-012-0095-7","volume":"4","author":"R Malhotra","year":"2013","unstructured":"Malhotra R, Khanna M (2013) Investigation of relationship between object-oriented metrics and change proneness. Int J Mach Learn Cybern 4(4):273\u2013286","journal-title":"Int J Mach Learn Cybern"},{"issue":"2","key":"9861_CR46","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","volume":"405","author":"BW Matthews","year":"1975","unstructured":"Matthews BW (1975) Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure 405(2):442\u2013451","journal-title":"Biochimica et Biophysica Acta (BBA)-Protein Structure"},{"key":"9861_CR47","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1177\/0272989X8900900307","volume":"9","author":"D McClish","year":"1989","unstructured":"McClish D (1989) Analyzing a portion of the ROC Curve. Medical decision making : an international journal of the Society for Medical Decision Making 9:190\u20135. https:\/\/doi.org\/10.1177\/0272989X8900900307","journal-title":"Medical decision making : an international journal of the Society for Medical Decision Making"},{"issue":"5","key":"9861_CR48","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1109\/TSE.2017.2693980","volume":"44","author":"S McIntosh","year":"2018","unstructured":"McIntosh S, Kamei Y (2018) Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction. IEEE Trans Software Eng 44(5):412\u2013428. https:\/\/doi.org\/10.1109\/TSE.2017.2693980","journal-title":"IEEE Trans Software Eng"},{"issue":"5","key":"9861_CR49","doi-asserted-by":"publisher","first-page":"1188","DOI":"10.1016\/j.jss.2012.01.017","volume":"85","author":"J Mendling","year":"2012","unstructured":"Mendling J, S\u00e1nchez-gonz\u00e1lez L, Garc\u00eda F, Rosa ML (2012) Thresholds for error probability measures of business process models. Journal of Systems and Software 85(5):1188\u20131197. https:\/\/doi.org\/10.1016\/j.jss.2012.01.017","journal-title":"Journal of Systems and Software"},{"key":"9861_CR50","doi-asserted-by":"publisher","unstructured":"Morasca S (2009) A probability-based approach for measuring external attributes of software artifacts. In: Proceedings of the 2009 3rd international symposium on empirical software engineering and measurement, ESEM \u201909, Lake Buena Vista, FL, USA, October 15-16, 2009. https:\/\/doi.org\/10.1109\/ESEM.2009.5316048. IEEE Computer Society, Washington, pp 44\u201355","DOI":"10.1109\/ESEM.2009.5316048"},{"key":"9861_CR51","doi-asserted-by":"publisher","unstructured":"Morasca S (2014) Using logistic regression to estimate the number of faulty software modules. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, EASE \u201914, May 13-14, 2014, ACM, New York, pp 26:1\u201326:9. https:\/\/doi.org\/10.1145\/2601248.2601297","DOI":"10.1145\/2601248.2601297"},{"key":"9861_CR52","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.infsof.2017.03.005","volume":"89","author":"S Morasca","year":"2017","unstructured":"Morasca S, Lavazza L (2017) Risk-averse slope-based thresholds: Definition and empirical evaluation. Inform Software Technol 89:37\u201363. https:\/\/doi.org\/10.1016\/j.infsof.2017.03.005","journal-title":"Inform Software Technol"},{"issue":"1","key":"9861_CR53","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1007\/s10664-017-9526-0","volume":"23","author":"A Murgia","year":"2018","unstructured":"Murgia A, Ortu M, Tourani P, Adams B, Demeyer S (2018) An exploratory qualitative and quantitative analysis of emotions in issue report comments of open source systems. Empir Softw Eng 23(1):521\u2013564. https:\/\/doi.org\/10.1007\/s10664-017-9526-0","journal-title":"Empir Softw Eng"},{"issue":"9","key":"9861_CR54","doi-asserted-by":"publisher","first-page":"874","DOI":"10.1109\/TSE.2017.2720603","volume":"44","author":"J Nam","year":"2018","unstructured":"Nam J, Fu W, Kim S, Menzies T, Tan L (2018) Heterogeneous defect prediction. IEEE Trans Software Eng 44(9):874\u2013896. https:\/\/doi.org\/10.1109\/TSE.2017.2720603","journal-title":"IEEE Trans Software Eng"},{"issue":"1","key":"9861_CR55","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1109\/TSE.2017.2659747","volume":"44","author":"DD Nucci","year":"2018","unstructured":"Nucci DD, Palomba F, Rosa GD, Bavota G, Oliveto R, Lucia AD (2018) A developer centered bug prediction model. IEEE Trans Software Eng 44(1):5\u201324. https:\/\/doi.org\/10.1109\/TSE.2017.2659747","journal-title":"IEEE Trans Software Eng"},{"key":"9861_CR56","doi-asserted-by":"crossref","unstructured":"Posnett D, Filkov V, Devanbu P (2011) Ecological inference in empirical software engineering. In: Proceedings of the 2011 26th IEEE\/ACM international conference on automated software engineering, IEEE Computer Society, pp 362\u2013371","DOI":"10.1109\/ASE.2011.6100074"},{"key":"9861_CR57","unstructured":"Powers DM (2011) Evaluation: from precision, recall and f-measure to roc, informedness markedness and correlation"},{"key":"9861_CR58","doi-asserted-by":"publisher","unstructured":"Powers DMW (2012) The problem of area under the curve. In: 2012 IEEE International conference on information science and technology, pp 567\u2013573. https:\/\/doi.org\/10.1109\/ICIST.2012.6221710","DOI":"10.1109\/ICIST.2012.6221710"},{"issue":"3","key":"9861_CR59","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1023\/A:1007601015854","volume":"42","author":"FJ Provost","year":"2001","unstructured":"Provost FJ, Fawcett T (2001) Robust classification for imprecise environments. Mach Learn 42(3):203\u2013231. https:\/\/doi.org\/10.1023\/A:1007601015854","journal-title":"Mach Learn"},{"issue":"2","key":"9861_CR60","first-page":"296","volume":"1","author":"S Pundir","year":"2012","unstructured":"Pundir S, Seshadri R (2012) A novel concept of partial lorenz curve and partial gini index. Int J Eng Sci Innov Technol 1(2):296\u2013301","journal-title":"Int J Eng Sci Innov Technol"},{"issue":"8","key":"9861_CR61","doi-asserted-by":"crossref","first-page":"1397","DOI":"10.1016\/j.infsof.2013.02.009","volume":"55","author":"D Radjenovi\u0107","year":"2013","unstructured":"Radjenovi\u0107 D, Heri\u010dko M, Torkar R, \u017eivkovi\u010d A (2013) Software fault prediction metrics: A systematic literature review. Inform Softw Technol 55(8):1397\u20131418","journal-title":"Inform Softw Technol"},{"issue":"4","key":"9861_CR62","doi-asserted-by":"publisher","first-page":"2464","DOI":"10.1007\/s10664-017-9564-7","volume":"23","author":"C Ragkhitwetsagul","year":"2018","unstructured":"Ragkhitwetsagul C, Krinke J, Clark D (2018) A comparison of code similarity analysers. Empir Softw Eng 23(4):2464\u20132519. https:\/\/doi.org\/10.1007\/s10664-017-9564-7","journal-title":"Empir Softw Eng"},{"key":"9861_CR63","unstructured":"van Rijsbergen CJ (1979) Information retrieval butterworth"},{"key":"9861_CR64","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1186\/1471-2105-12-77","volume":"12","author":"X Robin","year":"2011","unstructured":"Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC (2011) M\u00fcller, m.: proc: an open-source package for r and s+ to analyze and compare roc curves. BMC Bioinformatics 12:77. https:\/\/doi.org\/10.1186\/1471-2105-12-77","journal-title":"BMC Bioinformatics"},{"key":"9861_CR65","doi-asserted-by":"publisher","unstructured":"S\u00e1nchez-Gonz\u00e1lez L, Garc\u00eda F, Ruiz F, Mendling J (2012) A study of the effectiveness of two threshold definition techniques. In: 16Th international conference on evaluation & assessment in software engineering, EASE 2012, ciudad real, spain, may 14-15, 2012. proceedings, pp 197\u2013205. https:\/\/doi.org\/10.1049\/ic.2012.0026","DOI":"10.1049\/ic.2012.0026"},{"key":"9861_CR66","unstructured":"Schneidewind NF (2001) Investigation of logistic regression as a discriminant of software quality. In: 7Th IEEE international software metrics symposium (METRICS 2001), 4-6 april 2001, london, england, pp 328\u2013337"},{"issue":"2","key":"9861_CR67","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1109\/TSE.2010.9","volume":"36","author":"R Shatnawi","year":"2010","unstructured":"Shatnawi R (2010) A quantitative investigation of the acceptable risk levels of object-oriented metrics in open-source systems. IEEE Trans Software Eng 36(2):216\u2013225. https:\/\/doi.org\/10.1109\/TSE.2010.9","journal-title":"IEEE Trans Software Eng"},{"issue":"1","key":"9861_CR68","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/smr.404","volume":"22","author":"R Shatnawi","year":"2010","unstructured":"Shatnawi R, Li W, Swain J, Newman T (2010) Finding software metrics threshold values using ROC curves. Journal of Software Maintenance 22 (1):1\u201316. https:\/\/doi.org\/10.1002\/smr.404","journal-title":"Journal of Software Maintenance"},{"issue":"6","key":"9861_CR69","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1109\/TSE.2014.2322358","volume":"40","author":"M Shepperd","year":"2014","unstructured":"Shepperd M, Bowes D, Hall T (2014) Researcher bias: the use of machine learning in software defect prediction. IEEE Trans Softw Eng 40(6):603\u2013616","journal-title":"IEEE Trans Softw Eng"},{"issue":"8","key":"9861_CR70","doi-asserted-by":"crossref","first-page":"820","DOI":"10.1016\/j.infsof.2011.12.008","volume":"54","author":"M Shepperd","year":"2012","unstructured":"Shepperd M, MacDonell S (2012) Evaluating prediction systems in software project estimation. Inf Softw Technol 54(8):820\u2013827","journal-title":"Inf Softw Technol"},{"issue":"1","key":"9861_CR71","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s11219-009-9079-6","volume":"18","author":"Y Singh","year":"2010","unstructured":"Singh Y, Kaur A, Malhotra R (2010) Empirical validation of object-oriented metrics for predicting fault proneness models. Software Quality Journal 18(1):3","journal-title":"Software Quality Journal"},{"key":"9861_CR72","unstructured":"Swets J, Pickett R (1982) Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Academic Press series in cognition and perception. Academic Press. https:\/\/books.google.it\/books?id=MIx9AAAAMAAJ"},{"key":"9861_CR73","doi-asserted-by":"publisher","unstructured":"Tosun A, Bener AB (2009) Reducing false alarms in software defect prediction by decision threshold optimization. In: Proceedings of the Third international symposium on empirical software engineering and measurement, ESEM 2009, October 15-16, 2009, Lake Buena Vista, Florida, USA, pp 477\u2013480. https:\/\/doi.org\/10.1145\/1671248.1671311","DOI":"10.1145\/1671248.1671311"},{"key":"9861_CR74","unstructured":"Total Metrics (2007) Methods for software sizing \u2013 how to decide which method to use. www.totalmetrics.com\/function-point-resources\/downloads\/R185_Why-use-Function-Points.pdf"},{"key":"9861_CR75","unstructured":"Vilalta R, Oblinger D (2000) A quantification of distance bias between evaluation metrics in classification. In: Proceedings of the seventeenth international conference on machine learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000, pp 1087\u20131094"},{"issue":"1","key":"9861_CR76","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3","volume":"3","author":"WJ Youden","year":"1950","unstructured":"Youden WJ (1950) Index for rating diagnostic tests. Cancer 3 (1):32\u201335","journal-title":"Cancer"},{"issue":"6","key":"9861_CR77","doi-asserted-by":"crossref","first-page":"3186","DOI":"10.1007\/s10664-017-9516-2","volume":"22","author":"F Zhang","year":"2017","unstructured":"Zhang F, Keivanloo I, Zou Y (2017) Data transformation in cross-project defect prediction. Empir Softw Eng 22(6):3186\u20133218","journal-title":"Empir Softw Eng"},{"key":"9861_CR78","doi-asserted-by":"publisher","unstructured":"Zhang H, Cheung SC (2013) A cost-effectiveness criterion for applying software defect prediction models. In: Joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, ESEC\/FSE\u201913, saint petersburg, russian federation, august 18-26, 2013, pp 643\u2013646. https:\/\/doi.org\/10.1145\/2491411.2494581","DOI":"10.1145\/2491411.2494581"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09861-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-020-09861-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09861-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,8,18]],"date-time":"2021-08-18T23:21:51Z","timestamp":1629328911000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-020-09861-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,19]]},"references-count":78,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,9]]}},"alternative-id":["9861"],"URL":"https:\/\/doi.org\/10.1007\/s10664-020-09861-4","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,19]]},"assertion":[{"value":"19 August 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}