{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T07:48:18Z","timestamp":1777880898489,"version":"3.51.4"},"reference-count":70,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,2,17]],"date-time":"2023-02-17T00:00:00Z","timestamp":1676592000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,2,17]],"date-time":"2023-02-17T00:00:00Z","timestamp":1676592000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BioData Mining"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Binary classification is a common task for which machine learning and computational statistics are used, and the area under the receiver operating characteristic curve\u00a0(ROC\u00a0AUC) has become the common standard metric to evaluate binary classifications in most scientific fields. The ROC curve has <jats:italic>true positive rate<\/jats:italic>\u00a0(also called <jats:italic>sensitivity<\/jats:italic> or <jats:italic>recall<\/jats:italic>) on the <jats:italic>y<\/jats:italic> axis and false positive rate on the <jats:italic>x<\/jats:italic> axis, and the ROC\u00a0AUC can range from 0 (worst result) to 1\u00a0(perfect result). The ROC\u00a0AUC, however, has several flaws and drawbacks. This score is generated including predictions that obtained insufficient sensitivity and specificity, and moreover it does not say anything about <jats:italic>positive predictive value<\/jats:italic>\u00a0(also known as <jats:italic>precision<\/jats:italic>) nor negative predictive value\u00a0(NPV) obtained by the classifier, therefore potentially generating inflated overoptimistic results. Since it is common to include ROC\u00a0AUC alone without precision and negative predictive value, a researcher might erroneously conclude that their classification was successful. Furthermore, a given point in the ROC space does not identify a single confusion matrix nor a group of matrices sharing the same MCC value. Indeed, a given <jats:italic>(sensitivity, specificity)<\/jats:italic> pair can cover a broad MCC range, which casts doubts on the reliability of ROC\u00a0AUC as a performance measure. In contrast, the Matthews correlation coefficient\u00a0(MCC) generates a high score in its <jats:inline-formula><jats:alternatives><jats:tex-math>$$[-1; +1]$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mo>[<\/mml:mo>\n                    <mml:mo>-<\/mml:mo>\n                    <mml:mn>1<\/mml:mn>\n                    <mml:mo>\u037e<\/mml:mo>\n                    <mml:mo>+<\/mml:mo>\n                    <mml:mn>1<\/mml:mn>\n                    <mml:mo>]<\/mml:mo>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> interval only if the classifier scored a high value for all the four <jats:italic>basic rates<\/jats:italic> of the confusion matrix: sensitivity, specificity, precision, and negative predictive value. A high MCC (for example, MCC <jats:inline-formula><jats:alternatives><jats:tex-math>$$=$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mo>=<\/mml:mo>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> 0.9), moreover, always corresponds to a high ROC\u00a0AUC, and not vice versa. In this short study, we explain why the Matthews correlation coefficient should replace the ROC\u00a0AUC as standard statistic in all the scientific studies involving a binary classification, in all scientific fields.<\/jats:p>","DOI":"10.1186\/s13040-023-00322-4","type":"journal-article","created":{"date-parts":[[2023,2,17]],"date-time":"2023-02-17T13:06:12Z","timestamp":1676639172000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":458,"title":["The Matthews correlation coefficient\u00a0(MCC) should replace the ROC\u00a0AUC as the standard metric for assessing binary classification"],"prefix":"10.1186","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9655-7142","authenticated-orcid":false,"given":"Davide","family":"Chicco","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2705-5728","authenticated-orcid":false,"given":"Giuseppe","family":"Jurman","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,2,17]]},"reference":[{"issue":"1","key":"322_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-020-69813-2","volume":"10","author":"M Hassan","year":"2020","unstructured":"Hassan M, Ali S, Alquhayz H, Safdar K. Developing intelligent medical image modality classification system using deep transfer learning and LDA. Sci Rep. 2020;10(1):1\u201314.","journal-title":"Sci Rep."},{"key":"322_CR2","doi-asserted-by":"publisher","first-page":"103596","DOI":"10.1016\/j.bspc.2022.103596","volume":"75","author":"N Kumar","year":"2022","unstructured":"Kumar N, Sharma M, Singh VP, Madan C, Mehandia S. An empirical study of handcrafted and dense feature extraction techniques for lung and colon cancer classification from histopathological images. Biomed Signal Process Control. 2022;75:103596.","journal-title":"Biomed Signal Process Control."},{"issue":"12","key":"322_CR3","doi-asserted-by":"publisher","first-page":"5763","DOI":"10.1007\/s12652-021-03256-z","volume":"13","author":"M Sharma","year":"2022","unstructured":"Sharma M, Kumar N. Improved hepatocellular carcinoma fatality prognosis using ensemble learning approach. J Ambient Intell Humanized Comput. 2022;13(12):5763\u201377.","journal-title":"J Ambient Intell Humanized Comput."},{"issue":"1","key":"322_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13040-021-00244-z","volume":"14","author":"D Chicco","year":"2021","unstructured":"Chicco D, T\u00f6tsch N, Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Min. 2021;14(1):1\u201322.","journal-title":"BioData Min."},{"key":"322_CR5","unstructured":"Chen TY, Kuo FC, Merkel R. On the statistical properties of the F-measure. In: Proceedings of QSIC 2004 \u2013 the 4th International Conference on Quality Software.\u00a0New York City: IEEE; 2004. p. 146\u2013153."},{"issue":"1","key":"322_CR6","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1186\/s12864-019-6413-7","volume":"21","author":"D Chicco","year":"2020","unstructured":"Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics. 2020;21(1):6.","journal-title":"BMC Genomics."},{"issue":"1","key":"322_CR7","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/001316446502500101","volume":"25","author":"JP Guilford","year":"1965","unstructured":"Guilford JP. The minimal phi coefficient and the maximal phi. Educ Psychol Meas. 1965;25(1):3\u20138.","journal-title":"Educ Psychol Meas."},{"key":"322_CR8","doi-asserted-by":"crossref","unstructured":"Matthews BW. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta (BBA) Protein Struct. 1975;405(2):442\u2013451.","DOI":"10.1016\/0005-2795(75)90109-9"},{"key":"322_CR9","doi-asserted-by":"crossref","unstructured":"Yao J, Shepperd M. Assessing software defection prediction performance: why using the Matthews correlation coefficient matters. In: Proceedings of EASE 2020 \u2013 the 24th Evaluation and Assessment in Software Engineering.\u00a0New York City:\u00a0Association for Computing Machinery;\u00a02020. p. 120\u2013129.","DOI":"10.1145\/3383219.3383232"},{"issue":"1","key":"322_CR10","first-page":"163","volume":"8","author":"Y Liu","year":"2015","unstructured":"Liu Y, Cheng J, Yan C, Wu X, Chen F. Research on the Matthews correlation coefficients metrics of personalized recommendation algorithm evaluation. Int J Hybrid Inf Technol. 2015;8(1):163\u201372.","journal-title":"Int J Hybrid Inf Technol."},{"key":"322_CR11","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1016\/j.patrec.2020.03.030","volume":"136","author":"Q Zhu","year":"2020","unstructured":"Zhu Q. On the performance of Matthews correlation coefficient (MCC) for imbalanced dataset. Pattern Recognit Lett. 2020;136:71\u201380.","journal-title":"Pattern Recognit Lett."},{"issue":"1","key":"322_CR12","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1007\/s10115-018-1185-y","volume":"58","author":"SM Saqlain","year":"2019","unstructured":"Saqlain SM, Sher M, Shah FA, Khan I, Ashraf MU, Awais M, et al. Fisher score and Matthews correlation coefficient-based feature subset selection for heart disease diagnosis using support vector machines. Knowl Inf Syst. 2019;58(1):139\u201367.","journal-title":"Knowl Inf Syst."},{"key":"322_CR13","doi-asserted-by":"crossref","unstructured":"D\u2019Amato V, Oneto L, Camurri A, Anguita D. Keep it simple: handcrafting Feature and tuning Random Forests and XGBoost to face the affective Movement Recognition Challenge 2021. In: Proceedings of ACIIW 2021 \u2013 the 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos.\u00a0Nara: IEEE; 2021. p. 1\u20137.","DOI":"10.1109\/ACIIW52867.2021.9666428"},{"key":"322_CR14","unstructured":"Maier-Hein L, Reinke A, Christodoulou E, Glocker B, Godau P, Isensee F, et\u00a0al. Metrics reloaded: pitfalls and recommendations for image analysis validation. 2022. arXiv preprint arXiv:2206.01653."},{"issue":"8","key":"322_CR15","doi-asserted-by":"publisher","first-page":"e41882","DOI":"10.1371\/journal.pone.0041882","volume":"7","author":"G Jurman","year":"2012","unstructured":"Jurman G, Riccadonna S, Furlanello C. A comparison of MCC and CEN error measures in multi-class prediction. PLoS ONE. 2012;7(8):e41882.","journal-title":"PLoS ONE."},{"key":"322_CR16","doi-asserted-by":"publisher","first-page":"47112","DOI":"10.1109\/ACCESS.2021.3068614","volume":"9","author":"D Chicco","year":"2021","unstructured":"Chicco D, Starovoitov V, Jurman G. The Benefits of the Matthews correlation coefficient (MCC) Over the Diagnostic Odds Ratio (DOR) in Binary Classification Assessment. IEEE Access. 2021;9:47112\u201324.","journal-title":"IEEE Access."},{"key":"322_CR17","doi-asserted-by":"publisher","first-page":"78368","DOI":"10.1109\/ACCESS.2021.3084050","volume":"9","author":"D Chicco","year":"2021","unstructured":"Chicco D, Warrens MJ, Jurman G. The Matthews correlation coefficient (MCC) is more informative than Cohen\u2019s Kappa and Brier score in binary classification assessment. IEEE Access. 2021;9:78368\u201381.","journal-title":"IEEE Access."},{"key":"322_CR18","doi-asserted-by":"crossref","unstructured":"Redondo AR, Navarro J, Fern\u00e1ndez RR, de\u00a0Diego IM, Moguerza JM, Fern\u00e1ndez-Mu\u00f1oz JJ. Unified performance measure for binary classification problems. In: Proceedings of IDEAL 2020 \u2013 the 21st International Conference on Intelligent Data Engineering and Automated Learning. vol. 12490 of Lecture Notes in Computer Science. Berlin: Springer International Publishing; 2020. p. 104\u2013112.","DOI":"10.1007\/978-3-030-62365-4_10"},{"issue":"10","key":"322_CR19","doi-asserted-by":"publisher","first-page":"12049","DOI":"10.1007\/s10489-021-03041-7","volume":"52","author":"IMD Diego","year":"2022","unstructured":"Diego IMD, Redondo AR, Fern\u00e1ndez RR, Navarro J, Moguerza JM. General performance score for classification problems. Appl Intell. 2022;52(10):12049\u201363.","journal-title":"Appl Intell."},{"issue":"1","key":"322_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-020-61588-w","volume":"10","author":"YH Lai","year":"2020","unstructured":"Lai YH, Chen WN, Hsu TC, Lin C, Tsao Y, Wu S. Overall survival prediction of non-small cell lung cancer by integrating microarray and clinical data with deep learning. Sci Rep. 2020;10(1):1\u201311.","journal-title":"Sci Rep."},{"issue":"19","key":"322_CR21","doi-asserted-by":"publisher","first-page":"34","DOI":"10.12746\/swrccc.v5i19.391","volume":"5","author":"S Yang","year":"2017","unstructured":"Yang S, Berdine G. The receiver operating characteristic (ROC) curve. Southwest Respir Crit Care Chronicles. 2017;5(19):34\u20136.","journal-title":"Southwest Respir Crit Care Chronicles."},{"issue":"7","key":"322_CR22","doi-asserted-by":"publisher","first-page":"1145","DOI":"10.1016\/S0031-3203(96)00142-2","volume":"30","author":"AP Bradley","year":"1997","unstructured":"Bradley AP. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit. 1997;30(7):1145\u201359.","journal-title":"Pattern Recognit."},{"key":"322_CR23","volume-title":"Using the Receiver Operating Characteristic (ROC) curve to analyze a classification model","author":"C Gamez","year":"2009","unstructured":"Gamez C. Using the Receiver Operating Characteristic (ROC) curve to analyze a classification model. Salt Lake City: Department of Mathematics, University of Utah; 2009."},{"issue":"8","key":"322_CR24","doi-asserted-by":"publisher","first-page":"416","DOI":"10.1056\/NEJM197102252840805","volume":"284","author":"LB Lusted","year":"1971","unstructured":"Lusted LB. Decision-making studies in patient management. N Engl J Med. 1971;284(8):416\u201324.","journal-title":"N Engl J Med."},{"key":"322_CR25","doi-asserted-by":"crossref","unstructured":"Metz CE. Basic principles of ROC analysis. In: Seminars in Nuclear Medicine. vol.\u00a08.\u00a0Amsterdam: Elsevier; 1978. p. 283\u2013298.","DOI":"10.1016\/S0001-2998(78)80014-2"},{"issue":"4","key":"322_CR26","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1093\/clinchem\/39.4.561","volume":"39","author":"MH Zweig","year":"1993","unstructured":"Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39(4):561\u201377.","journal-title":"Clin Chem."},{"issue":"7","key":"322_CR27","doi-asserted-by":"publisher","first-page":"511","DOI":"10.1093\/jnci\/95.7.511","volume":"95","author":"SG Baker","year":"2003","unstructured":"Baker SG. The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer. J Natl Cancer Inst. 2003;95(7):511\u20135.","journal-title":"J Natl Cancer Inst."},{"issue":"1","key":"322_CR28","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1159\/000074404","volume":"101","author":"S Parodi","year":"2003","unstructured":"Parodi S, Muselli M, Fontana V, Bonassi S. ROC curves are a suitable and flexible tool for the analysis of gene expression profiles. Cytogenet Genome Res. 2003;101(1):90\u20131.","journal-title":"Cytogenet Genome Res."},{"issue":"6","key":"322_CR29","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1136\/emermed-2017-206735","volume":"34","author":"ZH Hoo","year":"2017","unstructured":"Hoo ZH, Candlish J, Teare D. What is an ROC curve? Emerg Med J. 2017;34(6):357\u20139.","journal-title":"Emerg Med J."},{"issue":"1","key":"322_CR30","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","volume":"143","author":"JA Hanley","year":"1982","unstructured":"Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29\u201336.","journal-title":"Radiology."},{"issue":"1","key":"322_CR31","first-page":"1","volume":"12","author":"L Gon\u00e7alves","year":"2014","unstructured":"Gon\u00e7alves L, Subtil A, Oliveira MR, de Zea Bermudez P. ROC curve estimation: an overview. REVSTAT-Stat J. 2014;12(1):1\u201320.","journal-title":"REVSTAT-Stat J."},{"key":"322_CR32","unstructured":"Google. Google Scholar. 2022. http:\/\/scholar.google.com. Accessed 5 July 2022."},{"issue":"2","key":"322_CR33","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1111\/j.1466-8238.2007.00358.x","volume":"17","author":"JM Lobo","year":"2008","unstructured":"Lobo JM, Jim\u00e9nez-Valverde A, Real R. AUC: a misleading measure of the performance of predictive distribution models. Glob Ecol Biogeogr. 2008;17(2):145\u201351.","journal-title":"Glob Ecol Biogeogr."},{"issue":"4","key":"322_CR34","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1007\/s00330-014-3487-0","volume":"25","author":"S Halligan","year":"2015","unstructured":"Halligan S, Altman DG, Mallett S. Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: a discussion and proposal for an alternative approach. Eur Radiol. 2015;25(4):932\u20139.","journal-title":"Eur Radiol."},{"key":"322_CR35","doi-asserted-by":"crossref","unstructured":"Klawonn F, H\u00f6ppner F, May S. An alternative to ROC and AUC analysis of classifiers. In: Proceedins of IDA 2011 \u2013 the 10th International Symposium on Intelligent Data Analysis.\u00a0Porto: Springer; 2011. p. 210\u2013221.","DOI":"10.1007\/978-3-642-24800-9_21"},{"issue":"1","key":"322_CR36","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1093\/bib\/bbr008","volume":"13","author":"D Berrar","year":"2012","unstructured":"Berrar D, Flach P. Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them). Brief Bioinforma. 2012;13(1):83\u201397.","journal-title":"Brief Bioinforma."},{"key":"322_CR37","doi-asserted-by":"crossref","unstructured":"Powers DM. The problem of area under the curve. In: Proceedings of ICIST 2012 - the 2nd IEEE International Conference on Information Science and Technology.\u00a0London: IEEE; 2012. p. 567\u2013573.","DOI":"10.1109\/ICIST.2012.6221710"},{"issue":"7","key":"322_CR38","doi-asserted-by":"publisher","first-page":"928","DOI":"10.1161\/CIRCULATIONAHA.106.672402","volume":"115","author":"NR Cook","year":"2007","unstructured":"Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928\u201335.","journal-title":"Circulation."},{"key":"322_CR39","doi-asserted-by":"crossref","unstructured":"Movahedi F, Padman R, Antaki JF. Limitations of receiver operating characteristic curve on imbalanced data: assist device mortality risk scores. J Thorac Cardiovasc Surg. 2021;in press:1\u201312.","DOI":"10.1016\/j.jtcvs.2021.07.041"},{"issue":"3","key":"322_CR40","doi-asserted-by":"publisher","first-page":"696","DOI":"10.1007\/s00357-019-09345-1","volume":"37","author":"J Muschelli","year":"2020","unstructured":"Muschelli J. ROC and AUC with a binary predictor: a potentially misleading metric. J Classif. 2020;37(3):696\u2013708.","journal-title":"J Classif."},{"issue":"1","key":"322_CR41","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1177\/0969141313517497","volume":"21","author":"NJ Wald","year":"2014","unstructured":"Wald NJ, Bestwick JP. Is the area under an ROC curve a valid measure of the performance of a screening or diagnostic test? J Med Screen. 2014;21(1):51\u20136.","journal-title":"J Med Screen."},{"key":"322_CR42","doi-asserted-by":"publisher","first-page":"S253","DOI":"10.1016\/j.fertnstert.2005.07.658","volume":"84","author":"BW Mol","year":"2005","unstructured":"Mol BW, Coppus SF, Van der Veen F, Bossuyt PM. Evaluating predictors for the outcome of assisted reproductive technology: ROC curves are misleading; calibration is not! Fertil Steril. 2005;84:S253\u20134.","journal-title":"Fertil Steril."},{"issue":"4","key":"322_CR43","doi-asserted-by":"publisher","first-page":"498","DOI":"10.1111\/j.1466-8238.2011.00683.x","volume":"21","author":"A Jim\u00e9nez-Valverde","year":"2012","unstructured":"Jim\u00e9nez-Valverde A. Insights into the area under the receiver operating characteristic curve (AUC) as a discrimination measure in species distribution modelling. Glob Ecol Biogeogr. 2012;21(4):498\u2013507.","journal-title":"Glob Ecol Biogeogr."},{"key":"322_CR44","doi-asserted-by":"crossref","unstructured":"Halimu C, Kasem A, Newaz SS. Empirical comparison of area under ROC curve (AUC) and Matthews correlation coefficient (MCC) for evaluating machine learning algorithms on imbalanced datasets for binary classification. In: Proceedings of ICMLSC 2019 \u2013 the 3rd International Conference on Machine Learning and Soft Computing.\u00a0New York City:\u00a0Association for Computing Machinery;\u00a02019. p. 1\u20136.","DOI":"10.1145\/3310986.3311023"},{"key":"322_CR45","doi-asserted-by":"crossref","unstructured":"Drummond C, Holte RC. Explicitly representing expected cost: an alternative to ROC representation. In: Proceedings of ACM SIGKDD 2000 \u2013 the 6th ACM International Conference on Knowledge Discovery and Data Mining.\u00a0New York City: ACM; 2000. p. 198\u2013207.","DOI":"10.1145\/347090.347126"},{"issue":"4","key":"322_CR46","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1177\/0272989X9301300408","volume":"13","author":"B Littenberg","year":"1993","unstructured":"Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med Dec Making. 1993;13(4):313\u201321.","journal-title":"Med Dec Making."},{"issue":"10","key":"322_CR47","doi-asserted-by":"publisher","first-page":"1348","DOI":"10.1093\/bioinformatics\/btq140","volume":"26","author":"SJ Swamidass","year":"2010","unstructured":"Swamidass SJ, Azencott CA, Daily K, Baldi P. A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval. Bioinformatics. 2010;26(10):1348\u201356.","journal-title":"Bioinformatics."},{"issue":"3","key":"322_CR48","doi-asserted-by":"publisher","first-page":"570","DOI":"10.1080\/13658816.2013.862623","volume":"28","author":"RG Pontius Jr","year":"2014","unstructured":"Pontius RG Jr, Si K. The total operating characteristic to measure diagnostic ability for multiple thresholds. Int J Geogr Inf Sci. 2014;28(3):570\u201383.","journal-title":"Int J Geogr Inf Sci."},{"issue":"3","key":"322_CR49","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1177\/0272989X8900900307","volume":"9","author":"DK McClish","year":"1989","unstructured":"McClish DK. Analyzing a portion of the ROC curve. Med Dec Making. 1989;9(3):190\u20135.","journal-title":"Med Dec Making."},{"issue":"1","key":"322_CR50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12911-019-1014-6","volume":"20","author":"AM Carrington","year":"2020","unstructured":"Carrington AM, Fieguth PW, Qazi H, Holzinger A, Chen HH, Mayr F, et al. A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Med Inform Decis Making. 2020;20(1):1\u201312.","journal-title":"BMC Med Inform Decis Making."},{"issue":"3","key":"322_CR51","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1177\/09622802211060515","volume":"31","author":"L Lavazza","year":"2022","unstructured":"Lavazza L, Morasca S. Considerations on the region of interest in the ROC space. Stat Methods Med Res. 2022;31(3):419\u201337.","journal-title":"Stat Methods Med Res."},{"issue":"5","key":"322_CR52","doi-asserted-by":"publisher","first-page":"3977","DOI":"10.1007\/s10664-020-09861-4","volume":"25","author":"S Morasca","year":"2020","unstructured":"Morasca S, Lavazza L. On the assessment of software defect prediction models via ROC curves. Empir Softw Eng. 2020;25(5):3977\u20134019.","journal-title":"Empir Softw Eng."},{"issue":"3","key":"322_CR53","doi-asserted-by":"publisher","first-page":"683","DOI":"10.1007\/s11634-017-0295-9","volume":"12","author":"JM Vivo","year":"2018","unstructured":"Vivo JM, Franco M, Vicari D. Rethinking an ROC partial area index for evaluating the classification performance at a high specificity range. Adv Data Anal Classif. 2018;12(3):683\u2013704.","journal-title":"Adv Data Anal Classif."},{"issue":"1","key":"322_CR54","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1177\/0962280212452199","volume":"25","author":"S Parodi","year":"2016","unstructured":"Parodi S, Muselli M, Carlini B, Fontana V, Haupt R, Pistoia V, et al. Restricted ROC curves are useful tools to evaluate the performance of tumour markers. Stat Methods Med Res. 2016;25(1):294\u2013314.","journal-title":"Stat Methods Med Res."},{"key":"322_CR55","doi-asserted-by":"crossref","unstructured":"Jim\u00e9nez-Valverde A. The uniform AUC: dealing with the representativeness effect in presence\u2013absence models. Methods Ecol Evol. 2022;13(6):1224\u201336.","DOI":"10.1111\/2041-210X.13826"},{"issue":"1","key":"322_CR56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-9-410","volume":"9","author":"S Parodi","year":"2008","unstructured":"Parodi S, Pistoia V, Muselli M. Not proper ROC curves as new tool for the analysis of differentially expressed genes in microarray experiments. BMC Bioinformatics. 2008;9(1):1\u201330.","journal-title":"BMC Bioinformatics."},{"key":"322_CR57","doi-asserted-by":"publisher","first-page":"827","DOI":"10.1038\/nbt.1665","volume":"28","author":"MAQC Consortium","year":"2010","unstructured":"MAQC Consortium. The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models. Nat Biotechnol. 2010;28:827\u201338.","journal-title":"Nat Biotechnol."},{"issue":"1","key":"322_CR58","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L. Random Forests. Mach Learn. 2001;45(1):5\u201332.","journal-title":"Mach Learn."},{"issue":"7","key":"322_CR59","doi-asserted-by":"publisher","first-page":"e0133515","DOI":"10.1371\/journal.pone.0133515","volume":"10","author":"Y Tachi","year":"2015","unstructured":"Tachi Y, Hirai T, Toyoda H, Tada T, Hayashi K, Honda T, et al. Predictive ability of laboratory indices for liver fibrosis in patients with chronic hepatitis C after the eradication of hepatitis C virus. PLoS ONE. 2015;10(7):e0133515.","journal-title":"PLoS ONE."},{"issue":"6","key":"322_CR60","doi-asserted-by":"publisher","first-page":"e0199920","DOI":"10.1371\/journal.pone.0199920","volume":"13","author":"S Al-Shamsi","year":"2018","unstructured":"Al-Shamsi S, Regmi D, Govender R. Chronic kidney disease in patients at high risk of cardiovascular disease in the United Arab Emirates: a population-based study. PLOS ONE. 2018;13(6):e0199920.","journal-title":"PLOS ONE."},{"key":"322_CR61","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.jbi.2015.09.012","volume":"58","author":"MS Santos","year":"2015","unstructured":"Santos MS, Abreu PH, Garc\u00eda-Laencina PJ, Sim\u00e3o A, Carvalho A. A new cluster-based oversampling method for improving survival prediction of hepatocellular carcinoma patients. J Biomed Inform. 2015;58:49\u201359.","journal-title":"J Biomed Inform."},{"issue":"1","key":"322_CR62","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13040-016-0121-5","volume":"10","author":"Chicco D Ten","year":"2017","unstructured":"Ten Chicco D. quick tips for machine learning in computational biology. BioData Min. 2017;10(1):1\u201317.","journal-title":"BioData Min."},{"issue":"21","key":"322_CR63","doi-asserted-by":"publisher","first-page":"2765","DOI":"10.1001\/jama.287.21.2765","volume":"287","author":"DG Altman","year":"2002","unstructured":"Altman DG. Poor-quality medical research: what can journals do? J Am Med Assoc. 2002;287(21):2765\u20137.","journal-title":"J Am Med Assoc."},{"issue":"9","key":"322_CR64","doi-asserted-by":"publisher","first-page":"e190","DOI":"10.1371\/journal.pcbi.0030190","volume":"3","author":"E Grosch","year":"2007","unstructured":"Grosch E. Reply to \u201cTen simple rules for getting published\u2019\u2019. PLOS Comput Biol. 2007;3(9):e190.","journal-title":"PLOS Comput Biol."},{"issue":"383","key":"322_CR65","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1080\/01621459.1983.10478008","volume":"78","author":"EB Fowlkes","year":"1983","unstructured":"Fowlkes EB, Mallows CL. A method for comparing two hierarchical clusterings. J Am Stat Assoc. 1983;78(383):553\u201369.","journal-title":"J Am Stat Assoc."},{"issue":"1","key":"322_CR66","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12880-015-0068-x","volume":"15","author":"AA Taha","year":"2015","unstructured":"Taha AA, Hanbury A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging. 2015;15(1):1\u201328.","journal-title":"BMC Med Imaging."},{"key":"322_CR67","doi-asserted-by":"crossref","unstructured":"Chicco D, Jurman G. The ABC recommendations for validation of supervised machine learning results in biomedical sciences. Front Big Data. 2022;5:1\u20136.","DOI":"10.3389\/fdata.2022.979465"},{"issue":"1","key":"322_CR68","doi-asserted-by":"publisher","first-page":"e1010778","DOI":"10.1371\/journal.pcbi.1010778","volume":"19","author":"D Chicco","year":"2023","unstructured":"Chicco D, Shiradkar R. Ten quick tips for computational analysis of medical imagesa. PLOS Comput Biol. 2023;19(1):e1010778.","journal-title":"PLOS Comput Biol."},{"key":"322_CR69","doi-asserted-by":"publisher","first-page":"e1","DOI":"10.1017\/S026988892100014X","volume":"37","author":"ME P\u00e9rez-Pons","year":"2022","unstructured":"P\u00e9rez-Pons ME, Parra-Dominguez J, Hern\u00e1ndez G, Herrera-Viedma E, Corchado JM. Evaluation metrics and dimensional reduction for binary classification algorithms: a case study on bankruptcy prediction. Knowl Eng Rev. 2022;37:e1.","journal-title":"Knowl Eng Rev."},{"issue":"1","key":"322_CR70","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13040-022-00312-y","volume":"15","author":"D Chicco","year":"2022","unstructured":"Chicco D, Alameer A, Rahmati S, Jurman G. Towards a potential pan-cancer prognostic signature for gene expression based on probesets and ensemble machine learning. BioData Min. 2022;15(1):1\u201323.","journal-title":"BioData Min."}],"container-title":["BioData Mining"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13040-023-00322-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13040-023-00322-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13040-023-00322-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,24]],"date-time":"2023-02-24T08:08:02Z","timestamp":1677226082000},"score":1,"resource":{"primary":{"URL":"https:\/\/biodatamining.biomedcentral.com\/articles\/10.1186\/s13040-023-00322-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,17]]},"references-count":70,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["322"],"URL":"https:\/\/doi.org\/10.1186\/s13040-023-00322-4","relation":{},"ISSN":["1756-0381"],"issn-type":[{"value":"1756-0381","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,17]]},"assertion":[{"value":"25 October 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 February 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 February 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 February 2023","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Erroneous code was removed from the article.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The consents for the usage of the patients\u2019 data employed in the use cases were obtained by the original curators of those datasets\u00a0[\u2013].","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"4"}}