{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T05:48:22Z","timestamp":1777528102766,"version":"3.51.4"},"reference-count":69,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,2,4]],"date-time":"2021-02-04T00:00:00Z","timestamp":1612396800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,2,4]],"date-time":"2021-02-04T00:00:00Z","timestamp":1612396800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BioData Mining"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Evaluating binary classifications is a pivotal task in statistics and machine learning, because it can influence decisions in multiple areas, including for example prognosis or therapies of patients in critical conditions. The scientific community has not agreed on a general-purpose statistical indicator for evaluating two-class confusion matrices (having true positives, true negatives, false positives, and false negatives) yet, even if advantages of the Matthews correlation coefficient (MCC) over accuracy and F<jats:sub>1<\/jats:sub> score have already been shown.In this manuscript, we reaffirm that MCC is a robust metric that summarizes the classifier performance in a single value, if positive and negative cases are of equal importance. We compare MCC to other metrics which value positive and negative cases equally: balanced accuracy (BA), bookmaker informedness (BM), and markedness (MK). We explain the mathematical relationships between MCC and these indicators, then show some use cases and a bioinformatics scenario where these metrics disagree and where MCC generates a more informative response.Additionally, we describe three exceptions where BM can be more appropriate: analyzing classifications where dataset prevalence is unrepresentative, comparing classifiers on different datasets, and assessing the random guessing level of a classifier. Except in these cases, we believe that MCC is the most informative among the single metrics discussed, and suggest it as standard measure for scientists of all fields. A Matthews correlation coefficient close to +1, in fact, means having high values for all the other confusion matrix metrics. The same cannot be said for balanced accuracy, markedness, bookmaker informedness, accuracy and F<jats:sub>1<\/jats:sub> score.<\/jats:p>","DOI":"10.1186\/s13040-021-00244-z","type":"journal-article","created":{"date-parts":[[2021,2,4]],"date-time":"2021-02-04T13:06:30Z","timestamp":1612443990000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":718,"title":["The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation"],"prefix":"10.1186","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9655-7142","authenticated-orcid":false,"given":"Davide","family":"Chicco","sequence":"first","affiliation":[]},{"given":"Niklas","family":"T\u00f6tsch","sequence":"additional","affiliation":[]},{"given":"Giuseppe","family":"Jurman","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,2,4]]},"reference":[{"key":"244_CR1","volume-title":"Model Selection and Error Estimation in a Nutshell","author":"O Luca","year":"2020","unstructured":"Luca O. Model Selection and Error Estimation in a Nutshell. Berlin: Springer; 2020."},{"key":"244_CR2","unstructured":"Naser MZ, Alavi A. Insights into performance fitness and error metrics for machine learning. 2020:1\u201325. arXiv preprint arXiv:2006.00887."},{"issue":"7","key":"244_CR3","doi-asserted-by":"publisher","first-page":"e67863","DOI":"10.1371\/journal.pone.0067863","volume":"8","author":"Q Wei","year":"2013","unstructured":"Wei Q, Dunbrack Jr. RL. The role of balanced training and testing data sets for binary classifiers in bioinformatics. PLoS ONE. 2013; 8(7):e67863.","journal-title":"PLoS ONE"},{"issue":"10","key":"244_CR4","first-page":"27","volume":"3","author":"M Bekkar","year":"2013","unstructured":"Bekkar M, Djemaa HK, Alitouche TA. Evaluation measures for models assessment over imbalanced data sets. J Inf Eng Appl. 2013; 3(10):27\u201338.","journal-title":"J Inf Eng Appl"},{"key":"244_CR5","volume-title":"Proceedings of Pacific Symposium on Biocomputing 2019","author":"R Ramola","year":"2019","unstructured":"Ramola R, Jain S, Radivojac P. Estimating classification accuracy in positive-unlabeled learning: characterization and correction strategies. In: Proceedings of Pacific Symposium on Biocomputing 2019. Singapore: World Scientific: 2019. p. 124\u201335."},{"key":"244_CR6","volume-title":"Proceedings of ICDM 2011 \u2013 the 11th IEEE International Conference on Data Mining Workshop","author":"C Parker","year":"2011","unstructured":"Parker C. An analysis of performance measures for binary classifiers. In: Proceedings of ICDM 2011 \u2013 the 11th IEEE International Conference on Data Mining Workshop. Vancouver: IEEE: 2011. p. 517\u201326."},{"issue":"4","key":"244_CR7","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","volume":"45","author":"M Sokolova","year":"2009","unstructured":"Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. Inf Process Manag. 2009; 45(4):427\u201337.","journal-title":"Inf Process Manag"},{"issue":"15","key":"244_CR8","doi-asserted-by":"publisher","first-page":"2811","DOI":"10.3390\/molecules24152811","volume":"24","author":"A R\u00e1cz","year":"2019","unstructured":"R\u00e1cz A, Bajusz D, H\u00e9berger K. Multi-level comparison of machine learning classifiers and their performance metrics. Molecules. 2019; 24(15):2811.","journal-title":"Molecules"},{"issue":"6","key":"244_CR9","doi-asserted-by":"publisher","first-page":"508","DOI":"10.1186\/cc3000","volume":"8","author":"V Bewick","year":"2004","unstructured":"Bewick V, Cheek L, Ball J. Statistics review 13: receiver operating characteristic curves. Critic Care. 2004; 8(6):508.","journal-title":"Critic Care"},{"issue":"3","key":"244_CR10","doi-asserted-by":"publisher","first-page":"e0118432","DOI":"10.1371\/journal.pone.0118432","volume":"10","author":"T Saito","year":"2015","unstructured":"Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE. 2015; 10(3):e0118432.","journal-title":"PLoS ONE"},{"issue":"4","key":"244_CR11","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1007\/s00330-014-3487-0","volume":"25","author":"S Halligan","year":"2015","unstructured":"Halligan S, Altman DG, Mallett S. Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: a discussion and proposal for an alternative approach. Eur Radiol. 2015; 25(4):932\u20139.","journal-title":"Eur Radiol"},{"issue":"1","key":"244_CR12","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1177\/0969141313517497","volume":"21","author":"NJ Wald","year":"2014","unstructured":"Wald NJ, Bestwick JP. Is the area under an ROC curve a valid measure of the performance of a screening or diagnostic test?J Med Screen. 2014; 21(1):51\u201356.","journal-title":"J Med Screen"},{"issue":"3","key":"244_CR13","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1145\/65943.65945","volume":"7","author":"V Raghavan","year":"1989","unstructured":"Raghavan V, Bollmann P, Jung GS. A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans Inf Syst (TOIS). 1989; 7(3):205\u201329.","journal-title":"ACM Trans Inf Syst (TOIS)"},{"key":"244_CR14","unstructured":"Cao C, Chicco D, Hoffman MM. The MCC-F1 curve: a performance evaluation technique for binary classification. 2020:1\u201317. arXiv preprint arXiv:2006.11278v1."},{"issue":"1","key":"244_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12864-019-6413-7","volume":"21","author":"D Chicco","year":"2020","unstructured":"Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom. 2020; 21(1):1\u201313.","journal-title":"BMC Genom"},{"issue":"35","key":"244_CR16","first-page":"1","volume":"10","author":"D Chicco","year":"2017","unstructured":"Chicco D. Ten quick tips for machine learning in computational biology. BioData Min. 2017; 10(35):1\u201317.","journal-title":"BioData Min"},{"issue":"2","key":"244_CR17","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","volume":"405","author":"BW Matthews","year":"1975","unstructured":"Matthews BW. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta (BBA) Protein Struct. 1975; 405(2):442\u201351.","journal-title":"Biochim Biophys Acta (BBA) Protein Struct"},{"key":"244_CR18","first-page":"1","volume":"743138","author":"R Dinga","year":"2019","unstructured":"Dinga R, Penninx BW, Veltman DJ, Schmaal L, Marquand AF. Beyond accuracy: measures for assessing machine learning models, pitfalls and guidelines. bioRxiv. 2019; 743138:1\u201320.","journal-title":"bioRxiv"},{"issue":"1","key":"244_CR19","doi-asserted-by":"publisher","first-page":"38","DOI":"10.2214\/AJR.18.20224","volume":"212","author":"GS Handelman","year":"2019","unstructured":"Handelman GS, Kok HK, Chandra RV, Razavi AH, Huang S, Brooks M, Lee MJ, Asadi H. Peering into the black box of artificial intelligence: evaluation metrics of machine learning methods. Am J Roentgenol. 2019; 212(1):38\u201343.","journal-title":"Am J Roentgenol"},{"issue":"8","key":"244_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0041882","volume":"7","author":"G Jurman","year":"2012","unstructured":"Jurman G, Riccadonna S, Furlanello C. A comparison of MCC and CEN error measures in multi-class prediction. PLoS ONE. 2012; 7(8):1\u20138.","journal-title":"PLoS ONE"},{"issue":"6","key":"244_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0177678","volume":"12","author":"S Boughorbel","year":"2017","unstructured":"Boughorbel S, Jarray F, El-Anbari M. Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric. PLoS ONE. 2017; 12(6):1\u201317.","journal-title":"PLoS ONE"},{"key":"244_CR22","volume-title":"Proceedings of ICPR 2010 \u2013 the 20th IAPR International Conference on Pattern Recognition","author":"KH Brodersen","year":"2010","unstructured":"Brodersen KH, Ong CS, Stephan KE, Buhmann JM. The balanced accuracy and its posterior distribution. In: Proceedings of ICPR 2010 \u2013 the 20th IAPR International Conference on Pattern Recognition. Istanbul: IEEE: 2010. p. 3121\u20134."},{"key":"244_CR23","volume-title":"Proceedings of ROBOT 2013 \u2013 the 1st Iberian Robotics Conference","author":"H Carrillo","year":"2014","unstructured":"Carrillo H, Brodersen KH, Castellanos JA. Probabilistic performance evaluation for multiclass classification using the posterior balanced accuracy. In: Proceedings of ROBOT 2013 \u2013 the 1st Iberian Robotics Conference. Madrid: Springer: 2014. p. 347\u201361."},{"issue":"4","key":"244_CR24","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1002\/gepi.20211","volume":"31","author":"DR Velez","year":"2007","unstructured":"Velez DR, White BC, Motsinger AA, Bush WS, Ritchie MD, Williams SM, Moore JH. A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction. Genet Epidemiol. 2007; 31(4):306\u201315.","journal-title":"Genet Epidemiol"},{"key":"244_CR25","volume-title":"Proceedings of GECCO 2008 \u2013 the 10th Annual Conference on Genetic and Evolutionary Computation","author":"NE Hardison","year":"2008","unstructured":"Hardison NE, Fanelli TJ, Dudek SM, Reif DM, Ritchie MD, Motsinger-Reif AA. A balanced accuracy fitness function leads to robust analysis using grammatical evolution neural networks in the case of class imbalance. In: Proceedings of GECCO 2008 \u2013 the 10th Annual Conference on Genetic and Evolutionary Computation. New York City: Association for Computing Machinery: 2008. p. 353\u2013354."},{"issue":"1","key":"244_CR26","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1007\/s00439-010-0905-5","volume":"129","author":"J Gui","year":"2011","unstructured":"Gui J, Moore JH, Kelsey KT, Marsit CJ, Karagas MR, Andrew AS. A novel survival multifactor dimensionality reduction method for detecting gene\u2013gene interactions with application to bladder cancer prognosis. Hum Genet. 2011; 129(1):101\u201310.","journal-title":"Hum Genet"},{"key":"244_CR27","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1016\/j.bbr.2013.11.003","volume":"259","author":"A Frick","year":"2014","unstructured":"Frick A, Gingnell M, Marquand AF, Howner K, Fischer H, Kristiansson M, Williams SCR, Fredrikson M, Furmark T. Classifying social anxiety disorder using multivoxel pattern analyses of brain function and structure. Behav Brain Res. 2014; 259:330\u20135.","journal-title":"Behav Brain Res"},{"issue":"16","key":"244_CR28","first-page":"S16","volume":"15","author":"S Ran","year":"2014","unstructured":"Ran S, Li Y, Zink D, Loo L-H. Supervised prediction of drug-induced nephrotoxicity based on interleukin-6 and-8 expression levels. BMC Bioinformatics. 2014; 15(16):S16.","journal-title":"BMC Bioinformatics"},{"issue":"2","key":"244_CR29","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1016\/j.cmpb.2012.11.003","volume":"110","author":"T Berber","year":"2013","unstructured":"Berber T, Alpkocak A, Balci P, Dicle O. Breast mass contour segmentation algorithm in digital mammograms. Comput Methods Prog Biomed. 2013; 110(2):150\u20139.","journal-title":"Comput Methods Prog Biomed"},{"key":"244_CR30","doi-asserted-by":"publisher","first-page":"1157","DOI":"10.3233\/JAD-190594","volume":"74","author":"A Morin","year":"2020","unstructured":"Morin A, Samper-Gonzalez J, Bertrand A, Str\u00f3er S, Dormont D, Mendes M, Coup\u00e9 P, Ahdidan J, L\u00e9vy M, Samri D, Hampel H, Dubois B, Teichmann M, Epelbaum S, Colliot O. Accuracy of MRI classification algorithms in a tertiary memory center clinical routine cohort. J Alzheimer\u2019s Dis. 2020; 74:1157\u201366.","journal-title":"J Alzheimer\u2019s Dis"},{"issue":"2","key":"244_CR31","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1016\/j.ygyno.2015.01.541","volume":"137","author":"MK McConechy","year":"2015","unstructured":"McConechy MK, Talhouk A, Li-Chang HH, Leung S, Huntsman DG, Blake Gilks C, McAlpine JN. Detection of DNA mismatch repair (MMR) deficiencies by immunohistochemistry can effectively diagnose the microsatellite instability (MSI) phenotype in endometrial carcinomas. Gynecol Oncol. 2015; 137(2):306\u201310.","journal-title":"Gynecol Oncol"},{"issue":"5","key":"244_CR32","doi-asserted-by":"publisher","first-page":"933","DOI":"10.1249\/MSS.0000000000000840","volume":"48","author":"K Ellis","year":"2016","unstructured":"Ellis K, Kerr J, Godbole S, Staudenmayer J, Lanckriet G. Hip and wrist accelerometer algorithms for free-living behavior classification. Med Sci Sports Exer. 2016; 48(5):933.","journal-title":"Med Sci Sports Exer"},{"key":"244_CR33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.compag.2014.09.016","volume":"110","author":"PM Szczypi\u0144ski","year":"2015","unstructured":"Szczypi\u0144ski PM, Klepaczko A, Zapotoczny P. Identifying barley varieties by computer vision. Comput Electron Agric. 2015; 110:1\u20138.","journal-title":"Comput Electron Agric"},{"key":"244_CR34","volume-title":"Proceedings of CEC 2005 \u2013 the 2005 IEEE Congress on Evolutionary Computation, vol. 3","author":"MR Peterson","year":"2005","unstructured":"Peterson MR, Raymer ML, Lamont GB. Balanced accuracy for feature subset selection with genetic algorithms. In: Proceedings of CEC 2005 \u2013 the 2005 IEEE Congress on Evolutionary Computation, vol. 3. Edinburgh: IEEE: 2005. p. 2506\u201313."},{"key":"244_CR35","volume-title":"Proceedings of IbPRIA 2009 \u2013 the 4th Iberian Conference on Pattern Recognition and Image Analysis","author":"V Garc\u00eda","year":"2009","unstructured":"Garc\u00eda V, Mollineda RA, S\u00e1nchez JS. Index of balanced accuracy: a performance measure for skewed class distributions. In: Proceedings of IbPRIA 2009 \u2013 the 4th Iberian Conference on Pattern Recognition and Image Analysis. Povoa de Varzim: Springer: 2009. p. 441\u20138."},{"key":"244_CR36","volume-title":"Proceedings of ICCS 2003 \u2013 the 2003 International Conference on Cognitive Science","author":"DM Powers","year":"2003","unstructured":"Powers DM. Recall & precision versus the bookmaker. In: Proceedings of ICCS 2003 \u2013 the 2003 International Conference on Cognitive Science. Berlin: Springer: 2003."},{"issue":"93","key":"244_CR37","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1126\/science.ns-4.93.453.b","volume":"93","author":"CS Peirce","year":"1884","unstructured":"Peirce CS. The numerical measure of the success of predictions. Science. 1884; 93(93):453\u20134.","journal-title":"Science"},{"issue":"1","key":"244_CR38","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3","volume":"3","author":"WJ Youden","year":"1950","unstructured":"Youden WJ. Index for rating diagnostic tests. Cancer. 1950; 3(1):32\u201335.","journal-title":"Cancer"},{"issue":"3","key":"244_CR39","first-page":"419","volume":"50","author":"MD Ruopp","year":"2008","unstructured":"Ruopp MD, Perkins NJ, Whitcomb BW, Schisterman EF. Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection. Biom J Math Methods Biosci. 2008; 50(3):419\u201330.","journal-title":"Biom J Math Methods Biosci"},{"issue":"4","key":"244_CR40","first-page":"428","volume":"47","author":"NJ Perkins","year":"2005","unstructured":"Perkins NJ, Schisterman EF. The Youden Index and the optimal cut-point corrected for measurement error. Biom J Math Methods Biosci. 2005; 47(4):428\u201341.","journal-title":"Biom J Math Methods Biosci"},{"issue":"1","key":"244_CR41","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1097\/01.ede.0000147512.81966.ba","volume":"16","author":"EF Schisterman","year":"2005","unstructured":"Schisterman EF, Perkins NJ, Liu A, Bondell H. Optimal cut-point and its corresponding Youden Index to discriminate individuals using pooled blood samples. Epidemiology. 2005; 16(1):73\u201381.","journal-title":"Epidemiology"},{"key":"244_CR42","volume-title":"Proceedings of CEC 2012 \u2013 the 2012 IEEE Congress on Evolutionary Computation","author":"A Atyabi","year":"2012","unstructured":"Atyabi A, Luerssen M, Fitzgibbon S, Powers DM. Evolutionary feature selection and electrode reduction for EEG classification. In: Proceedings of CEC 2012 \u2013 the 2012 IEEE Congress on Evolutionary Computation. Hangzhou: IEEE: 2012. p. 1\u20138."},{"key":"244_CR43","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1016\/j.fsigen.2018.10.019","volume":"38","author":"D Taylor","year":"2019","unstructured":"Taylor D, Kitselaar M, Powers DM. The generalisability of artificial neural networks used to classify electrophoretic data produced under different conditions. Forensic Sci Int Genet. 2019; 38:181\u20134.","journal-title":"Forensic Sci Int Genet"},{"key":"244_CR44","volume-title":"Proceedings of BI 2012 \u2013 the 2012 International Conference on Brain Informatics","author":"A Atyabi","year":"2012","unstructured":"Atyabi A, Luerssen MH, Fitzgibbon SP, Powers DM. The impact of PSO based dimension reduction on EEG classification. In: Proceedings of BI 2012 \u2013 the 2012 International Conference on Brain Informatics. Macau: Springer: 2012. p. 220\u201331."},{"issue":"2","key":"244_CR45","first-page":"123","volume":"5","author":"N Takayama","year":"2017","unstructured":"Takayama N, Takahashi H. Performance evaluation criteria for partial blur segmentation in classification and segmentation contexts. Inst Image Electron Eng Japan (IIEEJ) Trans Image Electron Vis Comput. 2017; 5(2):123\u201333.","journal-title":"Inst Image Electron Eng Japan (IIEEJ) Trans Image Electron Vis Comput"},{"issue":"2","key":"244_CR46","doi-asserted-by":"publisher","first-page":"142","DOI":"10.7763\/IJMLC.2015.V5.498","volume":"5","author":"HB Ali","year":"2015","unstructured":"Ali HB, Powers DM, Jia X, Zhang Y. Extended non-negative matrix factorization for face and facial expression recognition. Int J Mach Learn Comput. 2015; 5(2):142.","journal-title":"Int J Mach Learn Comput"},{"key":"244_CR47","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1016\/j.jbi.2015.11.004","volume":"58","author":"N Alvaro","year":"2015","unstructured":"Alvaro N, Conway M, Doan S, Lofi C, Overington J, Collier N. Crowdsourcing twitter annotations to identify first-hand experiences of prescription drug use. J Biomed Inf. 2015; 58:280\u20137.","journal-title":"J Biomed Inf"},{"key":"244_CR48","doi-asserted-by":"crossref","unstructured":"Fairley I, Mendzil A, Togneri M, Reeve DE. Detailed sediment class mapping using unmanned aerial vehicles. Coastal Sediments 2019, Preprint ahead of publication. 2019:2472\u201382.","DOI":"10.1142\/9789811204487_0212"},{"key":"244_CR49","doi-asserted-by":"publisher","first-page":"296","DOI":"10.1016\/j.scitotenv.2019.02.017","volume":"664","author":"O Rahmati","year":"2019","unstructured":"Rahmati O, Kornejady A, Samadi M, Deo RC, Conoscenti C, Lombardo L, Dayal K, Taghizadeh-Mehrjardi R, Pourghasemi HR, Kumar S, Bui TD. PMT: New analytical framework for automated evaluation of geo-environmental modelling approaches. Sci Total Environ. 2019; 664:296\u2013311.","journal-title":"Sci Total Environ"},{"issue":"4","key":"244_CR50","first-page":"1068","volume":"48","author":"C Garmon","year":"2017","unstructured":"Garmon C. The accuracy of hospital merger screening methods. Res Dev (RAND) J Econ. 2017; 48(4):1068\u2013102.","journal-title":"Res Dev (RAND) J Econ"},{"key":"244_CR51","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-405888-0.00005-2","volume-title":"Bayes\u2019 rule: Doing Bayesian Data Analysis","author":"JK Kruschke","year":"2015","unstructured":"Kruschke JK. Bayes\u2019 rule: Doing Bayesian Data Analysis. Amsterdam: Elsevier; 2015, pp. 99\u2013120."},{"key":"244_CR52","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1016\/j.patcog.2019.02.023","volume":"91","author":"A Luque","year":"2019","unstructured":"Luque A, Carrasco A, Mart\u00edn A, de las Heras A. The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recogn. 2019; 91:216\u201331.","journal-title":"Pattern Recogn"},{"key":"244_CR53","doi-asserted-by":"crossref","unstructured":"T\u00f6tsch N, Hoffmann D. Classifier uncertainty: evidence, potential impact, and probabilistic treatment. arXiv preprint arXiv:2006.11105. 2020.","DOI":"10.7717\/peerj-cs.398"},{"issue":"1","key":"244_CR54","first-page":"37","volume":"2","author":"DM Powers","year":"2011","unstructured":"Powers DM. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J Mach Learn Technol. 2011; 2(1):37\u201363.","journal-title":"J Mach Learn Technol"},{"key":"244_CR55","doi-asserted-by":"crossref","unstructured":"Sedgwick P. Pearson\u2019s correlation coefficient. Br Med J (BMJ). 2012:345.","DOI":"10.1136\/bmj.e4483"},{"issue":"3","key":"244_CR56","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1109\/2.485891","volume":"29","author":"AK Jain","year":"1996","unstructured":"Jain AK, Mao J, Moidin Mohiuddin K. Artificial neural networks: a tutorial. Computer. 1996; 29(3):31\u201344.","journal-title":"Computer"},{"issue":"12","key":"244_CR57","doi-asserted-by":"publisher","first-page":"6745","DOI":"10.1073\/pnas.96.12.6745","volume":"96","author":"U Alon","year":"1999","unstructured":"Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc Natl Acad Sci (PNAS). 1999; 96(12):6745\u201350.","journal-title":"Proc Natl Acad Sci (PNAS)"},{"issue":"1","key":"244_CR58","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1093\/bib\/bbl016","volume":"8","author":"A-L Boulesteix","year":"2006","unstructured":"Boulesteix A-L, Strimmer K. Partial least squares: a versatile tool for the analysis of high-dimensional genomic data. Brief Bioinforma. 2006; 8(1):32\u201344.","journal-title":"Brief Bioinforma"},{"key":"244_CR59","unstructured":"Boulesteix A-L, Durif G, Lambert-Lacroix S, Peyre J, Strimmer K. Package \u2018plsgenomics\u2019. 2018. https:\/\/cran.r-project.org\/web\/packages\/plsgenomics\/index.html. Accessed 10 Oct 2020."},{"key":"244_CR60","unstructured":"Alon U, Barkai N Notterman DA, Gish K, Ybarra S Mack D, Levine AJ. Data pertaining to the article \u2018Broad patterns of gene expression revealed by clustering of tumor and normal colon tissues probed by oligonucleotide arrays\u2019. 2000. http:\/\/genomics-pubs.princeton.edu\/oncology\/affydata\/index.html. Accessed 10 Oct 2020."},{"key":"244_CR61","volume-title":"Classification and regression trees (CART) theory and applications","author":"R Timofeev","year":"2004","unstructured":"Timofeev R. Classification and regression trees (CART) theory and applications. Berlin: Humboldt University; 2004."},{"key":"244_CR62","volume-title":"International Conference on Database Theory","author":"K Beyer","year":"1999","unstructured":"Beyer K, Goldstein J, Ramakrishnan R, Shaft U. When is \u201cnearest neighbor\u201d meaningful? In: International Conference on Database Theory. Berlin: Springer: 1999. p. 217\u201335."},{"issue":"11","key":"244_CR63","doi-asserted-by":"publisher","first-page":"1325","DOI":"10.1093\/bioinformatics\/btl094","volume":"22","author":"M Yousef","year":"2006","unstructured":"Yousef M, Nebozhyn M, Shatkay H, Kanterakis S, Showe LC, Showe MK. Combining multi-species genomic data for microRNA identification using a Naive Bayes\u0308 classifier. Bioinformatics. 2006; 22(11):1325\u201334.","journal-title":"Bioinformatics"},{"issue":"4","key":"244_CR64","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/5254.708428","volume":"13","author":"MA Hearst","year":"1998","unstructured":"Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B. Support vector machines. IEEE Intell Syst Appl. 1998; 13(4):18\u201328.","journal-title":"IEEE Intell Syst Appl"},{"issue":"8","key":"244_CR65","doi-asserted-by":"publisher","first-page":"938","DOI":"10.1016\/j.jclinepi.2009.11.009","volume":"63","author":"K Rufibach","year":"2010","unstructured":"Rufibach K. Use of Brier score to assess binary predictions. J Clin Epidemiol. 2010; 63(8):938\u20139.","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"244_CR66","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Measurement. 1960; 20(1):37\u201346.","journal-title":"Educ Psychol Measurement"},{"key":"244_CR67","volume-title":"Proceedings of ICTIR 2015 \u2013 the ACM SIGIR 2015 International Conference on the Theory of Information Retrieval","author":"F Sebastiani","year":"2015","unstructured":"Sebastiani F. An axiomatically derived measure for the evaluation of classification algorithms. In: Proceedings of ICTIR 2015 \u2013 the ACM SIGIR 2015 International Conference on the Theory of Information Retrieval. New York City: ACM: 2015. p. 11\u201320."},{"issue":"383","key":"244_CR68","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1080\/01621459.1983.10478008","volume":"78","author":"EB Fowlkes","year":"1983","unstructured":"Fowlkes EB, Mallows CL. A method for comparing two hierarchical clusterings. J Am Stat Assoc. 1983; 78(383):553\u201369.","journal-title":"J Am Stat Assoc"},{"key":"244_CR69","first-page":"242","volume":"270","author":"A Campagner","year":"2020","unstructured":"Campagner A, Sconfienza L, Cabitza F. H-accuracy, an alternative metric to assess classification models in medicine. Stud Health Technol Inf. 2020; 270:242\u20136.","journal-title":"Stud Health Technol Inf"}],"container-title":["BioData Mining"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13040-021-00244-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13040-021-00244-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13040-021-00244-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,29]],"date-time":"2023-01-29T00:07:36Z","timestamp":1674950856000},"score":1,"resource":{"primary":{"URL":"https:\/\/biodatamining.biomedcentral.com\/articles\/10.1186\/s13040-021-00244-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,4]]},"references-count":69,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["244"],"URL":"https:\/\/doi.org\/10.1186\/s13040-021-00244-z","relation":{},"ISSN":["1756-0381"],"issn-type":[{"value":"1756-0381","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,4]]},"assertion":[{"value":"1 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 January 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 February 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"13"}}