{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T23:53:08Z","timestamp":1781049188094,"version":"3.54.1"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,11,29]],"date-time":"2022-11-29T00:00:00Z","timestamp":1669680000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,29]],"date-time":"2022-11-29T00:00:00Z","timestamp":1669680000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100007195","name":"Universit\u00e0 degli Studi di Napoli Federico II","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100007195","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Stat"],"published-print":{"date-parts":[[2024,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Evaluating the ability of a classifier to make predictions on unseen data and increasing it by tweaking the learning algorithm are two of the main reasons motivating the evaluation of classifier predictive performance. In this study the behavior of Balanced <jats:inline-formula><jats:alternatives><jats:tex-math>$$AC_1$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mi>A<\/mml:mi>\n                    <mml:msub>\n                      <mml:mi>C<\/mml:mi>\n                      <mml:mn>1<\/mml:mn>\n                    <\/mml:msub>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>\u00a0\u2014\u00a0a novel classifier accuracy measure\u00a0\u2014\u00a0is investigated under different class imbalance conditions via a Monte Carlo simulation. The behavior of Balanced <jats:inline-formula><jats:alternatives><jats:tex-math>$$AC_1$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mi>A<\/mml:mi>\n                    <mml:msub>\n                      <mml:mi>C<\/mml:mi>\n                      <mml:mn>1<\/mml:mn>\n                    <\/mml:msub>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> is compared against that of several well-known performance measures based on binary confusion matrix. Study results reveal the suitability of Balanced <jats:inline-formula><jats:alternatives><jats:tex-math>$$AC_1$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mi>A<\/mml:mi>\n                    <mml:msub>\n                      <mml:mi>C<\/mml:mi>\n                      <mml:mn>1<\/mml:mn>\n                    <\/mml:msub>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> with both balanced and imbalanced data sets. A real example of the effects of class imbalance on the behavior of the investigated classifier performance measures is provided by comparing the performance of several machine learning algorithms in a churn prediction problem.<\/jats:p>","DOI":"10.1007\/s00180-022-01301-9","type":"journal-article","created":{"date-parts":[[2022,11,29]],"date-time":"2022-11-29T09:04:22Z","timestamp":1669712662000},"page":"363-383","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":40,"title":["Fair evaluation of classifier predictive performance based on binary confusion matrix"],"prefix":"10.1007","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0320-5541","authenticated-orcid":false,"given":"Amalia","family":"Vanacore","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria Sole","family":"Pellegrino","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Armando","family":"Ciardiello","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,11,29]]},"reference":[{"key":"1301_CR1","doi-asserted-by":"publisher","first-page":"220816","DOI":"10.1109\/ACCESS.2020.3042657","volume":"8","author":"J Ahn","year":"2020","unstructured":"Ahn J, Hwang J, Kim D, Choi H, Kang S (2020) A survey on churn analysis in various business domains. IEEE Access 8:220816\u2013220839","journal-title":"IEEE Access"},{"key":"1301_CR2","unstructured":"Akkaya B, \u00c7olako\u011flu N (2019) Comparison of multi-class classification algorithms on early diagnosis of heart diseases"},{"key":"1301_CR3","unstructured":"Ali M (2020) PyCaret: an open source, low-code machine learning library in Python. PyCaret version 1.0.0"},{"key":"1301_CR4","doi-asserted-by":"publisher","first-page":"7940","DOI":"10.1109\/ACCESS.2016.2619719","volume":"4","author":"A Amin","year":"2016","unstructured":"Amin A, Anwar S, Adnan A, Nawaz M, Howard N, Qadir J, Hawalah A, Hussain A (2016) Comparing oversampling techniques to handle the class imbalance problem: a customer churn prediction case study. IEEE Access 4:7940\u20137957","journal-title":"IEEE Access"},{"issue":"6","key":"1301_CR5","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1109\/TEVC.2003.819264","volume":"7","author":"W-H Au","year":"2003","unstructured":"Au W-H, Chan KC, Yao X (2003) A novel evolutionary data mining algorithm with applications to churn prediction. IEEE Trans Evol Comput 7(6):532\u2013545","journal-title":"IEEE Trans Evol Comput"},{"issue":"2","key":"1301_CR6","doi-asserted-by":"publisher","first-page":"825","DOI":"10.1016\/j.eswa.2006.10.022","volume":"34","author":"A Ben-David","year":"2008","unstructured":"Ben-David A (2008) Comparison of classification accuracy using Cohen\u2019s weighted kappa. Expert Syst Appl 34(2):825\u2013832","journal-title":"Expert Syst Appl"},{"key":"1301_CR7","doi-asserted-by":"crossref","unstructured":"Bhatnagar A, Srivastava S (2019) A robust model for churn prediction using supervised machine learning. In: 2019 IEEE 9th international conference on advanced computing (IACC), pp 45\u201349. IEEE","DOI":"10.1109\/IACC48062.2019.8971494"},{"key":"1301_CR8","doi-asserted-by":"crossref","unstructured":"Bisong E (2019) Building machine learning and deep learning models on Google cloud platform: a comprehensive guide for beginners. Apress","DOI":"10.1007\/978-1-4842-4470-8"},{"issue":"3","key":"1301_CR9","doi-asserted-by":"publisher","first-page":"4626","DOI":"10.1016\/j.eswa.2008.05.027","volume":"36","author":"J Burez","year":"2009","unstructured":"Burez J, Van den Poel D (2009) Handling class imbalance in customer churn prediction. Expert Syst Appl 36(3):4626\u20134636","journal-title":"Expert Syst Appl"},{"key":"1301_CR10","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1613\/jair.953","volume":"16","author":"NV Chawla","year":"2002","unstructured":"Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321\u2013357","journal-title":"J Artif Intell Res"},{"issue":"1","key":"1301_CR11","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37\u201346","journal-title":"Educ Psychol Meas"},{"issue":"1","key":"1301_CR12","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1016\/j.eswa.2006.09.038","volume":"34","author":"K Coussement","year":"2008","unstructured":"Coussement K, Van den Poel D (2008) Churn prediction in subscription services: an application of support vector machines while comparing two parameter-selection techniques. Expert Syst Appl 34(1):313\u2013327","journal-title":"Expert Syst Appl"},{"issue":"4","key":"1301_CR13","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1007\/s12559-016-9415-7","volume":"8","author":"K Dashtipour","year":"2016","unstructured":"Dashtipour K, Poria S, Hussain A, Cambria E, Hawalah AY, Gelbukh A, Zhou Q (2016) Multilingual sentiment analysis: state of the art and independent comparison of techniques. Cogn Comput 8(4):757\u2013771","journal-title":"Cogn Comput"},{"key":"1301_CR14","doi-asserted-by":"crossref","unstructured":"De S, Prabu P, Paulose J (2021) Effective ML techniques to predict customer churn. In: 2021 Third international conference on inventive research in computing applications (ICIRCA), pp 895\u2013902. IEEE","DOI":"10.1109\/ICIRCA51532.2021.9544785"},{"issue":"10","key":"1301_CR15","doi-asserted-by":"publisher","first-page":"12293","DOI":"10.1016\/j.eswa.2011.04.007","volume":"38","author":"KW De Bock","year":"2011","unstructured":"De Bock KW, Van den Poel D (2011) An empirical evaluation of rotation-based ensemble classifiers for customer churn prediction. Expert Syst Appl 38(10):12293\u201312301","journal-title":"Expert Syst Appl"},{"issue":"1","key":"1301_CR16","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1177\/1094670518795054","volume":"22","author":"A Dechant","year":"2019","unstructured":"Dechant A, Spann M, Becker JU (2019) Positive customer churn: an application to online dating. J Serv Res 22(1):90\u2013100","journal-title":"J Serv Res"},{"issue":"9","key":"1301_CR17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0222916","volume":"14","author":"R Delgado","year":"2019","unstructured":"Delgado R, Tibau X-A (2019) Why Cohen\u2019s Kappa should be avoided as performance measure in classification. PLoS One 14(9):e0222916","journal-title":"PLoS One"},{"key":"1301_CR18","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/j.rse.2011.11.020","volume":"118","author":"DC Duro","year":"2012","unstructured":"Duro DC, Franklin SE, Dub\u00e9 MG (2012) A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery. Remote Sens Environ 118:259\u2013272","journal-title":"Remote Sens Environ"},{"issue":"3","key":"1301_CR19","first-page":"23","volume":"9","author":"TY Fei","year":"2017","unstructured":"Fei TY, Shuan LH, Yan LJ, Xiaoning G, King SW (2017) Prediction on customer churn in the telecommunications sector using discretization and Na\u00efve Bayes classifier. Int J Adv Soft Comput Appl 9(3):23\u201335","journal-title":"Int J Adv Soft Comput Appl"},{"issue":"1","key":"1301_CR20","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1016\/j.patrec.2008.08.010","volume":"30","author":"C Ferri","year":"2009","unstructured":"Ferri C, Hern\u00e1ndez-Orallo J, Modroiu R (2009) An experimental comparison of performance measures for classification. Pattern Recognit Lett 30(1):27\u201338","journal-title":"Pattern Recognit Lett"},{"issue":"4","key":"1301_CR21","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1109\/TSMCC.2011.2161285","volume":"42","author":"M Galar","year":"2011","unstructured":"Galar M, Fernandez A, Barrenechea E, Bustince H, Herrera F (2011) A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans Syst Man Cybern Part C Appl Rev 42(4):463\u2013484","journal-title":"IEEE Trans Syst Man Cybern Part C Appl Rev"},{"issue":"6","key":"1301_CR22","first-page":"1","volume":"1","author":"K Gwet","year":"2002","unstructured":"Gwet K (2002) Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Stat Methods Inter-rater Reliab Assess 1(6):1\u20136","journal-title":"Stat Methods Inter-rater Reliab Assess"},{"key":"1301_CR23","doi-asserted-by":"crossref","unstructured":"Hassonah MA, Rodan A, Al-Tamimi A-K, Alsakran J (2019). Churn prediction: a comparative study using KNN and decision trees. In: 2019 Sixth HCT information technology trends (ITT), pp 182\u2013186. IEEE","DOI":"10.1109\/ITT48889.2019.9075077"},{"issue":"1","key":"1301_CR25","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1145\/1007730.1007737","volume":"6","author":"T Jo","year":"2004","unstructured":"Jo T, Japkowicz N (2004) Class imbalances versus small disjuncts. ACM SIGKDD Explor Newslett 6(1):40\u201349","journal-title":"ACM SIGKDD Explor Newslett"},{"key":"1301_CR26","unstructured":"Labatut V, Cherifi H (2011) Evaluation of performance measures for classifiers comparison. arXiv preprint arXiv:1112.4133"},{"issue":"3","key":"1301_CR27","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1109\/TG.2018.2888863","volume":"11","author":"E Lee","year":"2018","unstructured":"Lee E, Jang Y, Yoon D-M, Jeon J, Yang S-I, Lee S-K, Kim D-W, Chen PP, Guitart A, Bertens P et al (2018) Game data mining competition on churn prediction and survival analysis using commercial game log data. IEEE Trans Games 11(3):215\u2013226","journal-title":"IEEE Trans Games"},{"key":"1301_CR28","unstructured":"Mehra N, Gupta S (2013) Survey on multiclass classification methods"},{"key":"1301_CR29","doi-asserted-by":"crossref","unstructured":"Mishra A, Reddy US (2017) A comparative study of customer churn prediction in telecom industry using ensemble based classifiers. In: 2017 International conference on inventive computing and informatics (ICICI), pp 721\u2013725. IEEE","DOI":"10.1109\/ICICI.2017.8365230"},{"issue":"2","key":"1301_CR30","doi-asserted-by":"publisher","first-page":"2592","DOI":"10.1016\/j.eswa.2008.02.021","volume":"36","author":"EW Ngai","year":"2009","unstructured":"Ngai EW, Xiu L, Chau DC (2009) Application of data mining techniques in customer relationship management: a literature review and classification. Expert Syst Appl 36(2):2592\u20132602","journal-title":"Expert Syst Appl"},{"key":"1301_CR31","doi-asserted-by":"crossref","unstructured":"Nguyen NN, Duong AT (2021) Comparison of two main approaches for handling imbalanced data in churn prediction problem. J Adv Inf Technol 12(1)","DOI":"10.12720\/jait.12.1.29-35"},{"key":"1301_CR32","doi-asserted-by":"crossref","unstructured":"Qureshi SA, Rehman AS, Qamar AM, Kamal A, Rehman A (2013) Telecommunication subscribers\u2019 churn prediction model using machine learning. In: Eighth international conference on digital information management (ICDIM 2013), pp 131\u2013136. IEEE","DOI":"10.1109\/ICDIM.2013.6693977"},{"key":"1301_CR33","doi-asserted-by":"crossref","unstructured":"Sammut C, Webb GI (2011) Encyclopedia of machine learning. Springer","DOI":"10.1007\/978-0-387-30164-8"},{"issue":"4","key":"1301_CR34","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","volume":"45","author":"M Sokolova","year":"2009","unstructured":"Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45(4):427\u2013437","journal-title":"Inf Process Manag"},{"issue":"6","key":"1301_CR35","doi-asserted-by":"publisher","first-page":"731","DOI":"10.1007\/s12559-015-9360-x","volume":"7","author":"Z Tang","year":"2015","unstructured":"Tang Z, Lu J, Wang P (2015) A unified biologically-inspired prediction framework for classification of movement-related potentials based on a logistic regression model. Cogn Comput 7(6):731\u2013739","journal-title":"Cogn Comput"},{"key":"1301_CR36","doi-asserted-by":"crossref","unstructured":"Tharwat A (2020) Classification assessment methods. Appl Comput Inform","DOI":"10.1016\/j.aci.2018.08.003"},{"issue":"3","key":"1301_CR37","doi-asserted-by":"publisher","first-page":"2354","DOI":"10.1016\/j.eswa.2010.08.023","volume":"38","author":"W Verbeke","year":"2011","unstructured":"Verbeke W, Martens D, Mues C, Baesens B (2011) Building comprehensible customer churn prediction models with advanced rule induction techniques. Expert Syst Appl 38(3):2354\u20132364","journal-title":"Expert Syst Appl"},{"key":"1301_CR38","doi-asserted-by":"crossref","unstructured":"Xie H, Devlin S, Kudenko D, Cowling P (2015) Predicting player disengagement and first purchase with event-frequency based data representation. In: 2015 IEEE conference on computational intelligence and games (CIG), pp 230\u2013237. IEEE","DOI":"10.1109\/CIG.2015.7317919"},{"key":"1301_CR39","doi-asserted-by":"crossref","unstructured":"Zhang R, Li W, Tan W, Mo T (2017) Deep and shallow model for insurance churn prediction service. In: 2017 IEEE international conference on services computing (SCC), pp 346\u2013353. IEEE","DOI":"10.1109\/SCC.2017.51"},{"key":"1301_CR40","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1016\/j.ssci.2019.05.046","volume":"118","author":"J Zhou","year":"2019","unstructured":"Zhou J, Li E, Yang S, Wang M, Shi X, Yao S, Mitri HS (2019) Slope stability prediction for circular mode failure using gradient boosting machine approach based on an updated database of case histories. Saf Sci 118:505\u2013518","journal-title":"Saf Sci"}],"container-title":["Computational Statistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00180-022-01301-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00180-022-01301-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00180-022-01301-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T18:03:54Z","timestamp":1706724234000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00180-022-01301-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,29]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2]]}},"alternative-id":["1301"],"URL":"https:\/\/doi.org\/10.1007\/s00180-022-01301-9","relation":{},"ISSN":["0943-4062","1613-9658"],"issn-type":[{"value":"0943-4062","type":"print"},{"value":"1613-9658","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,29]]},"assertion":[{"value":"29 April 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 November 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}