{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T02:29:12Z","timestamp":1784255352555,"version":"3.55.0"},"reference-count":55,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,12,11]],"date-time":"2023-12-11T00:00:00Z","timestamp":1702252800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["SES-2120972"],"award-info":[{"award-number":["SES-2120972"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Feature selection metrics are commonly used in the machine learning pipeline to rank and select features before creating a predictive model. While many different metrics have been proposed for feature selection, final models are often evaluated by accuracy. In this paper, we consider the relationship between common feature selection metrics and accuracy. In particular, we focus on misorderings: cases where a feature selection metric may rank features differently than accuracy would. We analytically investigate the frequency of misordering for a variety of feature selection metrics as a function of parameters that represent how a feature partitions the data. Our analysis reveals that different metrics have systematic differences in how likely they are to misorder features which can happen over a wide range of partition parameters. We then perform an empirical evaluation with different feature selection metrics on several real-world datasets to measure misordering. Our empirical results generally match our analytical results, illustrating that misordering features happens in practice and can provide some insight into the performance of feature selection metrics.<\/jats:p>","DOI":"10.3390\/e25121646","type":"journal-article","created":{"date-parts":[[2023,12,11]],"date-time":"2023-12-11T11:24:43Z","timestamp":1702293883000},"page":"1646","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["On the Relationship between Feature Selection Metrics and Accuracy"],"prefix":"10.3390","volume":"25","author":[{"given":"Elise","family":"Epstein","sequence":"first","affiliation":[{"name":"Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH 44106, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naren","family":"Nallapareddy","sequence":"additional","affiliation":[{"name":"Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH 44106, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6673-6796","authenticated-orcid":false,"given":"Soumya","family":"Ray","sequence":"additional","affiliation":[{"name":"Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH 44106, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,12,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Hira, Z.M., and Gillies, D.F. (2015). A Review of Feature Selection and Feature Extraction Methods Applied on Microarray Data. Adv. Bioinform., 2015.","DOI":"10.1155\/2015\/198363"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Dinga, R., Penninx, B.W., Veltman, D.J., Schmaal, L., and Marquand, A.F. (2019). Beyond accuracy: Measures for assessing machine learning models, pitfalls and guidelines. bioRxiv.","DOI":"10.1101\/743138"},{"key":"ref_3","first-page":"1157","article-title":"An Introduction to Variable and Feature Selection","volume":"3","author":"Guyon","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_4","unstructured":"Nguyen, T., and Sanner, S. (2013, January 17\u201319). Algorithms for direct 0\u20131 loss optimization in binary classification. Proceedings of the International Conference on Machine Learning, PMLR, Atlanta, GA, USA."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3184","DOI":"10.1137\/21M1409445","article-title":"Quadratic convergence of smoothing Newton\u2019s method for 0\/1 loss optimization","volume":"31","author":"Zhou","year":"2021","journal-title":"SIAM J. Optim."},{"key":"ref_6","unstructured":"He, X., and Little, M.A. (2023). 1248 An efficient, provably exact, practical algorithm for the 0\u20131 loss linear classification problem. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.ins.2014.05.042","article-title":"A review of microarray datasets and applied feature selection methods","volume":"282","author":"Herrera","year":"2014","journal-title":"Inf. Sci."},{"key":"ref_8","first-page":"612","article-title":"Evaluating the Impact of GINI Index and Information Gain on Classification using Decision Tree Classifier Algorithm","volume":"11","author":"Tangirala","year":"2020","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Mingers, J. (1989). An Empirical Comparison of Selection Measures for Decision-Tree Induction, Springer. Machine Learning.","DOI":"10.1007\/BF00116837"},{"key":"ref_10","unstructured":"Nie, F., Huang, H., Cai, X., and Ding, C. (2010). Efficient and Robust Feature Selection via Joint \u21132,1-Norms Minimization. Neural Inf. Process. Syst., 23, Available online: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2010\/file\/09c6c3783b4a70054da74f2538ed47c6-Paper.pdf."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3048","DOI":"10.1016\/j.patcog.2011.12.008","article-title":"An unsupervised approach to feature discretization and selection","volume":"45","author":"Ferreria","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_12","unstructured":"Breiman, L., Friedman, J., Stone, C.J., and Olshen, R. (1984). Classification and Regression Trees, Routledge."},{"key":"ref_13","unstructured":"Meyen, S. (2016). Relation between Classification Accuracy and Mutual Information in Equally Weighted Classification Tasks. [Master\u2019s Thesis, University of Tuebingen]."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1093\/nsr\/nwx106","article-title":"A brief introduction to weakly supervised learning","volume":"5","author":"Zhou","year":"2017","journal-title":"Natl. Sci. Rev."},{"key":"ref_15","unstructured":"Quinlan, J.R. (1993). C4.5: Programs for Machine Learning, Morgan Kaufmann Publishers Inc."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J. (1984). Classification and Regression Trees. Biometrics, 40.","DOI":"10.2307\/2530946"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1515\/crll.1909.136.210","article-title":"Neue begr\u00fcndung der theorie quadratischer formen von unendlichvielen ver\u00e4nderlichen","volume":"1909","author":"Hellinger","year":"1909","journal-title":"J. Reine Angew. Math."},{"key":"ref_18","first-page":"401","article-title":"On a measure of divergence between two multinomial populations","volume":"7","author":"Bhattacharyya","year":"1946","journal-title":"Sankhya Indian J. Stat."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1703","DOI":"10.1016\/S0031-3203(03)00035-9","article-title":"Feature extraction based on the Bhattacharyya distance","volume":"36","author":"Choi","year":"2003","journal-title":"Pattern Recognit."},{"key":"ref_20","unstructured":"Guyon, I., Gunn, S., Ben-Hur, A., and Dror, G. (2023, November 22). Result Analysis of the NIPS 2003 Feature Selection Challenge with data retrieved from University of California Irvine Machine Learning Repository. Available online: https:\/\/papers.nips.cc\/paper\/2004\/file\/5e751896e527c862bf67251a474b3819-Paper.pdf."},{"key":"ref_21","unstructured":"Li, J. (2023, October 05). Feature Selection Datasets. Data retrieved from Arizona State University. Available online: https:\/\/jundongl.github.io\/scikit-feature\/datasets.html."},{"key":"ref_22","unstructured":"Dua, D., and Graff, C. (2023, November 22). UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/."},{"key":"ref_23","unstructured":"Grimaldi, M., Cunningham, P., and Kokaram, A. (2023, November 22). An Evaluation of Alternative Feature Selection Strategies and Ensemble Techniques for Classifying Music. Available online: https:\/\/citeseerx.ist.psu.edu\/document?repid=rep1&type=pdf&doi=22a1a59619809e8ecf7ff051ed262bea0f835f92#page=44."},{"key":"ref_24","unstructured":"Duda, R.O., Hart, P.E., and Stork, D.G. (2000). Pattern Classification, Wiley-Interscience Publication."},{"key":"ref_25","unstructured":"Hall, M.A. (1999). Correlation-Based Feature Selection for Machine Learning. [Ph.D. Thesis, The University of Waikato]. Available online: https:\/\/www.cs.waikato.ac.nz\/~mhall\/thesis.pdf."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.1109\/TKDE.2003.1245283","article-title":"Benchmarking Attribute Selection Techniques for Discrete Class Data Mining","volume":"15","author":"Hall","year":"2003","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1016\/j.patcog.2011.06.006","article-title":"An ensemble of filters and classifiers for microarray data classification","volume":"45","year":"2012","journal-title":"Pattern Recognit."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/S0004-3702(03)00079-1","article-title":"Consistency-based search in feature selection","volume":"151","author":"Dash","year":"2003","journal-title":"Artif. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"104396","DOI":"10.1016\/j.chemolab.2021.104396","article-title":"A hybrid ensemble-filter wrapper feature selection approach for medical data classification","volume":"217","author":"Singh","year":"2021","journal-title":"Chemom. Intell. Lab. Syst."},{"key":"ref_30","unstructured":"Kononenko, I. (1994). European Conference on Machine Learning, Springer."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1142\/S0219720005001004","article-title":"Minimum Redundancy Feature Selection from Microarray Gene Expression Data","volume":"3","author":"Ding","year":"2005","journal-title":"J. Bioinform. Comput. Biol."},{"key":"ref_32","unstructured":"Yu, L., and Liu, H. (2003, January 21\u201324). Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution. Proceedings of the Twentieth International Conference on Machine Learning, Washington, DC, USA."},{"key":"ref_33","unstructured":"Zhao, Z., and Liu, H. (2007, January 6\u201312). Searching for Interacting Features. Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression Shrinkage and Selection via the Lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. R. Stat. Soc."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"e237","DOI":"10.7717\/peerj-cs.237","article-title":"A Sparse-Modeling Based Approach for Class Specific Feature Selection","volume":"5","author":"Nardone","year":"2019","journal-title":"PeerJ Comput. Sci."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1023\/A:1012487302797","article-title":"Gene Selection for Cancer Classification using Support Vector Machines","volume":"46","author":"Guyon","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1007\/s11280-015-0381-x","article-title":"Aspect term extraction for sentiment analysis in large movie reviews using Gini Index feature selection method and SVM classifier","volume":"20","author":"Manek","year":"2016","journal-title":"World Wide Web"},{"key":"ref_38","unstructured":"Epstein, E. (2022). The Relationship between Common Feature Selection Metrics and Accuracy. [Master\u2019s Thesis, Case Western Reserve University]."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1093\/biomet\/13.1.25","article-title":"Notes on the History of Correlation","volume":"13","author":"Pearson","year":"1920","journal-title":"Biometrika"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1006\/jcss.1997.1504","article-title":"A decision-theoretic generalization of on-line learning and an application to boosting","volume":"55","author":"Freund","year":"1997","journal-title":"J. Comput. Syst. Sci."},{"key":"ref_41","unstructured":"Schapire, R.E. (1992). The Design and Analysis of Efficient Learning Algorithms. [Ph.D. Thesis, MIT Press ]."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1023\/A:1007400206189","article-title":"Learning to recognize volcanoes on Venus","volume":"30","author":"Burl","year":"1998","journal-title":"Mach. Learn."},{"key":"ref_43","unstructured":"Ouyang, T., Ray, S., Rabinovich, M., and Allman, M. (2011, January 20\u201322). Can network characteristics detect spam effectively in a stand-alone enterprise?. Proceedings of the Passive and Active Measurement: 12th International Conference, PAM 2011, Atlanta, GA, USA. Proceedings 12."},{"key":"ref_44","unstructured":"Guyon, I. (2023, November 22). Madelon. UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/dataset\/171\/madelon."},{"key":"ref_45","unstructured":"Guyon, I., Gunn, S., Ben-Hur, A., and Dror, G. (2023, November 22). Gisette. UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/dataset\/170\/gisette."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v036.i11","article-title":"Feature Selection with the Boruta Package","volume":"36","author":"Kursa","year":"2010","journal-title":"J. Stat. Softw."},{"key":"ref_47","unstructured":"Sui, B. (2013). Information Gain Feature Selection Based On Feature Interactions. [Master\u2019s Thesis, University of Houston]."},{"key":"ref_48","unstructured":"Koprinska, I. (2023, October 01). Feature Selection for Brain-Computer Interfaces. PAKDD 2009 International Workshops. Available online: https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-642-14640-4.pdf."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1007\/s10115-012-0487-8","article-title":"A Review of Feature Selection Methods on Synthetic Data","volume":"34","year":"2013","journal-title":"Knowl. Inf. Syst."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Wu, L., Wang, Y., Zhang, S., and Zhang, Y. (2017, January 19\u201321). Fusing Gini Index and Term Frequency for Text Feature Selection. Proceedings of the 2017 IEEE Third International Conference on Multimedia Big Data, Laguna Hills, CA, USA.","DOI":"10.1109\/BigMM.2017.65"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Jirapech-Umpai, T., and Aitken, S. (2005). Feature selection and classification for microarray data analysis: Evolutionary methods for identifying predictive genes. BMC Bioinform., 6.","DOI":"10.1186\/1471-2105-6-148"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Masoudi-Sobhanzadeh, Y., Motieghader, H., and Masoudi-Nejad, A. (2019). FeatureSelect: A software for feature selection based on machine. learning approaches. BMC Bioinform., 20.","DOI":"10.1186\/s12859-019-2754-0"},{"key":"ref_53","unstructured":"Phuong, H.T.M., Hanh, L.T.M., and Binh, N.T. (2021). A Comparative Analysis of Filter-Based Feature Selection Methods for Software Fault Prediction. Res. Dev. Inf. Commun. Technol."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Taheri, N., and Nezamabadi-pour, H. (2014, January 29\u201330). A hybrid feature selection method for high-dimensional data. Proceedings of the 4th International Conference on Computer and Knowledge Engineering, Mashhad, Iran.","DOI":"10.1109\/ICCKE.2014.6993381"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Bi, X., Liu, J.G., and Cao, Y.S. (2019, January 15\u201317). Classification of Low-grade and High-grade Glioma using Multiparametric Radiomics Model. Proceedings of the IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC 2019), Chengdu, China.","DOI":"10.1109\/ITNEC.2019.8729245"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/12\/1646\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:36:46Z","timestamp":1760132206000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/12\/1646"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,11]]},"references-count":55,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["e25121646"],"URL":"https:\/\/doi.org\/10.3390\/e25121646","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,11]]}}}