{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T01:07:59Z","timestamp":1783040879346,"version":"3.54.6"},"reference-count":33,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T00:00:00Z","timestamp":1632441600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["20K11939"],"award-info":[{"award-number":["20K11939"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Stability of feature selection algorithm refers to its robustness to the perturbations of the training set, parameter settings or initialization. A stable feature selection algorithm is crucial for identifying the relevant feature subset of meaningful and interpretable features which is extremely important in the task of knowledge discovery. Though there are many stability measures reported in the literature for evaluating the stability of feature selection, none of them follows all the requisite properties of a stability measure. Among them, the Kuncheva index and its modifications, are widely used in practical problems. In this work, the merits and limitations of the Kuncheva index and its existing modifications (Lustgarten, Wald, nPOG\/nPOGR, Nogueira) are studied and analysed with respect to the requisite properties of stability measure. One more limitation of the most recent modified similarity measure, Nogueira\u2019s measure, has been pointed out. Finally, corrections to Lustgarten\u2019s measure have been proposed to define a new modified stability measure that satisfies the desired properties and overcomes the limitations of existing popular similarity based stability measures. The effectiveness of the newly modified Lustgarten\u2019s measure has been evaluated with simple toy experiments.<\/jats:p>","DOI":"10.3390\/make3040038","type":"journal-article","created":{"date-parts":[[2021,9,27]],"date-time":"2021-09-27T23:08:31Z","timestamp":1632784111000},"page":"771-787","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["A Critical Study on Stability Measures of Feature Selection with a Novel Extension of Lustgarten Index"],"prefix":"10.3390","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7978-0916","authenticated-orcid":false,"given":"Rikta","family":"Sen","sequence":"first","affiliation":[{"name":"Department of Software and Information Science, Iwate Prefectural University, Sugo, Takizawa 020-0693, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2658-2745","authenticated-orcid":false,"given":"Ashis Kumar","family":"Mandal","sequence":"additional","affiliation":[{"name":"Department of Software and Information Science, Iwate Prefectural University, Sugo, Takizawa 020-0693, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Basabi","family":"Chakraborty","sequence":"additional","affiliation":[{"name":"Department of Software and Information Science, Iwate Prefectural University, Sugo, Takizawa 020-0693, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Brezo\u010dnik, L., Fister, I., and Podgorelec, V. (2018). Swarm Intelligence Algorithms for Feature Selection: A Review. Appl. Sci., 8.","DOI":"10.3390\/app8091521"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"109962","DOI":"10.1016\/j.measurement.2021.109962","article-title":"Effective feature selection with fuzzy entropy and similarity classifier for chatter vibration diagnosis","volume":"184","author":"Tran","year":"2021","journal-title":"Measurement"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tran, M.Q., Li, Y.C., Lan, C.Y., and Liu, M.K. (2020). Wind Farm Fault Detection by Monitoring Wind Speed in the Wake Region. Energies, 13.","DOI":"10.3390\/en13246559"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Tran, M.Q., Liu, M.K., and Elsisi, M. (2021). Effective multi-sensor data fusion for chatter detection in milling process. ISA Trans.","DOI":"10.1016\/j.isatra.2021.07.005"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Cilia, N.D., De Stefano, C., Fontanella, F., Raimondo, S., and Scotto di Freca, A. (2019). An Experimental Comparison of Feature-Selection and Classification Methods for Microarray Datasets. Information, 10.","DOI":"10.3390\/info10030109"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1007\/BF00993473","article-title":"Technical Note: Bias and the Quantification of Stability","volume":"20","author":"Turney","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Stiglic, G., and Kokol, P. (2010). Stability of ranked gene lists in large microarray analysis studies. J. Biomed. Biotechnol., 2010.","DOI":"10.1155\/2010\/616358"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Levner, I. (2005). Feature selection and nearest centroid classification for protein mass spectrometry. BMC Bioinform., 6.","DOI":"10.1186\/1471-2105-6-68"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"908","DOI":"10.1101\/gr.1837404","article-title":"Haplotype block partitioning and tag SNP selection using genotype data and their applications to association studies","volume":"14","author":"Zhang","year":"2004","journal-title":"Genome Res."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s10115-006-0040-8","article-title":"Stability of feature selection algorithms: A study on high-dimensional spaces","volume":"12","author":"Kalousis","year":"2007","journal-title":"Knowl. Inf. Syst."},{"key":"ref_11","unstructured":"Kuncheva, L.I. (2007, January 13\u201315). A stability index for feature selection. Proceedings of the 25th IASTED International Multi-Conference: Artificial Intelligence and Applications, Innsbruck, Austria."},{"key":"ref_12","unstructured":"Khaire, U.M., and Dhanalakshmi, R. (2019). Stability of feature selection algorithm: A review. J. King Saud Univ.-Comput. Inf. Sci."},{"key":"ref_13","first-page":"98","article-title":"A survey on feature selection stability measures","volume":"5","author":"Perumal","year":"2016","journal-title":"Int. J. Comput. Inf. Technol."},{"key":"ref_14","unstructured":"Dunne, K., Cunningham, P., and Azuaje, F. (2002). Solutions to instability problems with sequential wrapper-based approaches to feature selection. J. Mach. Learn. Res., 1\u201322."},{"key":"ref_15","first-page":"6345","article-title":"On the stability of feature selection algorithms","volume":"18","author":"Nogueira","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_16","unstructured":"Kalousis, A., Prados, J., and Hilario, M. (2005, January 27\u201330). Stability of feature selection algorithms. Proceedings of the Fifth IEEE International Conference on Data Mining (ICDM\u201905), Houston, TX, USA."},{"key":"ref_17","first-page":"262","article-title":"Stable gene selection from microarray data via sample weighting","volume":"9","author":"Yu","year":"2011","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yu, L., Ding, C., and Loscalzo, S. (2008, January 24\u201327). Stable feature selection via dense feature groups. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA.","DOI":"10.1145\/1401890.1401986"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zucknick, M., Richardson, S., and Stronach, E.A. (2008). Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods. Stat. Appl. Genet. Mol. Biol., 7.","DOI":"10.2202\/1544-6115.1307"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2057","DOI":"10.1093\/bioinformatics\/btn365","article-title":"Apparently low reproducibility of true differential expression discoveries in microarray studies","volume":"24","author":"Zhang","year":"2008","journal-title":"Bioinformatics"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1093\/bioinformatics\/19.1.90","article-title":"Gene selection: A Bayesian variable selection approach","volume":"19","author":"Lee","year":"2003","journal-title":"Bioinformatics"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2394","DOI":"10.1093\/bioinformatics\/bti319","article-title":"Bayesian model averaging: Development of an improved multi-class, gene selection and classification tool for microarray data","volume":"21","author":"Yeung","year":"2005","journal-title":"Bioinformatics"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Dutkowski, J., and Gambin, A. (2007). On consensus biomarker selection. BMC Bioinform., 8.","DOI":"10.1186\/1471-2105-8-S5-S5"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1084","DOI":"10.1093\/bioinformatics\/bti108","article-title":"Identifying differentially expressed genes from microarray experiments via statistic synthesis","volume":"21","author":"Yang","year":"2005","journal-title":"Bioinformatics"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1093\/bioinformatics\/btp630","article-title":"Robust biomarker identification for cancer diagnosis with ensemble feature selection methods","volume":"26","author":"Abeel","year":"2010","journal-title":"Bioinformatics"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2356","DOI":"10.1093\/bioinformatics\/btl400","article-title":"Reliable gene signatures for microarray classification: Assessment of stability and performance","volume":"22","author":"Davis","year":"2006","journal-title":"Bioinformatics"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1109\/TNB.2005.853657","article-title":"Multiple SVM-RFE for gene selection in cancer classification with expression data","volume":"4","author":"Duan","year":"2005","journal-title":"IEEE Trans. Nanobiosci."},{"key":"ref_28","first-page":"406","article-title":"Measuring stability of feature selection in biomedical datasets","volume":"Volume 2009","author":"Lustgarten","year":"2009","journal-title":"AMIA Annual Symposium Proceedings"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Khoshgoftaar, T.M., Fazelpour, A., Wang, H., and Wald, R. (2013, January 14\u201316). A survey of stability analysis of feature subset selection techniques. Proceedings of the 2013 IEEE 14th International Conference on Information Reuse & Integration (IRI), San Francisco, CA, USA.","DOI":"10.1109\/IRI.2013.6642502"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wald, R., Khoshgoftaar, T.M., and Napolitano, A. (2013, January 4\u20136). Stability of filter-and wrapper-based feature subset selection. Proceedings of the 2013 IEEE 25th International Conference on Tools with Artificial Intelligence, Herndon, VA, USA.","DOI":"10.1109\/ICTAI.2013.63"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1662","DOI":"10.1093\/bioinformatics\/btp295","article-title":"Evaluating reproducibility of differential expression discoveries in microarray studies by considering correlated molecular changes","volume":"25","author":"Zhang","year":"2009","journal-title":"Bioinformatics"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Nogueira, S., and Brown, G. (2015). Measuring the stability of feature selection with applications to ensemble methods. International Workshop on Multiple Classifier Systems, Springer.","DOI":"10.1007\/978-3-319-20248-8_12"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Nogueira, S., and Brown, G. (2016). Measuring the stability of feature selection. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer.","DOI":"10.1007\/978-3-319-46227-1_28"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/4\/38\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:04:38Z","timestamp":1760166278000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/4\/38"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,24]]},"references-count":33,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["make3040038"],"URL":"https:\/\/doi.org\/10.3390\/make3040038","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,24]]}}}