{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T00:44:42Z","timestamp":1774313082964,"version":"3.50.1"},"reference-count":35,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2017,4,1]],"date-time":"2017-04-01T00:00:00Z","timestamp":1491004800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004329","name":"Slovenian Research Agency","doi-asserted-by":"publisher","award":["P2-0241"],"award-info":[{"award-number":["P2-0241"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We propose a novel feature selection method based on quadratic mutual information which has its roots in Cauchy\u2013Schwarz divergence and Renyi entropy. The method uses the direct estimation of quadratic mutual information from data samples using Gaussian kernel functions, and can detect second order non-linear relations. Its main advantages are: (i) unified analysis of discrete and continuous data, excluding any discretization; and (ii) its parameter-free design. The effectiveness of the proposed method is demonstrated through an extensive comparison with mutual information feature selection (MIFS), minimum redundancy maximum relevance (MRMR), and joint mutual information (JMI) on classification and regression problem domains. The experiments show that proposed method performs comparably to the other methods when applied to classification problems, except it is considerably faster. In the case of regression, it compares favourably to the others, but is slower.<\/jats:p>","DOI":"10.3390\/e19040157","type":"journal-article","created":{"date-parts":[[2017,4,3]],"date-time":"2017-04-03T13:32:30Z","timestamp":1491226350000},"page":"157","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Quadratic Mutual Information Feature Selection"],"prefix":"10.3390","volume":"19","author":[{"given":"Davor","family":"Sluga","sequence":"first","affiliation":[{"name":"University of Ljubljana, Faculty of Computer and Information Science, Ljubljana 1000, Slovenia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Uro\u0161","family":"Lotri\u010d","sequence":"additional","affiliation":[{"name":"University of Ljubljana, Faculty of Computer and Information Science, Ljubljana 1000, Slovenia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2017,4,1]]},"reference":[{"key":"ref_1","first-page":"1157","article-title":"An introduction to variable and feature selection","volume":"3","author":"Guyon","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s00521-013-1368-0","article-title":"A review of feature selection methods based on mutual information","volume":"24","author":"Vergara","year":"2014","journal-title":"Neural Comput. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1016\/S0004-3702(97)00043-X","article-title":"Wrappers for feature subset selection","volume":"97","author":"Kohavi","year":"1997","journal-title":"Artif. Intell."},{"key":"ref_4","unstructured":"Hall, M.A. (July, January 29). Correlation-based feature selection of discrete and numeric class machine learning. Proceedings of the Seventeenth International Conference on Machine Learning, Stanford, CA, USA."},{"key":"ref_5","first-page":"1531","article-title":"Fast binary feature selection with conditional mutual information","volume":"5","author":"Fleuret","year":"2004","journal-title":"J. Mach. Learn. Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.compeleceng.2013.11.024","article-title":"A survey on feature selection methods","volume":"40","author":"Chandrashekar","year":"2014","journal-title":"Comput. Electr. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Principe, J.C. (2010). Information Theoretic Learning: Renyi\u2019s Entropy and Kernel Perspectives, Springer Science & Business Media.","DOI":"10.1007\/978-1-4419-1570-2"},{"key":"ref_8","unstructured":"Brown, G. (2009, January 16\u201318). A new perspective for information theoretic feature selection. Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS-09), Clearwater Beach, FL, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1590\/S0101-74382011000300006","article-title":"R\u00e9nyi entropy and Cauchy-Schwarz mutual information applied to mifs-u variable selection algorithm: A comparative study","volume":"31","author":"Macrini","year":"2011","journal-title":"Pesqui. Oper."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Sluga, D., and Lotric, U. (2013, January 4\u20136). Generalized information-theoretic measures for feature selection. Proceedings of the International Conference on Adaptive and Natural Computing Algorithms, Lausanne, Switzerland.","DOI":"10.1007\/978-3-642-37213-1_20"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1109\/TNN.2004.841414","article-title":"Estimating optimal feature subsets using efficient estimation of high-dimensional mutual information","volume":"16","author":"Chow","year":"2005","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"734","DOI":"10.1109\/TKDE.2012.35","article-title":"A survey of discretization techniques: Taxonomy and empirical analysis in supervised learning","volume":"25","author":"Garcia","year":"2013","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_13","unstructured":"Irani, K.B. (September, January 28). Multi-interval discretization of continuous-valued attributes for classification learning. Proceedings of the 13th International Joint Conference on Artificial Intelligence, Chambery, France."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1214\/aoms\/1177704472","article-title":"On estimation of a probability density function and mode","volume":"33","author":"Parzen","year":"1962","journal-title":"Ann. Math. Stat."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1641","DOI":"10.1016\/S0167-8655(02)00127-7","article-title":"Kernel density estimation with adaptive varying window size","volume":"23","author":"Katkovnik","year":"2002","journal-title":"Pattern Recognit. Lett."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"066138","DOI":"10.1103\/PhysRevE.69.066138","article-title":"Estimating mutual information","volume":"69","author":"Kraskov","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Walters-Williams, J., and Li, Y. (2009, January 14\u201316). Estimation of mutual information: A survey. Proceedings of the International Conference on Rough Sets and Knowledge Technology, Gold Coast, QLD, Australia.","DOI":"10.1007\/978-3-642-02962-2_49"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"80","DOI":"10.3390\/e15010080","article-title":"Machine learning with squared-loss mutual information","volume":"15","author":"Sugiyama","year":"2012","journal-title":"Entropy"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1080\/00107510902823517","article-title":"Generalised information and entropy measures in physics","volume":"50","author":"Beck","year":"2009","journal-title":"Contemp. Phys."},{"key":"ref_20","unstructured":"Renyi, A. (July, January 20). On measures of entropy and information. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1109\/TNN.2002.1031936","article-title":"Generalized information potential criterion for adaptive system training","volume":"13","author":"Erdogmus","year":"2002","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_22","unstructured":"Renyi, A. (1976). Some Fundamental Questions About Information Theory, Akademia Kiado."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1109\/72.298224","article-title":"Using mutual information for selecting features in supervised neural net learning","volume":"5","author":"Battiti","year":"1994","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1109\/72.977291","article-title":"Input feature selection for classification problems","volume":"13","author":"Kwak","year":"2002","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1109\/TPAMI.2005.159","article-title":"Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy","volume":"27","author":"Peng","year":"2005","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","unstructured":"Yang, H., and Moody, J. (1999, January 22\u201325). Feature selection based on joint mutual information. Proceedings of the International ICSC Symposium on Advances in Intelligent Data Analysis, Rochester, NY, USA."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Rajan, K., and Bialek, W. (2013). Maximally informative \u201cstimulus energies\u201d in the analysis of neural responses to natural signals. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0071959"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Fitzgerald, J.D., Rowekamp, R.J., Sincich, L.C., and Sharpee, T.O. (2011). Second order dimensionality reduction using minimum and maximum mutual information models. PLoS Comput. Biol., 7.","DOI":"10.1371\/journal.pcbi.1002249"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"45","DOI":"10.3109\/0954898X.2011.566303","article-title":"Analyzing multicomponent receptive fields from neural responses to natural stimuli","volume":"22","author":"Rowekamp","year":"2011","journal-title":"Netw. Comput. Neural Syst."},{"key":"ref_30","unstructured":"S\u00e1nchez-Maro\u00f1o, N., Alonso-Betanzos, A., and Tombilla-Sanrom\u00e1n, M. (2007, January 16\u201319). Filter methods for feature selection\u2014A comparative study. Proceedings of the International Conference on Intelligent Data Engineering and Automated Learning, Birmingham, UK."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neunet.2013.07.003","article-title":"Is mutual information adequate for feature selection in regression?","volume":"48","author":"Doquire","year":"2013","journal-title":"Neural Netw."},{"key":"ref_32","unstructured":"Silverman, B.W. (1986). Density Estimation for Statistics and Data Analysis, CRC Press."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Seth, S., and Pr\u00edncipe, J.C. (2009, January 14\u201319). On speeding up computation in information theoretic learning. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2009), Atlanta, GA, USA.","DOI":"10.1109\/IJCNN.2009.5178933"},{"key":"ref_34","unstructured":"Lichman, M. (2016, December 01). UCI Machine Learning Repository. Available online: http:\/\/archive.ics.uci.edu\/ml."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/1656274.1656278","article-title":"The WEKA data mining software: An update","volume":"11","author":"Hall","year":"2009","journal-title":"ACM SIGKDD Explor. Newsl."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/4\/157\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:31:50Z","timestamp":1760207510000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/4\/157"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,4,1]]},"references-count":35,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2017,4]]}},"alternative-id":["e19040157"],"URL":"https:\/\/doi.org\/10.3390\/e19040157","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,4,1]]}}}