{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T09:19:38Z","timestamp":1772788778612,"version":"3.50.1"},"reference-count":83,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2015,1,7]],"date-time":"2015-01-07T00:00:00Z","timestamp":1420588800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Clustering is an unsupervised process to determine which unlabeled objects in a set share interesting properties. The objects are grouped into k subsets (clusters) whose elements optimize a proximity measure. Methods based on information theory have proven to be feasible alternatives. They are based on the assumption that a cluster is one subset with the minimal possible degree of \u201cdisorder\u201d. They attempt to minimize the entropy of each cluster. We propose a clustering method based on the maximum entropy principle. Such a method explores the space of all possible probability distributions of the data to find one that maximizes the entropy subject to extra conditions based on prior information about the clusters. The prior information is based on the assumption that the elements of a cluster are \u201csimilar\u201d to each other in accordance with some statistical measure. As a consequence of such a principle, those distributions of high entropy that satisfy the conditions are favored over others. Searching the space to find the optimal distribution of object in the clusters represents a hard combinatorial problem, which disallows the use of traditional optimization techniques. Genetic algorithms are a good alternative to solve this problem. We benchmark our method relative to the best theoretical performance, which is given by the Bayes classifier when data are normally distributed, and a multilayer perceptron network, which offers the best practical performance when data are not normal. In general, a supervised classification method will outperform a non-supervised one, since, in the first case, the elements of the classes are known a priori. In what follows, we show that our method\u2019s effectiveness is comparable to a supervised one. This clearly exhibits the superiority of our method.<\/jats:p>","DOI":"10.3390\/e17010151","type":"journal-article","created":{"date-parts":[[2015,1,7]],"date-time":"2015-01-07T12:35:46Z","timestamp":1420634146000},"page":"151-180","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["A Clustering Method Based on the Maximum Entropy Principle"],"prefix":"10.3390","volume":"17","author":[{"given":"Edwin","family":"Aldana-Bobadilla","sequence":"first","affiliation":[{"name":"Instituto de Investigaciones en Matem\u00e1ticas Aplicadas y en Sistemas, Universidad Nacional Aut\u00f3noma de M\u00e9xico, Ciudad Universitaria, 04510 Ciudad de M\u00e9xico, Mexico"}]},{"given":"Angel","family":"Kuri-Morales","sequence":"additional","affiliation":[{"name":"Instituto Tecnol\u00f3gico Aut\u00f3nomo de M\u00e9xico, R\u00edo Hondo 1. Col. Progreso Tizap\u00e1n, 01080 Ciudad de M\u00e9xico, Mexico"}]}],"member":"1968","published-online":{"date-parts":[[2015,1,7]]},"reference":[{"key":"ref_1","first-page":"401","article-title":"On a measure of divergence between two statistical populations","volume":"7","author":"Bhattacharyya","year":"1946","journal-title":"Indian J. Stat."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02289565","article-title":"Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis","volume":"29","author":"Kruskal","year":"1964","journal-title":"Psychometrika"},{"key":"ref_3","unstructured":"Mahalanobis, P.C. (1936, January 16). On the generalized distance in statistics. Calcutta, India."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1023\/A:1012801612483","article-title":"On clustering validation techniques","volume":"17","author":"Halkidi","year":"2001","journal-title":"J. Intell. Inf. Syst."},{"key":"ref_5","unstructured":"Rokach, L., and Maimon, O. (2005). Data Mining and Knowledge Discovery Handbook, Springer."},{"key":"ref_6","first-page":"281","article-title":"Some methods for classification and analysis of multivariate observations","volume":"1","author":"Neyman","year":"1967","journal-title":"The Fifth Berkeley Symposium on Mathematical Statistics and Probability"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bezdek, J.C. (1981). Pattern Recognition with Fuzzy Objective Function Algorithms, Springer.","DOI":"10.1007\/978-1-4757-0450-1"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1080\/01969727308546046","article-title":"A fuzzy relative of the ISODATA process and its use in detecting vompact well-separated clusters","volume":"3","author":"Dunn","year":"1973","journal-title":"J. Cybern."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1080\/01969727408546059","article-title":"Well-separated clusters and optimal fuzzy partitions","volume":"4","author":"Dunn","year":"1974","journal-title":"J. Cybern."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Friedman, J., Hastie, T., and Tibshirani, R. (2001). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer. [2nd].","DOI":"10.1007\/978-0-387-21606-5"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Guha, S., Rastogi, R., and Shim, K. (1998, January 1\u20134). Cure: An efficient clustering algorithm for large databases. Seattle, WA, USA.","DOI":"10.1145\/276304.276312"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhang, T., Ramakrishnan, R., and Livny, M. (, 1996). Birch: An efficient data clustering method for very large databases. Montreal, QC, Canada.","DOI":"10.1145\/233269.233324"},{"key":"ref_13","unstructured":"Ester, M., Kriegel, H.P., Sander, J., and Xu, X. (1996, January 2\u20134). A density-based algorithm for discovering clusters in large spatial databases with noise. Portland, OR, USA."},{"key":"ref_14","unstructured":"Caruana, R., Elhaway, M., Nguyen, N., and Smith, C. (, January 18\u201322). Meta clustering. Hong Kong, China."},{"key":"ref_15","unstructured":"Das, S., Abraham, A., and Konar, A. (2009). Metaheuristic Clustering, Springer."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/BF02294245","article-title":"An examination of procedures for determining the number of clusters in a data set","volume":"50","author":"Milligan","year":"1985","journal-title":"Psychometrika"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1111\/1467-9868.00293","article-title":"Estimating the number of clusters in a data set via the gap statistic","volume":"63","author":"Tibshirani","year":"2001","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"467","DOI":"10.20965\/jaciii.1999.p0467","article-title":"Determining the optimal number of clusters by an extended RPCL algorithm","volume":"3","author":"Li","year":"1999","journal-title":"J. Adv. Comput. Intell. Intell. Inf."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1080\/01621459.1989.10478754","article-title":"Approximate confidence intervals for the number of clusters","volume":"84","author":"Peck","year":"1989","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_20","unstructured":"Yan, M. (2005). Methods of Determining the Number of Clusters in a Data Set and a New Clustering Criterion. [Ph.D. Thesis, Virginia Polytechnic Institute and State University]."},{"key":"ref_21","unstructured":"Cha, S.H. (2008, January 24\u201326). Taxonomy of nominal type histogram distance measures. Harvard, MA, USA."},{"key":"ref_22","first-page":"281","article-title":"A novel validity index for determination of the optimal number of clusters","volume":"84","author":"Kim","year":"2001","journal-title":"IEICE Trans. Inf. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Liu, Y., Li, Z., Xiong, H., Gao, X., and Wu, J. (2010, January 13\u201317). Understanding of internal clustering validation measures. Sydney, Australia.","DOI":"10.1109\/ICDM.2010.35"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Larsen, B., and Aone, C. (1999, January 15\u201318). Fast and effective text mining using linear-time document clustering. San Diego, CA, USA.","DOI":"10.1145\/312129.312186"},{"key":"ref_25","first-page":"583","article-title":"Cluster ensembles\u2014A knowledge reuse framework for combining multiple partitions","volume":"3","author":"Strehl","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A Mathematical Theory of Communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_27","unstructured":"Zhao, Y., and Karypis, G. Available online: http:\/\/glaros.dtc.umn.edu\/gkhome\/node\/165."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1111\/j.2517-6161.1986.tb01408.x","article-title":"A note on bayes factors for log-linear contingency table models with vague prior information","volume":"48","author":"Raftery","year":"1986","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"J. Comput. Appl. Math."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1109\/TPAMI.1979.4766909","article-title":"A cluster separation measure","volume":"PAMI-1","author":"Davies","year":"1979","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","unstructured":"Rend\u00f3n, E., Garcia, R., Abundez, I., Gutierrez, C., Gasca, E., del Razo, F., and Gonzalez, A. (2008, January 23\u201325). Niva: A robust cluster validity. Heraklion, Greece."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1080\/01621459.1971.10482356","article-title":"Objective criteria for the evaluation of clustering methods","volume":"66","author":"Rand","year":"1971","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/BF01908075","article-title":"Comparing partitions","volume":"2","author":"Hubert","year":"1985","journal-title":"J. Classif."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Graham, R.L., Knuth, D.E., and Patashnik, O. (1989). Concrete Mathematics: A Foundation for Computer Science, Addison-Wesley.","DOI":"10.1063\/1.4822863"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1016\/0305-0548(86)90048-1","article-title":"Future paths for integer programming and links to artificial intelligence","volume":"13","author":"Glover","year":"1986","journal-title":"Comput. Oper. Res."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Glover, F., and Laguna, M. (1997). Tabu Search, Kluwer Academic Publishers.","DOI":"10.1007\/978-1-4615-6089-0"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1016\/0305-0548(86)90050-X","article-title":"The general employee scheduling problem. An integration of MS and AI","volume":"13","author":"Glover","year":"1986","journal-title":"Comput. Oper. Res."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"975","DOI":"10.1007\/BF01009452","article-title":"Optimization by simulated annealing: Quantitative studies","volume":"34","author":"Kirkpatrick","year":"1984","journal-title":"J. Stat. Phys."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1126\/science.220.4598.671","article-title":"Optimization by simulated annealing","volume":"220","author":"Kirkpatrick","year":"1983","journal-title":"Science"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1016\/j.tcs.2005.05.020","article-title":"Ant colony optimization theory: A survey","volume":"344","author":"Dorigo","year":"2005","journal-title":"Theor. Comput. Sci."},{"key":"ref_41","unstructured":"Eberhart, R., and Kennedy, J. (1995, January 4\u20136). A new optimizer using particle swarm theory. Nagoya, Japan."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/evco.1993.1.1.1","article-title":"An overview of evolutionary algorithms for parameter optimization","volume":"1","author":"Schwefel","year":"1993","journal-title":"Evol. Comput."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Beyer, H.G., and Schwefel, H.P. (2002). Evolution Strategies\u2014A Comprehensive Introduction, Kluwer Academic Publishers.","DOI":"10.1023\/A:1015059928466"},{"key":"ref_44","unstructured":"Fogel, L.J. (1990, January 5\u20137). The future of evolutionary programming. Pacific Grove, CA, USA."},{"key":"ref_45","unstructured":"Banzhaf, W., Nordin, P., Keller, R.E., and Francone, F.D. (1997). Genetic Programming: An Introduction: On the Automatic Evolution of Computer Programs and Its Applications, Morgan Kaufmann Publishers."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Holland, J.H. (1992). Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence, MIT Press. [2nd].","DOI":"10.7551\/mitpress\/1090.001.0001"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1109\/72.265964","article-title":"Convergence Analysis of Canonical Genetic Algorithms","volume":"5","author":"Rudolph","year":"1994","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Kuri-Morales, A., and Aldana-Bobadilla, E. (2013, January 24\u201330). The best genetic algorithm I. Mexico City, Mexico.","DOI":"10.1007\/978-3-642-45111-9_1"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Kuri-Morales, A., Aldana-Bobadilla, E., and L\u00f3pez-Pe\u00f1a, I. (2013, January 24\u201330). The best genetic algorithm II. Mexico City, Mexico.","DOI":"10.1007\/978-3-642-45111-9_2"},{"key":"ref_50","unstructured":"Kuri-Morales, A. A statistical genetic algorithm. Available online: http:\/\/cursos.itam.mx\/akuri\/2006\/Algoritmos%20Gen%E9ticos\/2Statistical%20GA.pdf."},{"key":"ref_51","unstructured":"Kuri-Morales, A., and Villegas, C.Q. (1998, January 7\u201310). A universal eclectic genetic algorithm for constrained optimization. Aachen, Germany."},{"key":"ref_52","unstructured":"Abudalfa, S.I. (2010). Metaheuristic Clustering Algorithm. [Ph.D. Thesis, The Islamic University of Gaza]."},{"key":"ref_53","unstructured":"Caballero, R., Laguna, M., Mart\u00ed, R., and Molina, J. Available online: http:\/\/www.uv.es\/sestio\/TechRep\/tr02-06.pdf."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1016\/j.aca.2003.12.032","article-title":"An ant colony approach for clustering","volume":"509","author":"Shelokar","year":"2004","journal-title":"Anal. Chim. Acta."},{"key":"ref_55","unstructured":"Faivishevsky, L., and Goldberger, J. (2010, January 21\u201324). A nonparametric information theoretic clustering algorithm. Israel, Israel."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1109\/34.982897","article-title":"Information theoretic clustering","volume":"24","author":"Gokcay","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1162\/NECO_a_00628","article-title":"A nonparametric clustering algorithm with a quantile-based likelihood estimator","volume":"26","author":"Hino","year":"2014","journal-title":"Neural Comput."},{"key":"ref_58","unstructured":"Jenssen, R., Hild, K.E., Erdogmus, D., Principe, J.C., and Eltoft, T. (2003, January 20\u201324). Clustering using Renyi\u2019s entropy. Portland, OR, USA."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"18297","DOI":"10.1073\/pnas.0507432102","article-title":"Information-based clustering","volume":"102","author":"Slonim","year":"2005","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1162\/NECO_a_00534","article-title":"Information-maximization clustering based on squared-loss mutual information","volume":"26","author":"Sugiyama","year":"2014","journal-title":"Neural Comput."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley. [2nd].","DOI":"10.1002\/047174882X"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Cheng, C.-H., Fu, A.W., and Zhang, Y. (1999, January 15\u201318). Entropy-based subspace clustering for mining numerical data. San Diego, CA, USA.","DOI":"10.1145\/312129.312199"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Marques de S\u00e1, J.P. (2001). Pattern Recognition: Concepts, Methods, and Applications, Springer.","DOI":"10.1007\/978-3-642-56651-6"},{"key":"ref_64","unstructured":"Duda, R.O., Hart, P.E., and Stork, D.G. (2000). Pattern Classification, Wiley."},{"key":"ref_65","unstructured":"Haykin, S. (1999). Neural Networks: A Comprehensive Foundation, Prentice Hall. [2nd]."},{"key":"ref_66","unstructured":"Gallager, R.G. (1968). Information Theory and Reliable Communication, Wiley."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1103\/PhysRev.106.620","article-title":"Information theory and statistical mechanics","volume":"106","author":"Jaynes","year":"1957","journal-title":"Phys. Rev."},{"key":"ref_68","unstructured":"Deb, K. (2001). Multi-Objective Optimization Using Evolutionary Algorithms, Wiley."},{"key":"ref_69","unstructured":"Snyman, J. (2005). Practical Mathematical Optimization: An Introduction to Basic Optimization Theory and Classical and New Gradient-Based Algorithms, Springer."},{"key":"ref_70","unstructured":"Thomas, G.B., Finney, R.L., and Weir, M.D. (1988). Calculus and Analytic Geometry, Addison-Wesley."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1007\/BF01442131","article-title":"Pareto optimality in multiobjective problems","volume":"4","author":"Censor","year":"1977","journal-title":"Appl. Math. Optim."},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Sindhya, K., Sinha, A., Deb, K., and Miettinen, K. (2009, January 18\u201321). Local search based evolutionary multi-objective optimization algorithm for constrained and unconstrained problems. Trondheim, Norway.","DOI":"10.1109\/CEC.2009.4983310"},{"key":"ref_73","unstructured":"Zitzler, E., Laumanns, M., and Thiele, L. Available online: http:\/\/www.kddresearch.org\/Courses\/Spring-2007\/CIS830\/Handouts\/P8.pdf."},{"key":"ref_74","first-page":"137","article-title":"On Markov-Type Inequalities","volume":"58","author":"Steliga","year":"2010","journal-title":"Int. J. Pure Appl. Math."},{"key":"ref_75","unstructured":"Casella, G., and Robert, C.P. (1999). Monte Carlo Statistical Methods, Springer."},{"key":"ref_76","unstructured":"Johnson, J.L. (2003). Probability and Statistics for Computer Science, Wiley."},{"key":"ref_77","unstructured":"Abalone Data Set. Available online: http:\/\/archive.ics.uci.edu\/ml\/datasets\/Abalone."},{"key":"ref_78","unstructured":"Cars Data Set. Available online: http:\/\/archive.ics.uci.edu\/ml\/datasets\/Car+Evaluation."},{"key":"ref_79","unstructured":"Census Income Data Set. Available online: http:\/\/archive.ics.uci.edu\/ml\/datasets\/Census+Income."},{"key":"ref_80","unstructured":"Hepatitis Data Set. Available online: http:\/\/archive.ics.uci.edu\/ml\/datasets\/Hepatitis."},{"key":"ref_81","unstructured":"Yeast Data Set. Available online: http:\/\/archive.ics.uci.edu\/ml\/datasets\/Yeast."},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Agresti, A. (2002). Categorical Data Analysis, Wiley.","DOI":"10.1002\/0471249688"},{"key":"ref_83","unstructured":"Shampine, L.F., Allen, R.C., and Pruess, S. (1997). Fundamentals of Numerical Computing, Wiley."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/17\/1\/151\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:41:20Z","timestamp":1760215280000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/17\/1\/151"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1,7]]},"references-count":83,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2015,1]]}},"alternative-id":["e17010151"],"URL":"https:\/\/doi.org\/10.3390\/e17010151","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,1,7]]}}}