{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T16:59:54Z","timestamp":1784134794158,"version":"3.55.0"},"reference-count":43,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2017,7,14]],"date-time":"2017-07-14T00:00:00Z","timestamp":1499990400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"AFOSR MURI","award":["FA9550-15-1-0038"],"award-info":[{"award-number":["FA9550-15-1-0038"]}]}],"content-domain":{"domain":["www.mdpi.com"],"crossmark-restriction":true},"short-container-title":["Entropy"],"abstract":"<jats:p>Mixture distributions arise in many parametric and non-parametric settings\u2014for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this quantity has no closed-form expression, making some form of approximation necessary. We propose a family of estimators based on a pairwise distance function between mixture components, and show that this estimator class has many attractive properties. For many distributions of interest, the proposed estimators are efficient to compute, differentiable in the mixture parameters, and become exact when the mixture components are clustered. We prove this family includes lower and upper bounds on the mixture entropy. The Chernoff    \u03b1   -divergence gives a lower bound when chosen as the distance function, with the Bhattacharyaa distance providing the tightest lower bound for components that are symmetric and members of a location family. The Kullback\u2013Leibler divergence gives an upper bound when used as the distance function. We provide closed-form expressions of these bounds for mixtures of Gaussians, and discuss their applications to the estimation of mutual information. We then demonstrate that our bounds are significantly tighter than well-known existing bounds using numeric simulations. This estimator class is very useful in optimization problems involving maximization\/minimization of entropy and mutual information, such as MaxEnt and rate distortion problems.<\/jats:p>","DOI":"10.3390\/e19070361","type":"journal-article","created":{"date-parts":[[2017,7,14]],"date-time":"2017-07-14T10:45:02Z","timestamp":1500029102000},"page":"361","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":80,"title":["Estimating Mixture Entropy with Pairwise Distances"],"prefix":"10.3390","volume":"19","author":[{"given":"Artemy","family":"Kolchinsky","sequence":"first","affiliation":[{"name":"Santa Fe Institute, Santa Fe, NM 87501, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brendan","family":"Tracey","sequence":"additional","affiliation":[{"name":"Santa Fe Institute, Santa Fe, NM 87501, USA"},{"name":"Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2017,7,14]]},"reference":[{"key":"ref_1","unstructured":"McLachlan, G., and Peel, D. (2004). Finite Mixture Models, John Wiley & Sons."},{"key":"ref_2","unstructured":"Cover, T.M., and Thomas, J.A. (2012). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Goldberger, J., Gordon, S., and Greenspan, H. (2003, January 13\u201316). An Efficient Image Similarity Measure Based on Approximations of KL-Divergence Between Two Gaussian Mixtures. Proceedings of the 9th International Conference on Computer Vision, Nice, France.","DOI":"10.1109\/ICCV.2003.1238387"},{"key":"ref_4","unstructured":"Viola, P., Schraudolph, N.N., and Sejnowski, T.J. (1996). Empirical Entropy Manipulation for Real-World Problems. Advances in Neural Information Processing Systems, The MIT Press."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Hershey, J.R., and Olsen, P.A. (2007, January 15\u201320). Approximating the Kullback Leibler divergence between Gaussian mixture models. Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing, Honolulu, HI, USA.","DOI":"10.1109\/ICASSP.2007.366913"},{"key":"ref_6","unstructured":"Chen, J.Y., Hershey, J.R., Olsen, P.A., and Yashchin, E. (April, January 31). Accelerated monte carlo for kullback-leibler divergence between gaussian mixture models. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, NV, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Capristan, F.M., and Alonso, J.J. (2014, January 13\u201317). Range Safety Assessment Tool (RSAT): An analysis environment for safety assessment of launch and reentry vehicles. Proceedings of the 52nd Aerospace Sciences Meeting, National Harbor, MD, USA.","DOI":"10.2514\/6.2014-0304"},{"key":"ref_8","unstructured":"Schraudolph, N.N. (1995). Optimization of Entropy with Neural Networks. [Ph.D. Thesis, University of California]."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1109\/TNN.2004.828766","article-title":"Gradient-based manipulation of nonparametric entropy estimates","volume":"15","author":"Schraudolph","year":"2004","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1016\/j.sigpro.2004.11.022","article-title":"Fast kernel entropy estimation and optimization","volume":"85","author":"Shwartz","year":"2005","journal-title":"Signal Process."},{"key":"ref_11","unstructured":"Kolchinsky, A., Tracey, B.D., and Wolpert, D.H. (arXiv, 2017). Nonlinear Information Bottleneck, arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Contreras-Reyes, J.E., and Cort\u00e9s, D.D. (2016). Bounds on R\u00e9nyi and Shannon Entropies for Finite Mixtures of Multivariate Skew-Normal Distributions: Application to Swordfish (Xiphias gladius Linnaeus). Entropy, 18.","DOI":"10.3390\/e18110382"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1318","DOI":"10.1109\/34.888716","article-title":"Mode-finding for mixtures of Gaussian distributions","volume":"22","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1214\/14-EJS887","article-title":"Variational Bayesian inference with Gaussian-mixture approximations","volume":"8","author":"Zobay","year":"2014","journal-title":"Electron. J. Stat."},{"key":"ref_15","first-page":"17","article-title":"Nonparametric entropy estimation: An overview","volume":"6","author":"Beirlant","year":"1997","journal-title":"Int. J. Math. Stat. Sci."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1007\/BF00057735","article-title":"Estimation of entropy and other functionals of a multivariate density","volume":"41","author":"Joe","year":"1989","journal-title":"Ann. Inst. Stat. Math."},{"key":"ref_17","unstructured":"Nair, C., Prabhakar, B., and Shah, D. (arXiv, 2006). On entropy for mixtures of discrete and continuous variables, arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Huber, M.F., Bailey, T., Durrant-Whyte, H., and Hanebeck, U.D. (2008, January 20\u201322). On entropy approximation for Gaussian mixture random vectors. Proceedings of the IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Seoul, Korea.","DOI":"10.1109\/MFI.2008.4648062"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/BF00773669","article-title":"On the estimation of entropy","volume":"45","author":"Hall","year":"1993","journal-title":"Ann. Inst. Stat. Math."},{"key":"ref_20","first-page":"265","article-title":"Information theoretic learning","volume":"1","author":"Principe","year":"2000","journal-title":"Unsuperv. Adapt. Filter"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"5891","DOI":"10.1109\/TSP.2008.2005085","article-title":"A reproducing kernel Hilbert space framework for information-theoretic learning","volume":"56","author":"Xu","year":"2008","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Jebara, T., and Kondor, R. (2003). Bhattacharyya and expected likelihood kernels. Learning Theory and Kernel Machines, Springer.","DOI":"10.1007\/978-3-540-45167-9_6"},{"key":"ref_23","first-page":"819","article-title":"Probability product kernels","volume":"5","author":"Jebara","year":"2004","journal-title":"J. Mach. Learn. Res."},{"key":"ref_24","first-page":"1705","article-title":"Clustering with Bregman divergences","volume":"6","author":"Banerjee","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Cichocki, A., Zdunek, R., Phan, A.H., and Amari, S.I. (2009). Similarity Measures and Generalized Divergences. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation, JohnWiley & Sons.","DOI":"10.1002\/9780470747278"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3390\/e12061532","article-title":"Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities","volume":"12","author":"Cichocki","year":"2010","journal-title":"Entropy"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1016\/j.ins.2013.06.018","article-title":"R\u00e9nyi divergence measures for commonly used univariate continuous distributions","volume":"249","author":"Gil","year":"2013","journal-title":"Inf. Sci."},{"key":"ref_28","unstructured":"Crooks, G.E. (2017, July 12). On Measures of Entropy and Information. Available online: http:\/\/threeplusone.com\/on_information.pdf."},{"key":"ref_29","unstructured":"Nielsen, F. (arXiv, 2011). Chernoff information of exponential families, arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"3797","DOI":"10.1109\/TIT.2014.2320500","article-title":"R\u00e9nyi divergence and Kullback-Leibler divergence","volume":"60","author":"Harremos","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2451","DOI":"10.1214\/aos\/1030741081","article-title":"Mutual information, metric entropy and cumulative relative entropy risk","volume":"25","author":"Haussler","year":"1997","journal-title":"Ann. Stat."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Fukunaga, K. (1990). Introduction to Statistical Pattern Recognition, Academic Press. [2nd ed.].","DOI":"10.1016\/B978-0-08-047865-4.50007-7"},{"key":"ref_33","unstructured":"Paisley, J. (2010). Two Useful Bounds for Variational Inference, Princeton University."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5973","DOI":"10.1109\/TIT.2016.2603151","article-title":"f-Divergence Inequalities","volume":"62","author":"Sason","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_35","unstructured":"Hero, A.O., Ma, B., Michel, O., and Gorman, J. (2017, July 14). Alpha-Divergence for Classification, Indexing and Retrieval. Available online: https:\/\/pdfs.semanticscholar.org\/6d51\/fbf90c59c2bb8cbf0cb609a224f53d1b68fb.pdf."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"450","DOI":"10.1016\/0047-259X(82)90077-X","article-title":"The Fr\u00e9chet distance between multivariate normal distributions","volume":"12","author":"Dowson","year":"1982","journal-title":"J. Multivar. Anal."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1016\/0024-3795(82)90112-4","article-title":"The distance between two random vectors with given dispersion matrices","volume":"48","author":"Olkin","year":"1982","journal-title":"Linear Algebra Appl."},{"key":"ref_38","unstructured":"Pardo, L. (2005). Statistical Inference Based on Divergence Measures, CRC Press."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1016\/j.stamet.2009.03.001","article-title":"R\u00e9nyi statistics for testing equality of autocorrelation coefficients","volume":"6","author":"Hobza","year":"2009","journal-title":"Stat. Methodol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.patrec.2014.01.002","article-title":"Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means","volume":"42","author":"Nielsen","year":"2014","journal-title":"Pattern Recognit. Lett."},{"key":"ref_41","unstructured":"(2017, July 14). GitHub. Available online: https:\/\/www.github.com\/btracey\/mixent."},{"key":"ref_42","unstructured":"(2017, July 14). Gonum Numeric Library. Available online: https:\/\/www.gonum.org."},{"key":"ref_43","unstructured":"Tishby, N., Pereira, F., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA."}],"updated-by":[{"DOI":"10.3390\/e19110588","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2017,7,14]],"date-time":"2017-07-14T00:00:00Z","timestamp":1499990400000}}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/7\/361\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,3]],"date-time":"2025-08-03T23:09:30Z","timestamp":1754262570000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/7\/361"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,7,14]]},"references-count":43,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2017,7]]}},"alternative-id":["e19070361"],"URL":"https:\/\/doi.org\/10.3390\/e19070361","relation":{"correction":[{"id-type":"doi","id":"10.3390\/e19110588","asserted-by":"object"}]},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,7,14]]}}}