{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T11:21:40Z","timestamp":1775128900907,"version":"3.50.1"},"reference-count":30,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2010,4,9]],"date-time":"2010-04-09T00:00:00Z","timestamp":1270771200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Given iid samples drawn from a distribution with known parametric form, we propose the minimization of expected Bregman divergence to form Bayesian estimates of differential entropy and relative entropy, and derive such estimators for the uniform, Gaussian, Wishart, and inverse Wishart distributions. Additionally, formulas are given for a log gamma Bregman divergence and the differential entropy and relative entropy for the Wishart and inverse Wishart. The results, as always with Bayesian estimates, depend on the accuracy of the prior parameters, but example simulations show that the performance can be substantially improved compared to maximum likelihood or state-of-the-art nonparametric estimators.<\/jats:p>","DOI":"10.3390\/e12040818","type":"journal-article","created":{"date-parts":[[2010,4,12]],"date-time":"2010-04-12T11:23:10Z","timestamp":1271071390000},"page":"818-843","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Parametric Bayesian Estimation of Differential Entropy and Relative Entropy"],"prefix":"10.3390","volume":"12","author":[{"given":"Maya","family":"Gupta","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, University of Washington, Seattle WA 98195-2500, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Santosh","family":"Srivastava","sequence":"additional","affiliation":[{"name":"Computational Biology, Fred Hutchinson Cancer Research Center, Seattle WA 98109, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2010,4,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"895","DOI":"10.1109\/TIM.2006.887174","article-title":"A novel biometric system for identification and verification of haptic users","volume":"56","author":"Orozco","year":"2007","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_2","unstructured":"Choi, H. (2009). Adaptive Sampling and Forecasting with Mobile Sensor Networks\u00a0. [PhD Dissertation, MIT]."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/0165-1684(89)90132-1","article-title":"On estimation of entropy and mutual information of continuous distributions","volume":"16","author":"Moddemeijer","year":"1989","journal-title":"Signal Process."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.jmva.2003.10.003","article-title":"Estimation of the entropy of a multivariate normal distribution","volume":"92","author":"Misra","year":"2005","journal-title":"J. Multivariate Anal."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2392","DOI":"10.1109\/TIT.2009.2016060","article-title":"Divergence estimation for multi-dimensional densities via k nearest-neighbor distances","volume":"55","author":"Wang","year":"2009","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Ahmed, N.A., and Gokhale, D.V. (1989). Entropy expressions and their estimators for multivariate distributions. IEEE Trans. Inform. Theory, 688\u2013692.","DOI":"10.1109\/18.30996"},{"key":"ref_7","first-page":"17","article-title":"Nonparametric entropy estimation: An overview","volume":"6","author":"Beirlant","year":"1997","journal-title":"Intl. J. Math. Stat. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2330","DOI":"10.1109\/TIT.2007.899533","article-title":"On the estimation of differential entropy from data located on embedded manifolds","volume":"53","author":"Nilsson","year":"2007","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_9","first-page":"95","article-title":"Sample estimate of entropy of a random vector","volume":"23","author":"Kozachenko","year":"1987","journal-title":"Probl. Inform. Transm."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1080\/104852504200026815","article-title":"A new class of random vector entropy estimators and its applications in testing statistical hypotheses","volume":"17","author":"Goria","year":"2005","journal-title":"J. Nonparametric Stat."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"261","DOI":"10.3103\/S106653070803006X","article-title":"kn-Nearest neighbor estimators of entropy","volume":"17","author":"Mnatsakanov","year":"2008","journal-title":"Math. Method. Stat."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1921","DOI":"10.1109\/18.782114","article-title":"Asymptotic theory of greedy approximations to minimal k-point random graphs","volume":"45","author":"Hero","year":"1999","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1109\/MSP.2002.1028355","article-title":"Applications of entropic spanning graphs","volume":"19","author":"Hero","year":"2002","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2210","DOI":"10.1109\/TSP.2004.831130","article-title":"Geodesic entropic graphs for dimension and entropy estimation in manifold learning","volume":"52","author":"Costa","year":"2004","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1903","DOI":"10.1162\/0899766054323026","article-title":"Edgeworth approximation of multivariate differential entropy","volume":"17","author":"Hulle","year":"2005","journal-title":"Neural Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3064","DOI":"10.1109\/TIT.2005.853314","article-title":"Divergence estimation of continuous distributions based on data-dependent partitions","volume":"51","author":"Wang","year":"2005","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_17","unstructured":"Nguyen, X., Wainwright, M.J., and Jordan, M.I. (2007). Estimating divergence functional and the likelihood ratio by penalized convex risk minimization. Advances Neural Inform. Process. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, Q., Kulkarni, S.R., and Verd\u00fa, S. (2006, January 9\u201314). A nearest-neighbor approach to estimating divergence between continuous random vectors. Proceedings of the 2006 IEEE International Symposium on Information Theory, Seattle, WA, USA.","DOI":"10.1109\/ISIT.2006.261842"},{"key":"ref_19","unstructured":"P\u00e9rez-Cruz, F. (2009). Estimation of information-theoretic measures for continuous random variables. Adv. Neural Inform. Process. Syst. (NIPS)."},{"key":"ref_20","unstructured":"Lehmann, E.L., and Casella, G. (1998). Theory of Point Estimation, Springer. Chapter 4."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/0041-5553(67)90040-7","article-title":"The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming","volume":"7","author":"Bregman","year":"1967","journal-title":"USSR Comput. Math. Math. Phys."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2664","DOI":"10.1109\/TIT.2005.850145","article-title":"On the optimality of conditional expectation as a Bregman predictor","volume":"51","author":"Banerjee","year":"2005","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/18.50370","article-title":"General entropy criteria for inverse problems, with applications to data compression, pattern classification, and cluster analysis","volume":"36","author":"Jones","year":"1990","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"5130","DOI":"10.1109\/TIT.2008.929943","article-title":"Functional Bregman divergence and Bayesian estimation of distributions","volume":"54","author":"Frigyik","year":"2008","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_25","unstructured":"Amari, S., and Nagaoka, H. (2000). Methods of Information Geometry, Oxford University Press."},{"key":"ref_26","first-page":"188","article-title":"The geometry of asymptotic inference","volume":"4","author":"Kass","year":"1989","journal-title":"Stat. Sci."},{"key":"ref_27","first-page":"1287","article-title":"Bayesian quadratic discriminant analysis","volume":"8","author":"Srivastava","year":"2007","journal-title":"J. Mach. Learn. Res."},{"key":"ref_28","unstructured":"Havil, J. (2003). Gamma, Princeton University Press."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1687","DOI":"10.1109\/78.845926","article-title":"Estimating the entropy of a signal with applications","volume":"48","author":"Bercher","year":"2000","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_30","unstructured":"Bilodeau, M., and Brenner, D. (1999). Theory of Multivariate Statistics, Springer Texts in Statistics."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/12\/4\/818\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T22:02:05Z","timestamp":1760220125000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/12\/4\/818"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,4,9]]},"references-count":30,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2010,4]]}},"alternative-id":["e12040818"],"URL":"https:\/\/doi.org\/10.3390\/e12040818","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,4,9]]}}}