{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T01:35:22Z","timestamp":1775957722451,"version":"3.50.1"},"reference-count":29,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2013,10,31]],"date-time":"2013-10-31T00:00:00Z","timestamp":1383177600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We consider Bayesian estimation of information-theoretic quantities from data,  using a Dirichlet prior. Acknowledging the uncertainty of the event space size m and  the Dirichlet prior\u2019s concentration parameter c, we treat both as random variables set  by a hyperprior. We show that the associated hyperprior, P(c, m), obeys a simple  \u201cIrrelevance of Unseen Variables\u201d (IUV) desideratum iff P(c, m) = P(c)P(m). Thus,  requiring IUV greatly reduces the number of degrees of freedom of the hyperprior. Some  information-theoretic quantities can be expressed multiple ways, in terms of different event  spaces, e.g., mutual information. With all hyperpriors (implicitly) used in earlier work,  different choices of this event space lead to different posterior expected values of these  information-theoretic quantities. We show that there is no such dependence on the choice  of event space for a hyperprior that obeys IUV. We also derive a result that allows us to  exploit IUV to greatly simplify calculations, like the posterior expected mutual information  or posterior expected multi-information. We also use computer experiments to favorably  compare an IUV-based estimator of entropy to three alternative methods in common use. We  end by discussing how seemingly innocuous changes to the formalization of an estimation  problem can substantially affect the resultant estimates of posterior expectations.<\/jats:p>","DOI":"10.3390\/e15114668","type":"journal-article","created":{"date-parts":[[2013,10,31]],"date-time":"2013-10-31T11:49:09Z","timestamp":1383220149000},"page":"4668-4699","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Estimating Functions of Distributions Defined over Spaces of Unknown Size"],"prefix":"10.3390","volume":"15","author":[{"given":"David","family":"Wolpert","sequence":"first","affiliation":[{"name":"Santa Fe Institute, 1399 Hyde Park Rd., Santa Fe, NM 87501, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Simon","family":"DeDeo","sequence":"additional","affiliation":[{"name":"Santa Fe Institute, 1399 Hyde Park Rd., Santa Fe, NM 87501, USA"},{"name":"School of Informatics and Computing, Indiana University, 901 E 10th St, Bloomington, IN 47408, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2013,10,31]]},"reference":[{"key":"ref_1","unstructured":"Cover, T., and Thomas, J. (1991). Elements of Information Theory, Wiley-Interscience."},{"key":"ref_2","unstructured":"Mackay, D. (2003). Information Theory, Inference, and Learning Algorithms, Cambridge University Press."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_4","unstructured":"Grassberger, P. (2003). Entropy estimates from insufficient samplings."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"7176","DOI":"10.1073\/pnas.90.15.7176","article-title":"Covariation of mutations in the V3 loop of human immunodeficiency virus type 1 envelope protein: An information theoretic analysis","volume":"90","author":"Korber","year":"1993","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"6841","DOI":"10.1103\/PhysRevE.52.6841","article-title":"Estimating functions of probability distributions from a finite set of samples","volume":"52","author":"Wolpert","year":"1995","journal-title":"Phys. Rev. E"},{"key":"ref_7","unstructured":"Wolf, D.R., Wolf, D.R., and Wolpert, D.H. (1994). Estimating functions of probability distributions from a finite set of samples, Part II: Bayes Estimators for Mutual Information, Chi-Squared, Covariance, and other Statistics."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"6973","DOI":"10.1103\/PhysRevE.54.6973.2","article-title":"Erratum: Estimating functions of probability distributions from a finite set of samples","volume":"54","author":"Wolpert","year":"1996","journal-title":"Phys. Rev. E"},{"key":"ref_9","first-page":"399","article-title":"Distribution of mutual information","volume":"1","author":"Hutter","year":"2002","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"ref_10","unstructured":"Hurley, M., and Kao, E. (2013). Massachusetts Institute of Technology. Available online: http:\/\/www.dtic.mil\/dtic\/tr\/fulltext\/u2\/a580524.pdf."},{"key":"ref_11","unstructured":"Archer, E., Park, I., and Pillow, J. (2012, January 3\u20136). Bayesian estimation of discrete entropy with mixtures of stick-breaking priors. Advances in Neural Information Processing Systems 25, Proceedings of the 26th Annual Conference on Neural Information Processing Systems 2012, Lake Tahoe, NV, USA."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Dietterich, T. (2003). Advances in Neural Information Processing System, MIT Press.","DOI":"10.7551\/mitpress\/1120.001.0001"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e056111","DOI":"10.1103\/PhysRevE.69.056111","article-title":"Entropy and information in neural spike trains: Progress on the sampling problem","volume":"69","author":"Nemenman","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"e1000025","DOI":"10.1371\/journal.pcbi.1000025","article-title":"Neural coding of natural stimuli: information at sub-millisecond resolution","volume":"4","author":"Nemenman","year":"2008","journal-title":"PLoS Comput. Biol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1738","DOI":"10.3390\/e15051738","article-title":"Bayesian and quasi-Bayesian estimators for mutual information from discrete data","volume":"15","author":"Archer","year":"2013","journal-title":"Entropy"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4039","DOI":"10.1002\/sim.2942","article-title":"Coverage-adjusted entropy estimation","volume":"26","author":"Vu","year":"2007","journal-title":"Stat. Med."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Jaynes, E.T., and Bretthorst, G.L. (2003). Probability Theory : The Logic of Science, Cambridge University Press.","DOI":"10.1017\/CBO9780511790423"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"James, R.G., Ellison, C.J., and Crutchfield, J.P. (2011). Anatomy of a bit: Information in a time series observation.","DOI":"10.1063\/1.3637494"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2013","DOI":"10.3390\/e13122013","article-title":"Coincidences and estimation of entropies of random variables with large cardinalities","volume":"13","author":"Nemenman","year":"2011","journal-title":"Entropy"},{"key":"ref_20","unstructured":"Wolpert, D.H. (1996). Maximum Entropy and Bayesian Methods, Kluwer Academic Publishers."},{"key":"ref_21","unstructured":"Wolpert, D.H. (1995). The Mathematics of Generalization, Addison-Wesley."},{"key":"ref_22","unstructured":"Note that no particular property is required of the relation among multiple unseen random variables that may (or may not) exist."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1080\/01621459.1993.10594330","article-title":"Estimating the number of species: A review","volume":"88","author":"Bunge","year":"1993","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_24","unstructured":"The proof of this proposition uses moment-generating functions with Fourier decompositions of the prior, \u03c0. To ensure we do not divide by zero, we have to introduce the constant 1 + \u03f5 into that proof, and to ensure the convergence of our resultant Taylor decompositions, we have to assume infinite differentiability. This is the reason for the peculiar technical condition."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Berger, J.M. (1985). Statistical Decision theory and Bayesian Analysis, Springer-Verlag.","DOI":"10.1007\/978-1-4757-4286-2"},{"key":"ref_26","unstructured":"The proof of this proposition uses moment-generating functions with Fourier decompositions of the prior, \u03c0. To ensure we do not divide by zero, we have to introduce the constant 1 + \u03f5 into that proof, and to ensure the convergence of our resultant Taylor decompositions, we have to assume infinite differentiability. This is the reason for the peculiar technical condition."},{"key":"ref_27","unstructured":"Intuitively, for that P(c), the likelihood says that a dataset, {125, 125,\u2026}, would imply a relatively high value of c and, therefore, a high probability that \u03c1 is close to uniform over all bins. Given this, it also implies a low value of |Z|, since if there were any more bins than the eight that have counts, we almost definitely would have seen them seen them for almost-uniform \u03c1. Conversely, for the dataset, {691, \u2026, 6}, the implication is that c must be low where some rare bins trail out, and as a result, there might be a few more bins who had zero counts."},{"key":"ref_28","unstructured":"Of course, since the formulas in WW implicitly assume c = m, care must be taken to insert appropriate pseudo-counts into those formulas if we want to use a value, c, that differs from m."},{"key":"ref_29","unstructured":"Wolpert, D., and Wolf, D. (1994). Estimating functions of probability distributions from a finite set of samples, Part 1: Bayes Estimators and the Shannon Entropy."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/11\/4668\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:50:18Z","timestamp":1760219418000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/11\/4668"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,10,31]]},"references-count":29,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2013,11]]}},"alternative-id":["e15114668"],"URL":"https:\/\/doi.org\/10.3390\/e15114668","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,10,31]]}}}