{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T11:16:27Z","timestamp":1790766987044,"version":"4.1.0"},"reference-count":32,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T00:00:00Z","timestamp":1642118400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005961","name":"National University of Cuyo","doi-asserted-by":"publisher","award":["Grant 06 \/ C589"],"award-info":[{"award-number":["Grant 06 \/ C589"]}],"id":[{"id":"10.13039\/501100005961","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003074","name":"Agencia Nacional de Promoci\u00f3n Cient\u00edfica y Tecnol\u00f3gica","doi-asserted-by":"publisher","award":["Grants PICT Ra\u00edces 2016 1004 and PICT 2019 2113"],"award-info":[{"award-number":["Grants PICT Ra\u00edces 2016 1004 and PICT 2019 2113"]}],"id":[{"id":"10.13039\/501100003074","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Inferring the value of a property of a large stochastic system is a difficult task when the number of samples is insufficient to reliably estimate the probability distribution. The Bayesian estimator of the property of interest requires the knowledge of the prior distribution, and in many situations, it is not clear which prior should be used. Several estimators have been developed so far in which the proposed prior us individually tailored for each property of interest; such is the case, for example, for the entropy, the amount of mutual information, or the correlation between pairs of variables. In this paper, we propose a general framework to select priors that is valid for arbitrary properties. We first demonstrate that only certain aspects of the prior distribution actually affect the inference process. We then expand the sought prior as a linear combination of a one-dimensional family of indexed priors, each of which is obtained through a maximum entropy approach with constrained mean values of the property under study. In many cases of interest, only one or very few components of the expansion turn out to contribute to the Bayesian estimator, so it is often valid to only keep a single component. The relevant component is selected by the data, so no handcrafted priors are required. We test the performance of this approximation with a few paradigmatic examples and show that it performs well in comparison to the ad-hoc methods previously proposed in the literature. Our method highlights the connection between Bayesian inference and equilibrium statistical mechanics, since the most relevant component of the expansion can be argued to be that with the right temperature.<\/jats:p>","DOI":"10.3390\/e24010125","type":"journal-article","created":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T12:34:04Z","timestamp":1642163644000},"page":"125","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Inferring a Property of a Large System from a Small Number of Samples"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8995-7495","authenticated-orcid":false,"given":"Dami\u00e1n G.","family":"Hern\u00e1ndez","sequence":"first","affiliation":[{"name":"Department of Medical Physics, Centro At\u00f3mico Bariloche and Instituto Balseiro, San Carlos de Bariloche 8400, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5241-3697","authenticated-orcid":false,"given":"In\u00e9s","family":"Samengo","sequence":"additional","affiliation":[{"name":"Department of Medical Physics, Centro At\u00f3mico Bariloche and Instituto Balseiro, San Carlos de Bariloche 8400, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2013","DOI":"10.3390\/e13122013","article-title":"Coincidences and estimation of entropies of random variables with large cardinalities","volume":"13","author":"Nemenman","year":"2011","journal-title":"Entropy"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"066138","DOI":"10.1103\/PhysRevE.69.066138","article-title":"Estimating mutual information","volume":"69","author":"Kraskov","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"056111","DOI":"10.1103\/PhysRevE.69.056111","article-title":"Entropy and information in neural spike trains: Progress on the sampling problem","volume":"69","author":"Nemenman","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_4","first-page":"2833","article-title":"Bayesian entropy estimation for countable discrete distributions","volume":"15","author":"Archer","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1091","DOI":"10.1111\/2041-210X.12108","article-title":"Entropy and the species accumulation curve: A novel entropy estimator via discovery rates of new species","volume":"4","author":"Chao","year":"2013","journal-title":"Methods Ecol. Evol."},{"key":"ref_6","unstructured":"Grassberger, P. (2003). Entropy estimates from insufficient samplings. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1738","DOI":"10.3390\/e15051738","article-title":"Bayesian and quasi-Bayesian estimators for mutual information from discrete data","volume":"15","author":"Archer","year":"2013","journal-title":"Entropy"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hern\u00e1ndez, D.G., and Samengo, I. (2019). Estimating the Mutual Information between Two Discrete, Asymmetric Variables with Limited Samples. Entropy, 21.","DOI":"10.3390\/e21060623"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3913","DOI":"10.1007\/s11071-021-06498-w","article-title":"Mutual information matrix based on asymmetric Shannon entropy for nonlinear interactions of time series","volume":"104","year":"2021","journal-title":"Nonlinear Dyn."},{"key":"ref_10","unstructured":"Lehmann, E.L., and Casella, G. (1998). Theory of Point Estimation, Springer."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_12","unstructured":"Cover, T.M., and Thomas, J.A. (2012). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1103\/PhysRev.108.171","article-title":"Information Theory and Statistical Mechanics","volume":"108","author":"Jaynes","year":"1957","journal-title":"Phys. Rev."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1103\/PhysRev.106.620","article-title":"Information Theory and Statistical Mechanics II","volume":"106","author":"Jaynes","year":"1957","journal-title":"Phys. Rev."},{"key":"ref_15","unstructured":"Amari, S.I., and Hiroshi, N. (2000). Methods of Information Geometry, Oxford University Press."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1701","DOI":"10.1109\/18.930911","article-title":"Information Geometry on Hierarchy of Probability Distributions","volume":"47","author":"Amari","year":"2001","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1080\/0954898X.1996.11978656","article-title":"Analytical estimates of limited sampling biases in different information measures","volume":"7","author":"Panzeri","year":"1996","journal-title":"Netw. Comput. Neural Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"046124","DOI":"10.1103\/PhysRevE.65.046124","article-title":"Estimating probabilities from experimental frequencies","volume":"65","author":"Samengo","year":"2002","journal-title":"Phys. Rev. E"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Abramowitz, M., and Stegun, I.A. (1965). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, Dover Publications.","DOI":"10.1063\/1.3047921"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2913","DOI":"10.1162\/neco.2007.19.11.2913","article-title":"Tight data-robust bounds to mutual information combining shuffling and model selection techniques","volume":"19","author":"Montemurro","year":"2007","journal-title":"Neural Comput."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kolchinsky, A., and Tracey, B.D. (2017). Estimating mixture entropy with pairwise distances. Entropy, 19.","DOI":"10.3390\/e19070361"},{"key":"ref_23","unstructured":"Belghazi, M.I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. (2018, January 10\u201315). Mutual information neural estimation. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"053302","DOI":"10.1103\/PhysRevE.98.053302","article-title":"Information estimation using nonparametric copulas","volume":"98","author":"Safaai","year":"2018","journal-title":"Phys. Rev. E"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"022404","DOI":"10.1103\/PhysRevE.100.022404","article-title":"Estimation of mutual information for real-valued data with error bars and controlled bias","volume":"100","author":"Holmes","year":"2019","journal-title":"Phys. Rev. E"},{"key":"ref_26","unstructured":"Miller, G. (1955). Note on the bias of information estimates. Information Theory in Psychology: Problems and Methods, Free Press."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1023\/A:1026096204727","article-title":"Nonparametric estimation of Shannon\u2019s index of diversity when there are unseen species in sample","volume":"10","author":"Chao","year":"2003","journal-title":"Environ. Ecol. Stat."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"P03015","DOI":"10.1088\/1742-5468\/2013\/03\/P03015","article-title":"A simple method for estimating the entropy of neural activity","volume":"2013","author":"Dubuis","year":"2013","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_29","unstructured":"MacKay, D. (2003). Information Theory, Inference and Learning Algorithms, Cambridge University Press."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1080\/00018732.2016.1211393","article-title":"Statistical physics of inference: Thresholds and algorithms","volume":"65","author":"Krzakala","year":"2016","journal-title":"Adv. Phys."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood from incomplete data via the EM algorithm","volume":"39","author":"Dempster","year":"1977","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"158101","DOI":"10.1103\/PhysRevLett.100.158101","article-title":"Symbolic transfer entropy","volume":"100","author":"Staniek","year":"2008","journal-title":"Phys. Rev. Lett."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/1\/125\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:27:19Z","timestamp":1760362039000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/1\/125"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,14]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["e24010125"],"URL":"https:\/\/doi.org\/10.3390\/e24010125","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,14]]}}}