{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:30:55Z","timestamp":1760243455373,"version":"build-2065373602"},"reference-count":42,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2013,2,25]],"date-time":"2013-02-25T00:00:00Z","timestamp":1361750400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The Minimum Mutual Information (MinMI) Principle provides the least committed, maximum-joint-entropy (ME) inferential law that is compatible with prescribed marginal distributions and empirical cross constraints. Here, we estimate MI bounds (the MinMI values) generated by constraining sets Tcr comprehended by mcr linear and\/or nonlinear joint expectations, computed from samples of N iid outcomes. Marginals (and their entropy) are imposed by single morphisms of the original random variables.  N-asymptotic formulas are given both for the distribution of cross expectation\u2019s estimation errors, the MinMI estimation bias, its variance and distribution. A growing Tcr leads to an increasing MinMI, converging eventually to the total MI. Under N-sized samples, the MinMI increment relative to two encapsulated sets Tcr1 \u2282 Tcr2 (with numbers of constraints mcr1&lt;mcr2 ) is the test-difference \u03b4H = Hmax 1, N - Hmax 2, N \u2265 0    between the two respective estimated MEs. Asymptotically, \u03b4H follows a Chi-Squared distribution   1\/2N\u03a72 (mcr2-mcr1) whose upper quantiles determine if constraints in Tcr2\/Tcr1 explain significant extra MI. As an example, we have set marginals to being normally distributed (Gaussian) and have built a sequence of MI bounds, associated to successive non-linear correlations due to joint  non-Gaussianity. Noting that in real-world situations available sample sizes can be rather low, the relationship between MinMI bias, probability density over-fitting and outliers is put in evidence for under-sampled data.<\/jats:p>","DOI":"10.3390\/e15030721","type":"journal-article","created":{"date-parts":[[2013,2,25]],"date-time":"2013-02-25T10:53:32Z","timestamp":1361789612000},"page":"721-752","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Minimum Mutual Information and Non-Gaussianity through the Maximum Entropy Method: Estimation from Finite Samples"],"prefix":"10.3390","volume":"15","author":[{"given":"Carlos","family":"Pires","sequence":"first","affiliation":[{"name":"Instituto Dom Luiz (IDL), University of Lisbon (UL), Lisbon, P-1749-016, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5543-1754","authenticated-orcid":false,"given":"Rui","family":"Perdig\u00e3o","sequence":"additional","affiliation":[{"name":"Institute of Hydraulic Engineering and Water Resources Management, Vienna University of Technology, Vienna, A-1040, Austria"}]}],"member":"1968","published-online":{"date-parts":[[2013,2,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"The mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_2","unstructured":"Cover, T.M., and Thomas, J.A. (1991). Elements of Information Theory, John Wiley & Sons, Inc."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1038\/nrn1888","article-title":"Neural correlations, population coding and computation","volume":"7","author":"Averbeck","year":"2006","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_4","unstructured":"Goldie, C.M., and Pinch, R.G.E. (1991). London Mathematical Society Student Texts (No. 20), Cambridge University Press."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1257\/000282806777212431","article-title":"Rational Inattention: Beyond the Linear-Quadratic Case","volume":"96","author":"Sims","year":"2006","journal-title":"Am. Econ. Rev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1765","DOI":"10.3390\/e12071765","article-title":"Entropy and Information Approaches to Genetic Diversity and its Expression: Genomic Geography","volume":"12","author":"Sherwin","year":"2010","journal-title":"Entropy"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1348\/000712606X122760","article-title":"Characterizing linguistic structure with mutual information","volume":"98","author":"Pothos","year":"2007","journal-title":"Br. J. Psychol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1175\/MWR3407.1","article-title":"Non-Gaussianity and asymmetry of the winter monthly precipitation estimation from the NAO","volume":"135","author":"Pires","year":"2007","journal-title":"Mon. Wea. Rev."},{"key":"ref_9","unstructured":"Globerson, A., and Tishby, N. (2004, January 7\u201311). The minimum information principle for discriminative learning. Proceedings of the 20th conference on Uncertainty in artificial intelligence, Banff, Canada."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3490","DOI":"10.1073\/pnas.0806782106","article-title":"The minimum information principle and its application to neural code analysis","volume":"106","author":"Globerson","year":"2009","journal-title":"Proc. Natl. Accd. Sci. USA"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"010101(R):1","DOI":"10.1103\/PhysRevE.83.010101","article-title":"Grassberger, P. Lower bounds on mutual information","volume":"83","author":"Foster","year":"2011","journal-title":"Phys. Rev. E"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1103","DOI":"10.3390\/e14061103","article-title":"Minimum Mutual Information and Non-Gaussianity Through the Maximum Entropy Method: Theory and Properties","volume":"14","author":"Pires","year":"2012","journal-title":"Entropy"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1007\/978-3-642-01307-2_36","article-title":"Estimation of mutual information: A survey","volume":"5589","author":"Li","year":"2009","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"026209:1","DOI":"10.1103\/PhysRevE.76.026209","article-title":"Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data","volume":"76","author":"Khan","year":"2007","journal-title":"Phys. Rev. E"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1080\/0954898X.1996.11978656","article-title":"Analytical estimates of limited sampling biases in different information measures","volume":"7","author":"Panzeri","year":"1996","journal-title":"Comp. Neur. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2797","DOI":"10.1162\/089976600300014728","article-title":"Asymptotic Bias in Information Estimates and the Exponential (Bell) Polynomials","volume":"12","author":"Victor","year":"2000","journal-title":"Neur. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1064","DOI":"10.1152\/jn.00559.2007","article-title":"Train Information Measures Correcting for the Sampling Bias Problem in Spike Information Measures","volume":"98","author":"Panzeri","year":"2007","journal-title":"J. Neurophysiol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1103\/PhysRevLett.80.197","article-title":"Entropy and information in neural spike trains","volume":"86","author":"Strong","year":"1998","journal-title":"Phys. Rev. Lett."},{"key":"ref_20","unstructured":"Quastler, H. (1955). Information Theory in Psycholog, II-B Free Press."},{"key":"ref_21","unstructured":"Grassberger, P. (2008). Entropy Estimates from Insufficient Samplings."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"202001","DOI":"10.1088\/1751-8113\/41\/20\/202001","article-title":"Entropy estimates of small data sets","volume":"41","author":"Bonachela","year":"2008","journal-title":"J. Phys. A"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Nelsen, R.B. (1999). An Introduction to Copulas, Springer.","DOI":"10.1007\/978-1-4757-3076-0"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"68003","DOI":"10.1209\/0295-5075\/88\/68003","article-title":"An information-theoretic approach to statistical dependence: Copula information","volume":"88","author":"Calsaverini","year":"2009","journal-title":"Europhys. Lett."},{"key":"ref_25","unstructured":"Ma, J., and Sun, Z. (2008). Mutual information is copula entropy."},{"key":"ref_26","first-page":"2034","article-title":"How biased are maximum entropy models?","volume":"24","author":"Macke","year":"2011","journal-title":"Adv. Neur. Inf. Proc. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1016\/j.csda.2004.03.010","article-title":"Distribution of mutual information from complete and incomplete data","volume":"48","author":"Hutter","year":"2005","journal-title":"Comput. Stat. Data An."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1109\/PROC.1982.12425","article-title":"On the Rationale of Maximum-entropy methods","volume":"70","author":"Jaynes","year":"1982","journal-title":"P. IEEE"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/TIT.1980.1056144","article-title":"Axiomatic derivation of the principle of maximum entropy and the principle of the minimum cross-entropy","volume":"26","author":"Shore","year":"1980","journal-title":"IEEE Trans. Inform. Theor."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1111\/j.1751-5823.2010.00105.x","article-title":"Information Measures in Perspective","volume":"78","author":"Ebrahimi","year":"2010","journal-title":"Int. Stat. Rev."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Wackernagel, H. (1995). Multivariate Geostatistics\u2014An Introduction with Applications, Springer Verlag.","DOI":"10.1007\/978-3-662-03098-1"},{"key":"ref_32","unstructured":"Rank, J. (2007). Copulas: From Theory to Application in Finance, Risk Publications. Section 2."},{"key":"ref_33","first-page":"429","article-title":"On Covariance in Finite Population Sampling","volume":"34","author":"Tam","year":"1985","journal-title":"J. Roy. Stat. Soc. D-Sta."},{"key":"ref_34","unstructured":"Van det Vaart, A.W. (1998). Cambridge University Press."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/S0304-4076(01)00092-6","article-title":"Entropy densities with an application to autoregressive conditional skewness and kurtosis","volume":"106","author":"Rockinger","year":"2002","journal-title":"J. Econometrics"},{"key":"ref_36","unstructured":"Bates, D. Quadratic Forms of Random Variables. STAT 849 lectures. Available online: http:\/\/www.stat.wisc.edu\/~st849\u20131\/lectures\/Ch02.pdf."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Goebel, B., Dawy, Z., Hagenauer, J., and Mueller, J.C. (2005, January 16\u201320). An approximation to the distribution of finite sample size mutual information estimates. 2005. Proceedings of IEEE International Conference on Communications (ICC\u2019 05), Seoul, Korea.","DOI":"10.1109\/ICC.2005.1494518"},{"key":"ref_38","first-page":"3","article-title":"On the \u201cprobable error\u201d of a coefficient of correlation deduced from a small sample","volume":"1","author":"Fisher","year":"1921","journal-title":"Metron"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"318","DOI":"10.3758\/BF03193163","article-title":"Applying the bootstrap to the multivariate case: bootstrap component\/factor analysis","volume":"39","author":"Zientek","year":"2007","journal-title":"Behav. Res. Methods"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"262","DOI":"10.2307\/2346576","article-title":"Algorithm AS 84: Measures of multivariate skewness and kurtosis","volume":"24","author":"Mardia","year":"1975","journal-title":"Appl. Stat."},{"key":"ref_41","first-page":"291","article-title":"The North Atlantic Oscillation","volume":"26","author":"Hurrell","year":"2001","journal-title":"Science"},{"key":"ref_42","unstructured":"The NCEP\/NCAR Reanalysis Project, Available online: http:\/\/www.esrl.noaa.gov\/psd\/data\/reanalysis\/reanalysis.shtml\/."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/3\/721\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:45:06Z","timestamp":1760219106000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/3\/721"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,2,25]]},"references-count":42,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2013,3]]}},"alternative-id":["e15030721"],"URL":"https:\/\/doi.org\/10.3390\/e15030721","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2013,2,25]]}}}