{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T04:22:12Z","timestamp":1772252532455,"version":"3.50.1"},"reference-count":32,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2019,3,4]],"date-time":"2019-03-04T00:00:00Z","timestamp":1551657600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01 DC013698"],"award-info":[{"award-number":["R01 DC013698"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback-Leibler divergence and R\u00e9nyi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.<\/jats:p>","DOI":"10.3390\/e21030243","type":"journal-article","created":{"date-parts":[[2019,3,4]],"date-time":"2019-03-04T05:22:26Z","timestamp":1551676946000},"page":"243","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8333-6586","authenticated-orcid":false,"given":"Wentao","family":"Huang","sequence":"first","affiliation":[{"name":"Key Laboratory of Cognition and Intelligence and Information Science Academy of China Electronics Technology Group Corporation, Beijing 100086, China"},{"name":"Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kechen","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,3,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1038\/14731","article-title":"Information theory and neural coding","volume":"2","author":"Borst","year":"1999","journal-title":"Nat. Neurosci."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1038\/35039062","article-title":"Information processing with population codes","volume":"1","author":"Pouget","year":"2000","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1870","DOI":"10.1126\/science.1089662","article-title":"Communication in neuronal networks","volume":"301","author":"Laughlin","year":"2003","journal-title":"Science"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1038\/nn1228","article-title":"Multiple neural spike train data analysis: State-of-the-art and future challenges","volume":"7","author":"Brown","year":"2004","journal-title":"Nat. Neurosci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3327","DOI":"10.1016\/S0042-6989(97)00121-1","article-title":"The \u201cindependent components\u201d of natural scenes are edge filters","volume":"37","author":"Bell","year":"1997","journal-title":"Vision Res."},{"key":"ref_6","unstructured":"Huang, W., and Zhang, K. (2017, January 24\u201326). An Information-Theoretic Framework for Fast and Robust Unsupervised Learning via Neural Population Infomax. Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Huang, W., Huang, X., and Zhang, K. (2017, January 22\u201324). Information-theoretic interpretation of tuning curves for multiple motion directions. Proceedings of the 2017 51st Annual Conference on Information Sciences and Systems (CISS), Baltimore, MD, USA.","DOI":"10.1109\/CISS.2017.7926142"},{"key":"ref_8","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information, Wiley-Interscience. [2nd ed.]."},{"key":"ref_9","first-page":"100","article-title":"Note on the bias of information estimates","volume":"2","author":"Miller","year":"1955","journal-title":"Inf. Theory Psychol. Probl. Methods"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1037\/h0026857","article-title":"On the bias of information estimates","volume":"71","author":"Carlton","year":"1969","journal-title":"Psychol. Bull."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1162\/neco.1995.7.2.399","article-title":"The upward bias in measures of information derived from limited data samples","volume":"7","author":"Treves","year":"1995","journal-title":"Neural Comput."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2797","DOI":"10.1162\/089976600300014728","article-title":"Asymptotic bias in information estimates and the exponential (Bell) polynomials","volume":"12","author":"Victor","year":"2000","journal-title":"Neural Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"066138","DOI":"10.1103\/PhysRevE.69.066138","article-title":"Estimating mutual information","volume":"69","author":"Kraskov","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"026209","DOI":"10.1103\/PhysRevE.76.026209","article-title":"Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data","volume":"76","author":"Khan","year":"2007","journal-title":"Phys. Rev. E"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"053302","DOI":"10.1103\/PhysRevE.98.053302","article-title":"Information estimation using nonparametric copulas","volume":"98","author":"Safaai","year":"2018","journal-title":"Phys. Rev. E"},{"key":"ref_17","first-page":"81","article-title":"Information and accuracy attainable in the estimation of statistical parameters","volume":"37","author":"Rao","year":"1945","journal-title":"Bull. Calcutta Math. Soc."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Van Trees, H.L., and Bell, K.L. (2007). Bayesian Bounds for Parameter Estimation and Nonlinear Filtering\/Tracking, John Wiley.","DOI":"10.1109\/9780470544198"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1109\/18.54897","article-title":"Information-theoretic asymptotics of Bayes methods","volume":"36","author":"Clarke","year":"1990","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1109\/18.481776","article-title":"Fisher information and stochastic complexity","volume":"42","author":"Rissanen","year":"1996","journal-title":"IEEE Trans. Inform. Theory"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1731","DOI":"10.1162\/089976698300017115","article-title":"Mutual information, Fisher information, and population coding","volume":"10","author":"Brunel","year":"1998","journal-title":"Neural Comput."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"051904","DOI":"10.1103\/PhysRevE.64.051904","article-title":"Population coding in neuronal systems with correlated noise","volume":"64","author":"Sompolinsky","year":"2001","journal-title":"Phys. Rev. E"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4958","DOI":"10.1103\/PhysRevLett.86.4958","article-title":"Mutual information of population codes and distance measures in probability space","volume":"86","author":"Kang","year":"2001","journal-title":"Phys. Rev. Lett."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"885","DOI":"10.1162\/neco_a_01056","article-title":"Information-theoretic bounds and approximations in neural population coding","volume":"30","author":"Huang","year":"2018","journal-title":"Neural Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1103\/PhysRevLett.80.197","article-title":"Entropy and information in neural spike trains","volume":"80","author":"Strong","year":"1998","journal-title":"Phys. Rev. Lett."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"056111","DOI":"10.1103\/PhysRevE.69.056111","article-title":"Entropy and information in neural spike trains: Progress on the sampling problem","volume":"69","author":"Nemenman","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1064","DOI":"10.1152\/jn.00559.2007","article-title":"Correcting for the sampling bias problem in spike train information measures","volume":"98","author":"Panzeri","year":"2017","journal-title":"J. Neurophysiol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1162\/neco_a_01155","article-title":"Calculating the Mutual Information Between Two Spike Trains","volume":"31","author":"Houghton","year":"2019","journal-title":"Neural Comput."},{"key":"ref_29","unstructured":"R\u00e9nyi, A. (1961). On measures of entropy and information. Fourth Berkeley Symposium on Mathematical Statistics and Probability, The Regents of the University of California, University of California Press."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1214\/aoms\/1177729330","article-title":"A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations","volume":"23","author":"Chernoff","year":"1952","journal-title":"Ann. Math. Stat."},{"key":"ref_31","first-page":"99","article-title":"On a measure of divergence between two statistical populations defined by their probability distributions","volume":"35","author":"Bhattacharyya","year":"1943","journal-title":"Bull. Calcutta Math. Soc."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1214\/aos\/1176343842","article-title":"Minimum Hellinger distance estimates for parametric models","volume":"5","author":"Beran","year":"1977","journal-title":"Ann. Stat."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/3\/243\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:36:05Z","timestamp":1760186165000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/3\/243"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,4]]},"references-count":32,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2019,3]]}},"alternative-id":["e21030243"],"URL":"https:\/\/doi.org\/10.3390\/e21030243","relation":{"has-preprint":[{"id-type":"doi","id":"10.20944\/preprints201812.0209.v1","asserted-by":"object"},{"id-type":"doi","id":"10.20944\/preprints201812.0209.v2","asserted-by":"object"}]},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,3,4]]}}}