{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:31:49Z","timestamp":1760243509595,"version":"build-2065373602"},"reference-count":44,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2013,8,2]],"date-time":"2013-08-02T00:00:00Z","timestamp":1375401600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Maximum entropy models have become popular statistical models in neuroscience and other areas in biology and can be useful tools for obtaining estimates of mutual information in biological systems. However, maximum entropy models fit to small data sets can be subject to sampling bias; i.e., the true entropy of the data can be severely underestimated. Here, we study the sampling properties of estimates of the entropy obtained from maximum entropy models. We focus on pairwise binary models, which are used extensively to model neural population activity. We show that if the data is well described by a pairwise model, the bias is equal to the number of parameters divided by twice the number of observations. If, however, the higher order correlations in the data deviate from those predicted by the model, the bias can be larger. Using a phenomenological model of neural population recordings, we find that this additional bias is highest for small firing probabilities, strong correlations and large population sizes\u2014for the parameters we tested, a factor of about four higher. We derive guidelines for how long a neurophysiological experiment needs to be in order to ensure that the bias is less than a specified criterion. Finally, we show how a modified plug-in estimate of the entropy can be used for bias correction.<\/jats:p>","DOI":"10.3390\/e15083109","type":"journal-article","created":{"date-parts":[[2013,8,2]],"date-time":"2013-08-02T11:56:22Z","timestamp":1375444582000},"page":"3109-3129","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Estimation Bias in Maximum Entropy Models"],"prefix":"10.3390","volume":"15","author":[{"given":"Jakob","family":"Macke","sequence":"first","affiliation":[{"name":"Gatsby Computational Neuroscience Unit, UCL, London WC1N 3AR, UK"},{"name":"Max Planck Institute for Biological Cybernetics, Bernstein Center for Computational Neuroscience and Werner Reichardt Centre for Integrative Neuroscience, 72076 T\u00fcbingen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Iain","family":"Murray","sequence":"additional","affiliation":[{"name":"School for Informatics, University of Edinburgh, Edinburgh EH8 9AB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter","family":"Latham","sequence":"additional","affiliation":[{"name":"Gatsby Computational Neuroscience Unit, UCL, London WC1N 3AR, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2013,8,2]]},"reference":[{"key":"ref_1","unstructured":"Rieke, F., Warland, D., de Ruyter van Steveninck, R., and Bialek, W. (1999). Spikes: Exploring the Neural Code, The MIT Press."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1038\/14731","article-title":"Information theory and neural coding","volume":"2","author":"Borst","year":"1999","journal-title":"Nat. Neurosci."},{"key":"ref_3","unstructured":"Shannon, C., and Weaver, W. (1949). The Mathematical Theory of Communication, University of Illinois Press."},{"key":"ref_4","unstructured":"Cover, T., and Thomas, J. (1991). Elements of Information Theory, Wiley."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_6","unstructured":"Miller, G. (1955). Information Theory in Psychology II-B, Free Press."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1162\/neco.1995.7.2.399","article-title":"The upward bias in measures of information derived from limited data samples","volume":"7","author":"Treves","year":"1995","journal-title":"Neural Comput."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1064","DOI":"10.1152\/jn.00559.2007","article-title":"Correcting for the sampling bias problem in spike train information measures","volume":"98","author":"Panzeri","year":"2007","journal-title":"J. Neurophysiol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1038\/nrn1888","article-title":"Neural correlations, population coding and computation","volume":"7","author":"Averbeck","year":"2006","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1038\/nrn2578","article-title":"Extracting information from neuronal populations: Information theory and decoding approaches","volume":"10","author":"Panzeri","year":"2009","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_11","first-page":"60","article-title":"Open source tools for the information theoretic analysis of neural data","volume":"4","author":"Ince","year":"2010","journal-title":"Front Neurosci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/NECO_a_00058","article-title":"Model-based decoding, information estimation, and change-point detection techniques for multineuron spike trains","volume":"23","author":"Pillow","year":"2011","journal-title":"Neural Comput."},{"key":"ref_13","unstructured":"Tka\u010dik, G., Schneidman, E., Berry, M.J., and Bialek, W. (2009). Spin glass models for a network of real neurons."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/BF02980577","article-title":"Beitrag zur Theorie des Ferromagnetismus","volume":"31","author":"Ising","year":"1925","journal-title":"Zeitschrift f\u00fcr Physik"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1038\/nature04701","article-title":"Weak pairwise correlations imply strongly correlated network states in a neural population","volume":"440","author":"Schneidman","year":"2006","journal-title":"Nature"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"8254","DOI":"10.1523\/JNEUROSCI.1282-06.2006","article-title":"The structure of multi-neuron firing patterns in primate retina","volume":"26","author":"Shlens","year":"2006","journal-title":"J. Neurosci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5022","DOI":"10.1523\/JNEUROSCI.5187-08.2009","article-title":"The structure of large-scale synchronized firing in primate retina","volume":"29","author":"Shlens","year":"2009","journal-title":"J. Neurosci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1523\/JNEUROSCI.3359-07.2008","article-title":"A maximum entropy model applied to spatial and temporal correlations from cortical networks in vitro","volume":"28","author":"Tang","year":"2008","journal-title":"J. Neurosci."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1038\/nature09178","article-title":"Sparse coding and high-order correlations in fine-scale cortical networks","volume":"466","author":"Ohiorhenuan","year":"2010","journal-title":"Nature"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2891","DOI":"10.1093\/cercor\/bhn047","article-title":"A small world of neuronal synchrony","volume":"18","author":"Yu","year":"2008","journal-title":"Cereb Cortex"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"051915","DOI":"10.1103\/PhysRevE.79.051915","article-title":"Ising model for neural data: model quality and approximate methods for extracting functional connectivity","volume":"79","author":"Roudi","year":"2009","journal-title":"Phys. Rev. E Stat. Nonlin. Soft. Matter Phys."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Roudi, Y., Aurell, E., and Hertz, J. (2009). Statistical physics of pairwise probability models. Front. Comput. Neurosci., 3.","DOI":"10.3389\/neuro.10.022.2009"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"5405","DOI":"10.1073\/pnas.1001705107","article-title":"Maximum entropy models for antibody diversity","volume":"107","author":"Mora","year":"2010","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"12494","DOI":"10.1073\/pnas.0902237106","article-title":"Maximum-entropy network analysis reveals a role for tumor necrosis factor in peripheral nerve development and function","volume":"106","author":"Dhadialla","year":"2009","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"512","DOI":"10.1038\/nature03991","article-title":"Evolutionary information for specifying a protein fold","volume":"437","author":"Socolich","year":"2005","journal-title":"Nature"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1162\/neco.2008.02-08-713","article-title":"Generating spike trains with specified correlation coefficients","volume":"21","author":"Macke","year":"2009","journal-title":"Neural Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"208102","DOI":"10.1103\/PhysRevLett.106.208102","article-title":"Common input explains higher-order correlations and entropy in a simple model of neural population activity","volume":"106","author":"Macke","year":"2011","journal-title":"Phys. Rev. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1162\/089976603321043720","article-title":"Synchronous firing and higher-order interactions in neuron pool","volume":"15","author":"Amari","year":"2003","journal-title":"Neural Comput."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"17514","DOI":"10.1523\/JNEUROSCI.3127-11.2011","article-title":"Higher-order interactions characterized in cortical activity","volume":"31","author":"Yu","year":"2011","journal-title":"J. Neurosci."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"056111","DOI":"10.1103\/PhysRevE.69.056111","article-title":"Entropy and information in neural spike trains: Progress on the sampling problem","volume":"69","author":"Nemenman","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_31","first-page":"2024","article-title":"Bayesian estimation of discrete entropy with mixtures of stick-breaking priors","volume":"25","author":"Archer","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1109\/18.30996","article-title":"Entropy expressions and their estimators for multivariate distributions","volume":"35","author":"Ahmed","year":"1989","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_33","unstructured":"Oyman, O., Nabar, R.U., Bolcskei, H., and Paulraj, A.J. (2002, January 3\u20136). Characterizing the Statistical Properties of Mutual Information in MIMO Channels: Insights into Diversity-multiplexing Tradeoff. Proceedings of the IEEE Conference Record of the Thirty-Sixth Asilomar Conference on Signals, Systems and Computers, Monterey, CA, USA."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.jmva.2003.10.003","article-title":"Estimation of the entropy of a multivariate normal distribution","volume":"92","author":"Misra","year":"2005","journal-title":"J. Multivar. Anal."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"805","DOI":"10.3390\/e13040805","article-title":"Large-sample asymptotic approximations for the sampling and posterior distributions of differential entropy for multivariate normal distributions","volume":"13","author":"Marrelec","year":"2011","journal-title":"Entropy"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1093\/biomet\/89.2.462","article-title":"On some models for multivariate binary variables parallel in complexity with the multivariate Gaussian distribution","volume":"89","author":"Cox","year":"2002","journal-title":"Biometrika"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"3297","DOI":"10.1098\/rsta.2009.0082","article-title":"The impact of high-order interactions on the rate of synchronous discharge and information transmission in somatosensory cortex","volume":"367","author":"Montani","year":"2009","journal-title":"Philos. Trans. R. Soc. A Math. Phys. Eng. Sci."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1016\/j.neunet.2010.05.008","article-title":"Information-theoretic methods for studying population codes","volume":"23","author":"Ince","year":"2010","journal-title":"Neural Netw."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"e1002922","DOI":"10.1371\/journal.pcbi.1002922","article-title":"Stimulus-dependent maximum entropy models of neural population codes","volume":"9","author":"Segev","year":"2013","journal-title":"PLoS Comput. Biol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"9146","DOI":"10.1523\/JNEUROSCI.23-27-09146.2003","article-title":"Encoding of whisker vibration by rat barrel cortex neurons: Implications for texture discrimination","volume":"23","author":"Arabzadeh","year":"2003","journal-title":"J. Neurosci."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1038\/nn.2731","article-title":"How advances in neural recording affect data analysis","volume":"14","author":"Stevenson","year":"2011","journal-title":"Nat. Neurosci."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2913","DOI":"10.1162\/neco.2007.19.11.2913","article-title":"Tight data-robust bounds to mutual information combining shuffling and model selection techniques","volume":"19","author":"Montemurro","year":"2007","journal-title":"Neural Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1007\/978-3-540-27819-1_33","article-title":"Performance Guarantees for Regularized Maximum Entropy Density Estimation","volume":"Volume 3120","author":"Singer","year":"2004","journal-title":"Learning Theory"},{"key":"ref_44","unstructured":"Schmidt M. minFunc. http:\/\/www.di.ens.fr\/\u223cmschmidt\/Software\/minFunc.html."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/8\/3109\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:48:23Z","timestamp":1760219303000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/15\/8\/3109"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,8,2]]},"references-count":44,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2013,8]]}},"alternative-id":["e15083109"],"URL":"https:\/\/doi.org\/10.3390\/e15083109","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2013,8,2]]}}}