{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:56:42Z","timestamp":1760237802465,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2020,6,28]],"date-time":"2020-06-28T00:00:00Z","timestamp":1593302400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/L027208\/1","EP\/F500385\/1"],"award-info":[{"award-number":["EP\/L027208\/1","EP\/F500385\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000268","name":"Biotechnology and Biological Sciences Research Council","doi-asserted-by":"publisher","award":["BB\/F529254\/1"],"award-info":[{"award-number":["BB\/F529254\/1"]}],"id":[{"id":"10.13039\/501100000268","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In this work we explore encoding strategies learned by statistical models of sensory coding in noisy spiking networks. Early stages of sensory communication in neural systems can be viewed as encoding channels in the information-theoretic sense. However, neural populations face constraints not commonly considered in communications theory. Using restricted Boltzmann machines as a model of sensory encoding, we find that networks with sufficient capacity learn to balance precision and noise-robustness in order to adaptively communicate stimuli with varying information content. Mirroring variability suppression observed in sensory systems, informative stimuli are encoded with high precision, at the cost of more variable responses to frequent, hence less informative stimuli. Curiously, we also find that statistical criticality in the neural population code emerges at model sizes where the input statistics are well captured. These phenomena have well-defined thermodynamic interpretations, and we discuss their connection to prevailing theories of coding and statistical criticality in neural populations.<\/jats:p>","DOI":"10.3390\/e22070714","type":"journal-article","created":{"date-parts":[[2020,6,29]],"date-time":"2020-06-29T03:40:07Z","timestamp":1593402007000},"page":"714","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Optimal Encoding in Stochastic Latent-Variable Models"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4196-774X","authenticated-orcid":false,"given":"Michael E.","family":"Rule","sequence":"first","affiliation":[{"name":"Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0182-7443","authenticated-orcid":false,"given":"Martino","family":"Sorbaro","sequence":"additional","affiliation":[{"name":"Institute of Neuroinformatics, University of Z\u00fcrich and ETH, 8057 Z\u00fcrich, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7270-5817","authenticated-orcid":false,"given":"Matthias H.","family":"Hennig","sequence":"additional","affiliation":[{"name":"Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, UK"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1068\/p010371","article-title":"Single units and sensation: A neuron doctrine for perceptual psychology?","volume":"1","author":"Barlow","year":"1972","journal-title":"Perception"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1002\/j.1538-7305.1948.tb00917.x","article-title":"A mathematical theory of communication, Part I, Part II","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2379","DOI":"10.1364\/JOSAA.4.002379","article-title":"Relations between the statistics of natural images and the response properties of cortical cells","volume":"4","author":"Field","year":"1987","journal-title":"Josa A"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1129","DOI":"10.1162\/neco.1995.7.6.1129","article-title":"An information-maximization approach to blind separation and blind deconvolution","volume":"7","author":"Bell","year":"1995","journal-title":"Neural Comput."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1273","DOI":"10.1126\/science.287.5456.1273","article-title":"Sparse coding and decorrelation in primary visual cortex during natural vision","volume":"287","author":"Vinje","year":"2000","journal-title":"Science"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1038\/nn.2501","article-title":"Stimulus onset quenches neural variability: A widespread cortical phenomenon","volume":"13","author":"Churchland","year":"2010","journal-title":"Nat. Neurosci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1016\/j.neuron.2016.09.038","article-title":"Neural variability and sampling-based probabilistic representations in the visual cortex","volume":"92","author":"Berkes","year":"2016","journal-title":"Neuron"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Prentice, J.S., Marre, O., Ioffe, M.L., Loback, A.R., Tka\u010dik, G., and Berry, M.J. (2016). Error-robust modes of the retinal population code. PLoS Comput. Biol., 12.","DOI":"10.1371\/journal.pcbi.1005148"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3119","DOI":"10.1162\/neco_a_01011","article-title":"Noise-Robust Modes of the Retinal Population Code Have the Geometry of \u201cRidges\u201d and Correspond to Neuronal Communities","volume":"29","author":"Loback","year":"2017","journal-title":"Neural Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00422-009-0328-3","article-title":"The Wilson\u2013Cowan model, 36 years later","volume":"101","author":"Destexhe","year":"2009","journal-title":"Biol. Cybern."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1038\/nature04701","article-title":"Weak pairwise correlations imply strongly correlated network states in a neural population","volume":"440","author":"Schneidman","year":"2006","journal-title":"Nature"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"8254","DOI":"10.1523\/JNEUROSCI.1282-06.2006","article-title":"The structure of multi-neuron firing patterns in primate retina","volume":"26","author":"Shlens","year":"2006","journal-title":"J. Neurosci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"K\u00f6ster, U., Sohl-Dickstein, J., Gray, C.M., and Olshausen, B.A. (2014). Modeling higher-order correlations within cortical microcolumns. PLoS Comput. Biol., 10.","DOI":"10.1371\/journal.pcbi.1003684"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"11508","DOI":"10.1073\/pnas.1514188112","article-title":"Thermodynamics and signatures of criticality in a network of neurons","volume":"112","author":"Mora","year":"2015","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_15","unstructured":"Hinton, G.E., and Brown, A.D. (2000). Spiking boltzmann machines. Advances in Neural Information Processing Systems, The MIT Press."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"P03006","DOI":"10.1088\/1742-5468\/2013\/03\/P03006","article-title":"Spatio-temporal spike train analysis for large scale networks using the maximum entropy principle and Monte Carlo method","volume":"2013","author":"Nasser","year":"2013","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_17","unstructured":"Zanotto, M., Volpi, R., Maccione, A., Berdondini, L., Sona, D., and Murino, V. (2017). Modeling retinal ganglion cell population activity with restricted Boltzmann machines. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3267","DOI":"10.1073\/pnas.1718710115","article-title":"Blindfold learning of an accurate neural metric","volume":"115","author":"Gardella","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Turcsany, D., Bargiela, A., and Maul, T. (2014, January 27\u201330). Modelling Retinal Feature Detection with Deep Belief Networks in a Simulated Environment. Proceedings of the 28th European Conference on Modelling and Simulation (ECMS), Brescia, Italy.","DOI":"10.7148\/2014-0364"},{"key":"ref_20","unstructured":"Shao, L.Y. (2013, January 2\u20134). Linear-Nonlinear-Poisson Neurons Can Do Inference on Deep Boltzmann Machines. Proceedings of the 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, AZ, USA."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"068102","DOI":"10.1103\/PhysRevLett.113.068102","article-title":"Zipf\u2019s law and criticality in multivariate data without fine-tuning","volume":"113","author":"Schwab","year":"2014","journal-title":"Phys. Rev. Lett."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"P10012","DOI":"10.1088\/1742-5468\/2011\/10\/P10012","article-title":"On the criticality of inferred models","volume":"2011","author":"Mastromatteo","year":"2011","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"163","DOI":"10.3389\/fphys.2012.00163","article-title":"Being critical of criticality in the brain","volume":"3","author":"Beggs","year":"2012","journal-title":"Front. Physiol."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Aitchison, L., Corradi, N., and Latham, P.E. (2016). Zipf\u2019s Law Arises Naturally When There Are Underlying, Unobserved Variables. PLoS Comput. Biol., 12.","DOI":"10.1371\/journal.pcbi.1005110"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"012413","DOI":"10.1103\/PhysRevE.95.012413","article-title":"Power-law statistics and universal scaling in the absence of criticality","volume":"95","author":"Touboul","year":"2017","journal-title":"Phys. Rev. E"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1162\/089976602760128018","article-title":"Training products of experts by minimizing contrastive divergence","volume":"14","author":"Hinton","year":"2002","journal-title":"Neural Comput."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hinton, G.E. (2012). A practical guide to training restricted Boltzmann machines. Neural Networks: Tricks of the Trade, Springer.","DOI":"10.1007\/978-3-642-35289-8_32"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1158","DOI":"10.1126\/science.7761831","article-title":"The \u201cwake-sleep\u201d algorithm for unsupervised neural networks","volume":"268","author":"Hinton","year":"1995","journal-title":"Science"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"889","DOI":"10.1162\/neco.1995.7.5.889","article-title":"The Helmholtz Machine","volume":"7","author":"Dayan","year":"1995","journal-title":"Neural Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1007\/s10955-011-0229-4","article-title":"Are Biological Systems Poised at Criticality?","volume":"144","author":"Mora","year":"2011","journal-title":"J. Stat. Phys."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Sorbaro, M., Herrmann, J.M., and Hennig, M. (2019). Statistical models of neural activity, criticality, and Zipf\u2019s law. The Functional Role of Critical Dynamics in Neural Systems, Springer.","DOI":"10.1007\/978-3-030-20965-0_13"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1007\/s10955-017-1770-6","article-title":"PCA meets RG","volume":"167","author":"Bradde","year":"2017","journal-title":"J. Stat. Phys."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"178103","DOI":"10.1103\/PhysRevLett.123.178103","article-title":"Coarse graining, fixed points, and scaling in a large population of neurons","volume":"123","author":"Meshulam","year":"2019","journal-title":"Phys. Rev. Lett."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1038\/s41586-019-1346-5","article-title":"High-dimensional geometry of population responses in visual cortex","volume":"571","author":"Stringer","year":"2019","journal-title":"Nature"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ioffe, M.L., and Berry, M.J. (2017). The structured \u2018low temperature\u2019 phase of the retinal population code. PLoS Comput. Biol., 13.","DOI":"10.1371\/journal.pcbi.1005792"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"P03005","DOI":"10.1088\/1742-5468\/2013\/03\/P03005","article-title":"The effect of nonstationarity on models inferred from neural data","volume":"2013","author":"Tyrcha","year":"2013","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Nonnenmacher, M., Behrens, C., Berens, P., Bethge, M., and Macke, J.H. (2017). Signatures of criticality arise from random subsampling in simple population models. PLoS Comput. Biol., 13.","DOI":"10.1371\/journal.pcbi.1005718"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1329","DOI":"10.1162\/NECO_a_00607","article-title":"On criticality in high-dimensional data","volume":"26","author":"Saremi","year":"2014","journal-title":"Neural Comput."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1103\/PhysRevLett.58.86","article-title":"Nonuniversal critical dynamics in Monte Carlo simulations","volume":"58","author":"Swendsen","year":"1987","journal-title":"Phys. Rev. Lett."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"018701","DOI":"10.1103\/PhysRevLett.110.018701","article-title":"Statistical Thermodynamics of Natural Images","volume":"110","author":"Stephens","year":"2013","journal-title":"Phys. Rev. Lett."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"118102","DOI":"10.1103\/PhysRevLett.97.118102","article-title":"Does the 1\/f frequency scaling of brain signals reflect self-organized critical states?","volume":"97","author":"Bedard","year":"2006","journal-title":"Phys. Rev. Lett."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"041116","DOI":"10.1103\/PhysRevE.84.041116","article-title":"Relating Fisher information to order parameters","volume":"84","author":"Prokopenko","year":"2011","journal-title":"Phys. Rev. E"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1016\/j.copbio.2008.06.008","article-title":"Sloppiness, robustness, and evolvability in systems biology","volume":"19","author":"Daniels","year":"2008","journal-title":"Curr. Opin. Biotechnol."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Gutenkunst, R.N., Waterfall, J.J., Casey, F.P., Brown, K.S., Myers, C.R., and Sethna, J.P. (2007). Universally sloppy parameter sensitivities in systems biology models. PLoS Comput. Biol., 3.","DOI":"10.1371\/journal.pcbi.0030189"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"8480","DOI":"10.1523\/JNEUROSCI.4421-14.2015","article-title":"Sloppiness in spontaneously active neuronal networks","volume":"35","author":"Panas","year":"2015","journal-title":"J. Neurosci."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"15732","DOI":"10.1523\/JNEUROSCI.0301-09.2011","article-title":"Synergy from silence in a combinatorial neural code","volume":"31","author":"Schneidman","year":"2011","journal-title":"J. Neurosci."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2383","DOI":"10.1152\/jn.00723.2011","article-title":"Suppression of cortical neural variability is stimulus-and state-dependent","volume":"108","author":"White","year":"2012","journal-title":"J. Neurophysiol."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Festa, D., Aschner, A., Davila, A., Kohn, A., and Coen-Cagli, R. (2020). Neuronal variability reflects probabilistic inference tuned to natural image statistics. BioRxiv.","DOI":"10.1101\/2020.06.17.142182"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1038\/nrn2787","article-title":"The free-energy principle: A unified brain theory?","volume":"11","author":"Friston","year":"2010","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"052140","DOI":"10.1103\/PhysRevE.99.052140","article-title":"Correspondence between thermodynamics and inference","volume":"99","author":"LaMont","year":"2019","journal-title":"Phys. Rev. E"},{"key":"ref_51","unstructured":"Aitchison, L., Hennequin, G., and Lengyel, M. (2018). Sampling-based probabilistic inference emerges from learning in neural circuits with a cost on reliability. arXiv."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"123406","DOI":"10.1088\/1742-5468\/aaf10f","article-title":"Resolution and relevance trade-offs in deep learning","volume":"2018","author":"Song","year":"2018","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"063402","DOI":"10.1088\/1742-5468\/ab16c8","article-title":"Statistical criticality arises in most informative representations","volume":"2019","author":"Cubero","year":"2019","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_54","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features From Tiny Images. [Master\u2019s Thesis, University of Toronto]."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/2200000006","article-title":"Learning deep architectures for AI","volume":"2","author":"Bengio","year":"2009","journal-title":"Found. Trends Mach. Learn."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/7\/714\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:43:58Z","timestamp":1760175838000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/7\/714"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,28]]},"references-count":55,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2020,7]]}},"alternative-id":["e22070714"],"URL":"https:\/\/doi.org\/10.3390\/e22070714","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2020,6,28]]}}}