{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T23:36:20Z","timestamp":1773704180505,"version":"3.50.1"},"reference-count":27,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2018,10,1]],"date-time":"2018-10-01T00:00:00Z","timestamp":1538352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100001201","name":"Kavli Foundation","doi-asserted-by":"publisher","award":["-"],"award-info":[{"award-number":["-"]}],"id":[{"id":"10.13039\/100001201","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005416","name":"Norges Forskningsr\u00e5d","doi-asserted-by":"publisher","award":["223262"],"award-info":[{"award-number":["223262"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In the Minimum Description Length (MDL) principle, learning from the data is equivalent to an optimal coding problem. We show that the codes that achieve optimal compression in MDL are critical in a very precise sense. First, when they are taken as generative models of samples, they generate samples with broad empirical distributions and with a high value of the relevance, defined as the entropy of the empirical frequencies. These results are derived for different statistical models (Dirichlet model, independent and pairwise dependent spin models, and restricted Boltzmann machines). Second, MDL codes sit precisely at a second order phase transition point where the symmetry between the sampled outcomes is spontaneously broken. The order parameter controlling the phase transition is the coding cost of the samples. The phase transition is a manifestation of the optimality of MDL codes, and it arises because codes that achieve a higher compression do not exist. These results suggest a clear interpretation of the widespread occurrence of statistical criticality as a characterization of samples which are maximally informative on the underlying generative process.<\/jats:p>","DOI":"10.3390\/e20100755","type":"journal-article","created":{"date-parts":[[2018,10,2]],"date-time":"2018-10-02T08:23:50Z","timestamp":1538468630000},"page":"755","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Minimum Description Length Codes Are Critical"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0002-1867","authenticated-orcid":false,"given":"Ryan John","family":"Cubero","sequence":"first","affiliation":[{"name":"Kavli Institute for Systems Neuroscience and Centre for Neural Computation, Norwegian University of Science and Technology (NTNU), 7030 Trondheim, Norway"},{"name":"The Abdus Salam International Center for Theoretical Physics, 34151 Trieste, Italy"},{"name":"Scuola Internazionale Superiore di Studi Avanzati, 34136 Trieste, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9437-6989","authenticated-orcid":false,"given":"Matteo","family":"Marsili","sequence":"additional","affiliation":[{"name":"The Abdus Salam International Center for Theoretical Physics, 34151 Trieste, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6989-3420","authenticated-orcid":false,"given":"Yasser","family":"Roudi","sequence":"additional","affiliation":[{"name":"Kavli Institute for Systems Neuroscience and Centre for Neural Computation, Norwegian University of Science and Technology (NTNU), 7030 Trondheim, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,10,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"031001","DOI":"10.1103\/RevModPhys.90.031001","article-title":"Colloquium: Criticality and dynamical scaling in living systems","volume":"90","year":"2018","journal-title":"Rev. Mod. Phys."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1080\/00107510500052444","article-title":"Power laws, Pareto distributions and Zipf\u2019s law","volume":"46","author":"Newman","year":"2005","journal-title":"Contemp. Phys."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bak, P. (1996). How Nature Works: The Science of Self-Organized Criticality, Copernicus.","DOI":"10.1007\/978-1-4757-5426-1"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1007\/s10955-011-0229-4","article-title":"Are biological systems poised at criticality?","volume":"144","author":"Mora","year":"2011","journal-title":"J. Stat. Phys."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1038\/nature10856","article-title":"A universal model for mobility and migration patterns","volume":"484","author":"Simini","year":"2012","journal-title":"Nature"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"068102","DOI":"10.1103\/PhysRevLett.113.068102","article-title":"Zipf\u2019s law and criticality in multivariate data without fine-tuning","volume":"113","author":"Schwab","year":"2014","journal-title":"Phys. Rev. Lett."},{"key":"ref_7","first-page":"1267","article-title":"On sampling and modeling complex systems","volume":"9","author":"Marsili","year":"2013","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"P10013","DOI":"10.1088\/1742-5468\/2015\/10\/P10013","article-title":"Criticality of mostly informative samples: A bayesian model selection approach","volume":"10","author":"Haimovici","year":"2015","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_9","unstructured":"Cubero, R.J., Jo, J., Marsili, M., Roudi, Y., and Song, J. (arXiv, 2018). Minimally sufficient representations, maximally informative samples and Zipf\u2019s law, arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Song, J., Marsili, M., and Jo, J. (arXiv, 2017). Resolution and relevance trade-offs in deep learning, arXiv.","DOI":"10.1088\/1742-5468\/aaf10f"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Gr\u00fcnwald, P.D. (2007). The Minimum Description Length Principle, MIT Press.","DOI":"10.7551\/mitpress\/4643.001.0001"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1243092","DOI":"10.1126\/science.1243092","article-title":"Hyperdominance in the Amazonian tree flora","volume":"342","author":"Pitman","year":"2013","journal-title":"Science"},{"key":"ref_13","unstructured":"Condit, R., Lao, S., P\u00e9rez, R., Dolins, S.B., Foster, R., and Hubbell, S. (2018, October 01). Barro Colorado Forest Census Plot Data (Version 2012). Available online: https:\/\/repository.si.edu\/handle\/10088\/20925."},{"key":"ref_14","unstructured":"(2018, October 01). Combine Your Old LEGO\u00ae to Build New Creations. Available online: https:\/\/rebrickable.com\/."},{"key":"ref_15","first-page":"021023","article-title":"Statistics of shared components in complex component systems","volume":"8","author":"Mazzolini","year":"2018","journal-title":"Phys. Rev. X"},{"key":"ref_16","first-page":"133","article-title":"Regulondb version 9.0: High-level integration of gene regulation, coexpression, motif clustering and beyond","volume":"44","author":"Salgado","year":"2015","journal-title":"Nucleic Acids Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"bar062","DOI":"10.1093\/database\/bar062","article-title":"Yeastmine\u2014An integrated data warehouse for Saccharomyces cerevisiae data as a multipurpose tool-kit","volume":"2012","author":"Balakrishnan","year":"2012","journal-title":"Database"},{"key":"ref_18","unstructured":"Cover, T.M., and Thomas, J.A. (2012). Elements of Information Theory, John Wiley & Sons."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Gr\u00fcnwald, P.D. (arXiv, 2004). A tutorial introduction to the minimum description length principle, arXiv.","DOI":"10.7551\/mitpress\/1114.003.0005"},{"key":"ref_20","first-page":"175","article-title":"Universal sequential coding of single messages","volume":"23","author":"Shtarkov","year":"1987","journal-title":"Transl. Prob. Inf. Transm."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1109\/18.481776","article-title":"Fisher information and stochastic complexity","volume":"42","author":"Rissanen","year":"1996","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_22","unstructured":"Grnwald, P.D., Myung, I.J., and Pitt, M.A. (2005). MDL, Bayesian inference, and the geometry of the space of probability distributions. Advances in Minimum Description Length: Theory and Applications, The MIT Press."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Beretta, A., Battistin, C., de Mulatier, C., Mastromatteo, I., and Marsili, M. (arXiv, 2017). The stochastic complexity of spin models: How simple are simple spin models?, arXiv.","DOI":"10.3390\/e20100739"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1162\/089976602760128018","article-title":"Training products of experts by minimizing contrastive divergence","volume":"14","author":"Hinton","year":"2002","journal-title":"Neural Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Mezard, M., and Montanari, A. (2009). Information, Physics, and Computation, Oxford University Press.","DOI":"10.1093\/acprof:oso\/9780198570837.001.0001"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"P09030","DOI":"10.1088\/1742-5468\/2014\/09\/P09030","article-title":"On the concentration of large deviations for fat tailed distributions, with application to financial data","volume":"9","author":"Filiasi","year":"2014","journal-title":"J. Stat. Mech. Theory Exp."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/20\/10\/755\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:23:38Z","timestamp":1760196218000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/20\/10\/755"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,10,1]]},"references-count":27,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2018,10]]}},"alternative-id":["e20100755"],"URL":"https:\/\/doi.org\/10.3390\/e20100755","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,10,1]]}}}