{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:12:33Z","timestamp":1760364753418,"version":"build-2065373602"},"reference-count":26,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2020,1,9]],"date-time":"2020-01-09T00:00:00Z","timestamp":1578528000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We supply corrected proofs of the invariance of completion and the chain rule for the Shannon information measures of arbitrary fields, as stated by D\u0119bowski in 2009. Our corrected proofs rest on a number of auxiliary approximation results for Shannon information measures, which may be of an independent interest. As also discussed briefly in this article, the generalized calculus of Shannon information measures for fields, including the invariance of completion and the chain rule, is useful in particular for studying the ergodic decomposition of stationary processes and its links with statistical modeling of natural language.<\/jats:p>","DOI":"10.3390\/e22010079","type":"journal-article","created":{"date-parts":[[2020,1,10]],"date-time":"2020-01-10T04:06:51Z","timestamp":1578629211000},"page":"79","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Approximating Information Measures for Fields"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7136-5283","authenticated-orcid":false,"given":"\u0141ukasz","family":"D\u0119bowski","sequence":"first","affiliation":[{"name":"Institute of Computer Science, Polish Academy of Sciences, ul. Jana Kazimierza 5, 01-248 Warszawa, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1260","DOI":"10.1016\/j.spl.2009.01.016","article-title":"A general definition of conditional information and its application to ergodic decomposition","volume":"79","year":"2009","journal-title":"Stat. Probab. Lett."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"4589","DOI":"10.1109\/TIT.2011.2145170","article-title":"On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts","volume":"57","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"D\u0119bowski, \u0141. (2018). Is Natural Language a Perigraphic Process? The Theorem about Facts and Words Revisited. Entropy, 20.","DOI":"10.3390\/e20020085"},{"key":"ref_4","first-page":"745","article-title":"Towards the general definition of the amount of information","volume":"111","author":"Gelfand","year":"1956","journal-title":"Dokl. Akad. Nauk. SSSR"},{"key":"ref_5","first-page":"3","article-title":"A general formulation of the fundamental Shannon theorems in information theory","volume":"14","author":"Dobrushin","year":"1959","journal-title":"Uspekhi Mat. Nauk."},{"key":"ref_6","unstructured":"Pinsker, M.S. (1964). Information and Information Stability of Random Variables and Processes, Holden-Day."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/S0019-9958(78)90026-8","article-title":"A definition of conditional mutual information for arbitrary ensembles","volume":"38","author":"Wyner","year":"1978","journal-title":"Inf. Control."},{"key":"ref_8","unstructured":"Billingsley, P. (1979). Probability and Measure, John Wiley."},{"key":"ref_9","unstructured":"Cover, T.M., and Thomas, J.A. (1991). Elements of Information Theory, John Wiley."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1063\/1.1530990","article-title":"Regularities unseen, randomness observed: The entropy convergence hierarchy","volume":"15","author":"Crutchfield","year":"2003","journal-title":"Chaos"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1073\/pnas.17.2.656","article-title":"Proof of the ergodic theorem","volume":"17","author":"Birkhoff","year":"1932","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_12","first-page":"1","article-title":"On the fundamental ideas of measure theory","volume":"10","author":"Rokhlin","year":"1962","journal-title":"Am. Math. Soc. Transl. Ser. 1"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1109\/TIT.1974.1055287","article-title":"The ergodic decomposition of stationary discrete random processses","volume":"20","author":"Gray","year":"1974","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"385","DOI":"10.3390\/e110300385","article-title":"Properties of the Statistical Complexity Functional and Partially Deterministic HMMs","volume":"11","year":"2009","journal-title":"Entropy"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"050106","DOI":"10.1103\/PhysRevE.91.050106","article-title":"Signatures of infinity: Nonergodicity and resource scaling in prediction, complexity, and learning","volume":"91","author":"Crutchfield","year":"2015","journal-title":"Phys. Rev. E"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1515\/FREQ.1990.44.9-10.243","article-title":"Der bekannte Grenzwert der redundanzfreien Information in Texten\u2014eine Fehlinterpretation der Shannonschen Experimente?","volume":"44","author":"Hilberg","year":"1990","journal-title":"Frequenz"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1002\/j.1538-7305.1951.tb01366.x","article-title":"Prediction and entropy of printed English","volume":"30","author":"Shannon","year":"1951","journal-title":"Bell Syst. Tech. J."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Takahira, R., Tanaka-Ishii, K., and D\u0119bowski, \u0141. (2016). Entropy Rate Estimates for Natural Language\u2014A New Extrapolation of Compressed Large-Scale Corpora. Entropy, 18.","DOI":"10.3390\/e18100364"},{"key":"ref_19","unstructured":"Herdan, G. (1964). Quantitative Linguistics, Butterworths."},{"key":"ref_20","unstructured":"Heaps, H.S. (1978). Information Retrieval\u2014Computational and Theoretical Aspects, Academic Press."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hahn, M., and Futrell, R. (2019). Estimating Predictive Rate-Distortion Curves via Neural Variational Inference. Entropy, 21.","DOI":"10.3390\/e21070640"},{"key":"ref_22","unstructured":"Braverman, M., Chen, X., Kakade, S.M., Narasimhan, K., Zhang, C., and Zhang, Y. (2019). Calibration, Entropy Rates, and Memory in Language Models. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3392","DOI":"10.1109\/TIT.2012.2190708","article-title":"Mixing, Ergodic, and Nonergodic Processes with Rapidly Growing Information between Blocks","volume":"58","year":"2012","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1007\/s10959-012-0468-6","article-title":"On Hidden Markov Processes with Infinite Excess Entropy","volume":"27","year":"2014","journal-title":"J. Theor. Probab."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1396","DOI":"10.3390\/e16031396","article-title":"Infinite Excess Entropy Processes with Countable-State Generators","volume":"16","author":"Travers","year":"2014","journal-title":"Entropy"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2212","DOI":"10.1109\/TIT.2017.2733535","article-title":"Maximal Repetition and Zero Entropy Rate","volume":"64","year":"2018","journal-title":"IEEE Trans. Inf. Theory"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/1\/79\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:29:10Z","timestamp":1760362150000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/1\/79"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,9]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,1]]}},"alternative-id":["e22010079"],"URL":"https:\/\/doi.org\/10.3390\/e22010079","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2020,1,9]]}}}