{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T02:26:38Z","timestamp":1773714398677,"version":"3.50.1"},"reference-count":35,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2016,10,12]],"date-time":"2016-10-12T00:00:00Z","timestamp":1476230400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>One of the fundamental questions about human language is whether its entropy rate is positive. The entropy rate measures the average amount of information communicated per unit time. The question about the entropy of language dates back to experiments by Shannon in 1951, but in 1990 Hilberg raised doubt regarding a correct interpretation of these experiments. This article provides an in-depth empirical analysis, using 20 corpora of up to 7.8 gigabytes across six languages (English, French, Russian, Korean, Chinese, and Japanese), to conclude that the entropy rate is positive. To obtain the estimates for data length tending to infinity, we use an extrapolation function given by an ansatz. Whereas some ansatzes were proposed previously, here we use a new stretched exponential extrapolation function that has a smaller error of fit. Thus, we conclude that the entropy rates of human languages are positive but approximately 20% smaller than without extrapolation. Although the entropy rate estimates depend on the script kind, the exponent of the ansatz function turns out to be constant across different languages and governs the complexity of natural language in general. In other words, in spite of typological differences, all languages seem equally hard to learn, which partly confirms Hilberg\u2019s hypothesis.<\/jats:p>","DOI":"10.3390\/e18100364","type":"journal-article","created":{"date-parts":[[2016,10,12]],"date-time":"2016-10-12T10:18:49Z","timestamp":1476267529000},"page":"364","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":42,"title":["Entropy Rate Estimates for Natural Language\u2014A New Extrapolation of Compressed Large-Scale Corpora"],"prefix":"10.3390","volume":"18","author":[{"given":"Ryosuke","family":"Takahira","sequence":"first","affiliation":[{"name":"Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kumiko","family":"Tanaka-Ishii","sequence":"additional","affiliation":[{"name":"Research Center for Advanced Science and Technology, University of Tokyo, Tokyo 153-8904, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7136-5283","authenticated-orcid":false,"given":"\u0141ukasz","family":"D\u0119bowski","sequence":"additional","affiliation":[{"name":"Institute of Computer Science, Polish Academy of Sciences, Warszawa 01-248, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2016,10,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1002\/j.1538-7305.1951.tb01366.x","article-title":"Prediction and entropy of printed English","volume":"30","author":"Shannon","year":"1951","journal-title":"Bell Syst. Tech. J."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Genzel, D., and Charniak, E. (2002, January 7\u201312). Entropy Rate Constancy in Text. Proceedings of the 40th Annual Meeting of the Association for the ACL, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073117"},{"key":"ref_4","unstructured":"Levy, R., and Jaeger, T.F. (2012, January 12\u201315). Speakers Optimize Information Density through Syntactic Reduction. Proceedings of the 19th International Conference on Neural Information Processing Systems, Doha, Qatar."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1109\/TIT.1978.1055912","article-title":"A Convergent Gambling Estimate of the Entropy of English","volume":"24","author":"Cover","year":"1978","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_6","first-page":"31","article-title":"An Estimate of an Upper Bound for the Entropy of English","volume":"18","author":"Brown","year":"1983","journal-title":"Comput. Linguist."},{"key":"ref_7","unstructured":"Kontoyiannis, I. (1997). The Complexity and Entropy of Literary Styles, Department of Statistics, Stanford University. Technical Report 97."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1063\/1.166191","article-title":"Entropy estimation of symbol sequences","volume":"6","author":"Grassberger","year":"1996","journal-title":"Chaos"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1515\/FREQ.1990.44.9-10.243","article-title":"Der Bekannte Grenzwert der Redundanzfreien Information in Texten\u2014Eine Fehlinterpretation der Shannonschen Experimente?","volume":"44","author":"Hilberg","year":"1990","journal-title":"Frequenz"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"5903","DOI":"10.3390\/e17085903","article-title":"Maximal Repetitions in Written Texts: Finite Energy Hypothesis vs. Strong Hilberg Conjecture","volume":"17","year":"2015","journal-title":"Entropy"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1063\/1.1530990","article-title":"Regularities unseen, randomness observed: The entropy convergence hierarchy","volume":"15","author":"Crutchfield","year":"2003","journal-title":"Chaos"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1209\/0295-5075\/14\/3\/001","article-title":"Entropy of Symbolic Sequences: The Role of Correlations","volume":"14","author":"Ebeling","year":"1991","journal-title":"Europhys. Lett."},{"key":"ref_13","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley-Interscience."},{"key":"ref_14","first-page":"124","article-title":"Entropy and the complexity of trajectories of a dynamical system","volume":"44","author":"Brudno","year":"1982","journal-title":"Trans. Moscovian Math. Soc."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"71","DOI":"10.3390\/entropy-e10020071","article-title":"Estimating the Entropy of Binary Time Series: Methodology, Some Theory and a Simulation Study","volume":"10","author":"Gao","year":"2008","journal-title":"Entropy"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"669","DOI":"10.1109\/18.30993","article-title":"Estimating the information content of symbol sequences and efficient codes","volume":"35","author":"Grassberger","year":"1989","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_17","unstructured":"Farach, M., Noordewier, M., Savari, S., Shepp, L., Wyner, A., and Ziv, J. (1995, January 22\u201324). On the Entropy of DNA: Algorithms and Measurements Based on Memory and Rapid Convergence. Proceedings of the Sixth Annual ACM-SIAM Symposium on Discrete Algorithms, San Francisco, CA, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1214\/aop\/1176989934","article-title":"Entropy and Prefixes","volume":"20","author":"Shields","year":"1992","journal-title":"Ann. Probab."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1109\/18.382012","article-title":"The Context Tree Weighting Method: Basic Properties","volume":"41","author":"Willems","year":"1995","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1109\/TIT.1977.1055714","article-title":"A Universal Algorithm for Sequential Data Compression","volume":"23","author":"Ziv","year":"1977","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_21","unstructured":"Bell, T.C., Cleary, J.G., and Witten, I.H. (1990). Text Compression, Prentice Hall."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1109\/18.841160","article-title":"Grammar-based codes: A new class of universal lossless source codes","volume":"46","author":"Kieffer","year":"2000","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1613\/jair.374","article-title":"Identifying hierarchical structure in sequences: A linear-time algorithm","volume":"7","author":"Witten","year":"1997","journal-title":"J. Artif. Intell. Res."},{"key":"ref_24","unstructured":"Grassberger, P. (2002). Data Compression and Entropy Estimates by Non-Sequential Recursive Pair Substitution."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Woungang, I., Misra, S., and Misra, S.C. (2010). Selected Topics in Information and Coding Theory, World Scientific Publishing.","DOI":"10.1142\/9789812837172"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5708","DOI":"10.1109\/TIT.2015.2466693","article-title":"A Preadapted Universal Switch Distribution for Testing Hilberg\u2019s Conjecture","volume":"61","year":"2015","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Baayen, R.H. (2001). Word Frequency Distributions, Kluwer Academic Publishers.","DOI":"10.1007\/978-94-010-0844-0"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1017\/S1351324996001246","article-title":"Distribution of content words and phrases in text and language modelling","volume":"2","author":"Katz","year":"1996","journal-title":"Nat. Lang. Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Altmann, E.G., Pierrehumbert, J.B., and Motter, A.E. (2009). Beyond Word Frequency: Bursts, Lulls, and Scaling in the Temporal Distributions of Words. PLoS ONE, 4.","DOI":"10.1371\/journal.pone.0007678"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1109\/18.567640","article-title":"On the average redundancy rate of the Lempel-Ziv code","volume":"43","author":"Louchard","year":"1997","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2743","DOI":"10.1109\/18.720554","article-title":"The Minimum Description Length Principle in Coding and Modeling","volume":"44","author":"Barron","year":"1998","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2104","DOI":"10.1109\/18.782149","article-title":"The Asymptotic Redundancy of Bayes Rules for Markov Chains","volume":"45","author":"Atteson","year":"1999","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1080\/09296174.2015.1106268","article-title":"The Relaxed Hilberg Conjecture: A Review and New Experimental Support","volume":"22","year":"2015","journal-title":"J. Quant. Linguist."},{"key":"ref_34","unstructured":"Daniels, P.T., and Bright, W. (1996). The World\u2019s Writing Systems, Oxford University Press."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1162\/COLI_a_00228","article-title":"Computational Constancy Measures of Texts\u2014Yule\u2019s K and R\u00e9nyi\u2019s Entropy","volume":"41","author":"Shunsuke","year":"2015","journal-title":"Comput. Linguist."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/18\/10\/364\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:32:49Z","timestamp":1760211169000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/18\/10\/364"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,10,12]]},"references-count":35,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2016,10]]}},"alternative-id":["e18100364"],"URL":"https:\/\/doi.org\/10.3390\/e18100364","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,10,12]]}}}