{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T12:21:44Z","timestamp":1768566104975,"version":"3.49.0"},"reference-count":52,"publisher":"MIT Press - Journals","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2019,9]]},"abstract":"<jats:p> In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf\u2019s law, Heaps\u2019 law, Ebeling\u2019s method, Taylor\u2019s law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test n-gram language models, a probabilistic context-free grammar, language models based on Simon\/Pitman-Yor processes, neural language models, and generative adversarial networks for text generation. Our analysis reveals that language models based on recurrent neural networks with a gating mechanism (i.e., long short-term memory; a gated recurrent unit; and quasi-recurrent neural networks) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor\u2019s law is a good indicator of model quality. <\/jats:p>","DOI":"10.1162\/coli_a_00355","type":"journal-article","created":{"date-parts":[[2019,6,25]],"date-time":"2019-06-25T15:16:15Z","timestamp":1561475775000},"page":"481-513","source":"Crossref","is-referenced-by-count":19,"title":["Evaluating Computational Language Models with Scaling Properties of Natural Language"],"prefix":"10.1162","volume":"45","author":[{"given":"Shuntaro","family":"Takahashi","sequence":"first","affiliation":[{"name":"Graduate School of Engineering, The University of Tokyo, Department of Advanced Interdisciplinary Studies."}]},{"given":"Kumiko","family":"Tanaka-Ishii","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Research Center for Advanced Science and Technology."}]}],"member":"281","reference":[{"key":"bib1","first-page":"7","volume-title":"Creativity and Universality in Language","author":"Altmann Eduardo G.","year":"2017"},{"key":"bib2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0007678"},{"key":"bib3","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(2000)51:1<69::AID-ASI10>3.0.CO;2-C"},{"key":"bib4","volume-title":"Proceedings of International Conference on Learning Representations","author":"Bradbury James","year":"2017"},{"key":"bib5","author":"Che Tong","year":"2017","journal-title":"arXiv preprint arXiv:1702.07983"},{"key":"bib6","doi-asserted-by":"publisher","DOI":"10.1006\/csla.1999.0128"},{"key":"bib7","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1179"},{"key":"bib8","doi-asserted-by":"publisher","DOI":"10.1137\/070710111"},{"key":"bib9","doi-asserted-by":"publisher","DOI":"10.1016\/0378-4371(95)00025-3"},{"key":"bib10","doi-asserted-by":"publisher","DOI":"10.1209\/0295-5075\/26\/4\/001"},{"key":"bib11","doi-asserted-by":"publisher","DOI":"10.1080\/00018730801893043"},{"key":"bib12","volume-title":"Proceedings of International Conference on Learning Representations","author":"Fedus William","year":"2018"},{"key":"bib13","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1973.9030"},{"key":"bib14","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevX.3.021006"},{"key":"bib15","first-page":"2335","volume":"12","author":"Goldwater Sharon","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"bib16","volume-title":"Proceedings of International Conference on Learning Representations","author":"Grave Edouard","year":"2017"},{"key":"bib17","first-page":"5141","volume-title":"Proceedings of the Thirty-Second AAAI Conference","author":"Guo Jiaxian","year":"2018"},{"key":"bib18","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"bib19","doi-asserted-by":"publisher","DOI":"10.1109\/TASSP.1987.1165125"},{"key":"bib20","doi-asserted-by":"publisher","DOI":"10.1112\/plms\/s3-13.1.337"},{"key":"bib21","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.1995.479394"},{"key":"bib22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1105"},{"key":"bib23","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2004.03.006"},{"key":"bib24","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.79.066101"},{"key":"bib26","first-page":"74","volume-title":"Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics Workshop","author":"Lin Chin Yew","year":"2004"},{"key":"bib27","doi-asserted-by":"publisher","DOI":"10.3390\/e19070299"},{"key":"bib28","first-page":"3155","volume-title":"Advances in Neural Information Processing Systems","author":"Lin Kevin","year":"2017"},{"key":"bib29","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"bib30","doi-asserted-by":"publisher","DOI":"10.3115\/1118108.1118117"},{"key":"bib31","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0014139"},{"key":"bib32","author":"Lu Sidi","year":"2018","journal-title":"arXiv preprint arXiv:1804.03782"},{"key":"bib33","volume-title":"Foundations of Statistical Natural Language Processing","author":"Manning Chris","year":"1999"},{"key":"bib34","volume-title":"Proceedings of International Conference on Learning Representations","author":"Melis G\u00e1bor","year":"2018"},{"key":"bib35","author":"Merity Stephen","year":"2018","journal-title":"arXiv preprint arXiv:1803.08240"},{"key":"bib36","volume-title":"Proceedings of International Conference on Learning Representations","author":"Merity Stephen","year":"2018"},{"key":"bib37","volume-title":"Proceedings of International Conference on Learning Representations","author":"Merity Stephen","year":"2016"},{"key":"bib38","first-page":"1045","volume-title":"Proceedings of the 11th Annual Conference of the International Speech Communication Association","author":"Mikolov Tom\u00e1\u0161","year":"2010"},{"key":"bib39","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2012.6424228"},{"key":"bib40","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002"},{"key":"bib41","first-page":"241","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics Workshop","author":"Rajeswar Sai","year":"2017"},{"key":"bib42","doi-asserted-by":"publisher","DOI":"10.2307\/2333389"},{"key":"bib43","doi-asserted-by":"publisher","DOI":"10.1017\/S0021859600050516"},{"key":"bib44","first-page":"901","volume-title":"International Conference on Spoken Language Processing","author":"Stolcke Andreas","year":"2002"},{"key":"bib45","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0189326"},{"key":"bib46","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0164658"},{"key":"bib47","doi-asserted-by":"publisher","DOI":"10.1088\/2399-6528\/aaefb2"},{"key":"bib48","doi-asserted-by":"publisher","DOI":"10.1038\/189732a0"},{"key":"bib49","doi-asserted-by":"publisher","DOI":"10.3115\/1220175.1220299"},{"key":"bib50","volume-title":"Proceedings of International Conference on Learning Representations","author":"Yang Zhilin","year":"2018"},{"key":"bib51","first-page":"2852","volume-title":"Proceedings of The Thirty-First AAAI Conference","author":"Yu Lantao","year":"2017"},{"key":"bib52","author":"Zhang Yizhe","year":"2017","journal-title":"arXiv preprint arXiv:1706.03850"},{"key":"bib53","author":"Zhu Yaoming","year":"2018","journal-title":"arXiv preprint arXiv:1802.01886"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/coli_a_00355","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:28:25Z","timestamp":1615584505000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/45\/3\/481-513\/93374"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9]]},"references-count":52,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,9]]}},"alternative-id":["10.1162\/coli_a_00355"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00355","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9]]}}}