{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T01:04:43Z","timestamp":1792803883672,"version":"4.5.0"},"reference-count":60,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2015,7,13]],"date-time":"2015-07-13T00:00:00Z","timestamp":1436745600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"U.S. Army Research Laboratory and the U. S. Army 395 Research Office","award":["W911NF-12-1-0288"],"award-info":[{"award-number":["W911NF-12-1-0288"]}],"id":[{"id":"https:\/\/ror.org\/011hc8f90","id-type":"ROR","asserted-by":"crossref"}]},{"name":"U.S. Army Research Laboratory and the U. S. Army 395 Research Office","award":["W911NF-13-1-0390"],"award-info":[{"award-number":["W911NF-13-1-0390"]}],"id":[{"id":"https:\/\/ror.org\/011hc8f90","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Renewal processes are broadly used to model stochastic behavior consisting of isolated events separated by periods of quiescence, whose durations are specified by a given probability law. Here, we identify the minimal sufficient statistic for their prediction (the set of causal states), calculate the historical memory capacity required to store those states (statistical complexity), delineate what information is predictable (excess entropy), and decompose the entropy of a single measurement into that shared with the past, future, or both. The causal state equivalence relation defines a new subclass of renewal processes with a finite number of causal states despite having an unbounded interevent count distribution. We use the resulting formulae to analyze the output of the parametrized Simple Nonunifilar Source, generated by a simple two-state hidden Markov model, but with an infinite-state machine presentation. All in all, the results lay the groundwork for analyzing more complex processes with infinite statistical complexity and infinite excess entropy.<\/jats:p>","DOI":"10.3390\/e17074891","type":"journal-article","created":{"date-parts":[[2015,7,13]],"date-time":"2015-07-13T10:38:10Z","timestamp":1436783890000},"page":"4891-4917","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Informational and Causal Architecture of Discrete-Time Renewal Processes"],"prefix":"10.3390","volume":"17","author":[{"given":"Sarah","family":"Marzen","sequence":"first","affiliation":[{"name":"Department of Physics, University of California at Berkeley, Berkeley, CA 94720-5800, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4466-5410","authenticated-orcid":false,"given":"James","family":"Crutchfield","sequence":"additional","affiliation":[{"name":"Complexity Sciences Center and Department of Physics, University of California at Davis, One Shields Avenue, Davis, CA 95616, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2015,7,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1111\/j.2517-6161.1958.tb00294.x","article-title":"Renewal Theory and Its Ramifications","volume":"20","author":"Smith","year":"1958","journal-title":"J. R. Stat. Soc. B"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gerstner, W., and Kistler, W.M. (2002). Spiking Neuron Models: Single Neurons, Populations, Plasticity, Cambridge University Press. Chapter 5.2.","DOI":"10.1017\/CBO9780511815706"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Beichelt, F. (2006). Stochastic Processes in Science, Engineering and Finance, Chapman and Hall.","DOI":"10.1201\/9781420010459"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Barbu, V.S., and Limnios, N. (2008). Semi-Markov Chains and Hidden Semi-Markov Models toward Applications: Their Use in Reliability and DNA Analysis, Springer.","DOI":"10.1007\/978-0-387-73173-5_3"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"992","DOI":"10.1103\/PhysRevE.47.992","article-title":"Fractal Renewal Processes Generate 1\/f Noise","volume":"47","author":"Lowen","year":"1993","journal-title":"Phys. Rev. E"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1669","DOI":"10.1109\/18.259653","article-title":"Fractal Renewal Processes","volume":"39","author":"Lowen","year":"1993","journal-title":"IEEE Trans. Inf. Theory."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"021108","DOI":"10.1103\/PhysRevE.74.021108","article-title":"Dynamical Origin of Memory and Renewal","volume":"74","author":"Cakir","year":"2006","journal-title":"Phys. Rev. E"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"031133","DOI":"10.1103\/PhysRevE.81.031133","article-title":"Characterization of intermittency in renewal processes: Application to earthquakes","volume":"81","author":"Akimoto","year":"2010","journal-title":"Phys. Rev. E"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"012116","DOI":"10.1103\/PhysRevE.87.012116","article-title":"Monotonic Continuous-time Random Walks with Drift and Stochastic Reset Events","volume":"87","author":"Montero","year":"2013","journal-title":"Phys. Rev. E"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"062106","DOI":"10.1103\/PhysRevE.88.062106","article-title":"Renewal and Memory Origin of Anomalous Diffusion: A discussion of their joint action","volume":"88","author":"Bologna","year":"2013","journal-title":"Phys. Rev. E"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"061911","DOI":"10.1103\/PhysRevE.75.061911","article-title":"Brain Music Non-Poisson Renewal Processes","volume":"75","author":"Bianco","year":"2007","journal-title":"Phys. Rev. E"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1073\/pnas.0707378105","article-title":"Multiscale Complex Network of Protein Conformational Fluctuations in Single-Molecule Time Series","volume":"105","author":"Li","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e29703","DOI":"10.1371\/journal.pone.0029703","article-title":"A New Method for Inferring Hidden Markov Models from Noisy Time Sequences","volume":"7","author":"Kelly","year":"2012","journal-title":"PLoS ONE"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"042817","DOI":"10.1103\/PhysRevE.89.042817","article-title":"Bursting Transition in a Linear Self-exciting Point Process","volume":"89","author":"Onaga","year":"2014","journal-title":"Phys. Rev. E"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"052803","DOI":"10.1103\/PhysRevE.89.052803","article-title":"Inhomogeneous Point-Process Entropy: An Instantaneous measure of complexity in discrete systems","volume":"89","author":"Valenza","year":"2014","journal-title":"Phys. Rev. E"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1142\/S0219525902000481","article-title":"Information Bottlenecks, Causal States, and Statistical Relevance Bases: How to Represent Relevant Information in Memoryless Transduction","volume":"5","author":"Shalizi","year":"2002","journal-title":"Adv. Complex Syst."},{"key":"ref_17","unstructured":"Still, S., and Crutchfield, J.P. (2007). Structure or Noise?, arXiv, 0708.0654."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"037111","DOI":"10.1063\/1.3489885","article-title":"Optimal Causal Inference: Estimating Stored Information and Approximating Causal Architecture","volume":"20","author":"Still","year":"2010","journal-title":"Chaos"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4713","DOI":"10.3390\/e16094713","article-title":"Information Anatomy of Stochastic Equilibria","volume":"16","author":"Marzen","year":"2014","journal-title":"Entropy"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"e63400","DOI":"10.1371\/journal.pone.0063400","article-title":"Information driven self-organization of complex robotics behaviors","volume":"8","author":"Martius","year":"2013","journal-title":"PLoS ONE"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1063\/1.1530990","article-title":"Regularities Unseen Randomness Observed: Levels of Entropy Convergence","volume":"13","author":"Crutchfield","year":"2003","journal-title":"Chaos"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1007\/s10959-012-0468-6","article-title":"On Hidden Markov Processes with Infinite Excess Entropy","volume":"27","author":"Debowski","year":"2012","journal-title":"J. Theor. Prob."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1396","DOI":"10.3390\/e16031396","article-title":"Infinite Excess Entropy Processes with Countable-State Generators","volume":"16","author":"Travers","year":"2014","journal-title":"Entropy"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"042119","DOI":"10.1103\/PhysRevE.89.042119","article-title":"Bayesian Structural Inference for Hidden Processes","volume":"89","author":"Strelioff","year":"2014","journal-title":"Phys. Rev. E"},{"key":"ref_25","unstructured":"Crutchfield, J.P., Riechers, P., and Ellison, C.J. (2013). Exact Complexity: Spectral Decomposition of Intrinsic Computation, arXiv, 1309.3792. cond-mat.stat-mech."},{"key":"ref_26","unstructured":"Blackwell, D. (1957). The Entropy of Functions of Finite-state Markov Chains, Publishing House of the Czechoslovak Academy of Sciences."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1038\/nphys2190","article-title":"Between Order Chaos","volume":"8","author":"Crutchfield","year":"2012","journal-title":"Nat. Phys."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Marzen, S.E., DeWeese, M.R., and Crutchfield, J.P. (2015). Time Resolution Dependence of Information Measures for Spiking Neurons: Atoms, Scaling and Universality, arXiv, 1504.04756.","DOI":"10.3389\/fncom.2015.00105"},{"key":"ref_29","unstructured":"Marzen, S.E., and Crutchfield, J.P. (2015). Long-Range Memory in Stationary Renewal Processes, in preparation."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1518","DOI":"10.1109\/TIT.2002.1003838","article-title":"Hidden Markov Processes","volume":"48","author":"Ephraim","year":"2002","journal-title":"IEEE Trans. Inf. Theory."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley. [2nd ed].","DOI":"10.1002\/047174882X"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1103\/PhysRevLett.63.105","article-title":"Inferring Statistical Complexity","volume":"63","author":"Crutchfield","year":"1989","journal-title":"Phys. Rev. Let"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1023\/A:1010388907793","article-title":"Computational Mechanics: Pattern and Prediction, Structure and Simplicity","volume":"104","author":"Shalizi","year":"2001","journal-title":"J. Stat. Phys."},{"key":"ref_34","unstructured":"Paz, A. (1971). Introduction to Probabilistic Automata, Academic Press."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MASSP.1986.1165342","article-title":"An Introduction to Hidden Markov Models","volume":"3","author":"Rabiner","year":"1986","journal-title":"IEEE ASSP Mag."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1109\/5.18626","article-title":"A Tutorial on Hidden Markov Models and Selected Applications","volume":"77","author":"Rabiner","year":"1989","journal-title":"IEEE Proc."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"094101","DOI":"10.1103\/PhysRevLett.103.094101","article-title":"Time\u2019s Barbed Arrow: Irreversibility, Crypticity, and Stored Information","volume":"103","author":"Crutchfield","year":"2009","journal-title":"Phys. Rev. Lett."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"037107","DOI":"10.1063\/1.3637490","article-title":"Information Symmetries in Irreversible Processes","volume":"21","author":"Ellison","year":"2011","journal-title":"Chaos"},{"key":"ref_39","unstructured":"Yeung, R.W. (2008). Information Theory and Network Coding, Springer."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"037109","DOI":"10.1063\/1.3637494","article-title":"Anatomy of a Bit: Information in a Time Series Observation","volume":"21","author":"James","year":"2011","journal-title":"Chaos"},{"key":"ref_41","unstructured":"Amari, S.-i., Cichocki, A., Makino, S., and Murata, N. (2003). The Co-information Lattice, Springer."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ara, P.M., James, R.G., and Crutchfield, J.P. (2015). The Elusive Present: Hidden Past and Future Correlation and Why We Build Models, arXiv, 1507.00672.","DOI":"10.1103\/PhysRevE.93.022143"},{"key":"ref_43","first-page":"223","article-title":"Computation at the Onset of Chaos","volume":"VIII","author":"Zurek","year":"1990","journal-title":"Entropy, Complexity, and the Physics of Information; SFI Studies in the Sciences of Complexity"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2409","DOI":"10.1162\/089976601753195969","article-title":"Predictability Complexity Learning","volume":"13","author":"Bialek","year":"2001","journal-title":"Neural Comp."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/0167-2789(94)90273-9","article-title":"The Calculi of Emergence: Computation, Dynamics, and Induction","volume":"75","author":"Crutchfield","year":"1994","journal-title":"Physica D"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1005","DOI":"10.1007\/s10955-009-9808-z","article-title":"Prediction Retrodiction the Amount of Information Stored in the Present","volume":"136","author":"Ellison","year":"2009","journal-title":"J. Stat. Phys."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Godreche, C., Majumdar, S.N., and Schehr, G. (2015). Statistics of the Longest Interval in Renewal Processes. J. Stat. Mech., P03014.","DOI":"10.1088\/1742-5468\/2015\/03\/P03014"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"930","DOI":"10.1214\/aoms\/1177704462","article-title":"Approximations for the Entropy for Functions of Markov Chains","volume":"33","author":"Birch","year":"1962","journal-title":"Ann. Math. Stat."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1107\/S2053273315005264","article-title":"Pairwise Correlations in Layered Close-Packed Structures","volume":"71","author":"Riechers","year":"2015","journal-title":"Acta Cryst. A"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"037111","DOI":"10.1063\/1.3489885","article-title":"Optimal Causal Inference: Estimating stored information and approximating causal architecture","volume":"20","author":"Still","year":"2010","journal-title":"Chaos"},{"key":"ref_51","unstructured":"Yamaguti, M. (1994). Towards the Harnessing of Chaos, Elsevier Science."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"R1239","DOI":"10.1103\/PhysRevE.55.R1239","article-title":"Statistical Complexity of Simple One-Dimensional Spin Systems","volume":"55","author":"Crutchfield","year":"1997","journal-title":"Phys. Rev. E"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"051103","DOI":"10.1103\/PhysRevE.67.051104","article-title":"Structural Information in Two-Dimensional Patterns: Entropy Convergence and Excess Entropy","volume":"67","author":"Feldman","year":"2003","journal-title":"Phys. Rev. E"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1007\/s10955-013-0833-6","article-title":"Predictive Information in a Nonequilibrium Critical Model","volume":"153","author":"Tchernookov","year":"2013","journal-title":"J. Stat. Phys."},{"key":"ref_55","unstructured":"Zipf, G.K. (1935). The Psycho-Biology of Language: An Introduction to Dynamic Philology, Houghton Mifflin Company."},{"key":"ref_56","unstructured":"Jackson, W. (1953). Communication Theory, Butterworths."},{"key":"ref_57","first-page":"311","article-title":"Some effects of intermittent silence","volume":"70","author":"Miller","year":"1957","journal-title":"Am. J. Psychiatry"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"11167","DOI":"10.1523\/JNEUROSCI.23-35-11167.2003","article-title":"Neuronal Avalanches in Neocortical Circuits","volume":"23","author":"Beggs","year":"2003","journal-title":"J. Nuerosci."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"e75808","DOI":"10.1371\/journal.pone.0075818","article-title":"Collective Phenomena and Non-Finite State Computation in a Human Social System","volume":"8","author":"Dedeo","year":"2013","journal-title":"PLoS ONE"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"360","DOI":"10.1016\/j.jphysparis.2013.03.001","article-title":"Spike Train statistics and Gibbs Distributions","volume":"107","author":"Cessac","year":"2013","journal-title":"J. Physiol. Paris."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/17\/7\/4891\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:49:05Z","timestamp":1760215745000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/17\/7\/4891"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,7,13]]},"references-count":60,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2015,7]]}},"alternative-id":["e17074891"],"URL":"https:\/\/doi.org\/10.3390\/e17074891","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,7,13]]}}}