{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:56:24Z","timestamp":1760241384362,"version":"build-2065373602"},"reference-count":73,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2018,1,23]],"date-time":"2018-01-23T00:00:00Z","timestamp":1516665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The Kullback\u2013Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and R\u00f6ssler systems.<\/jats:p>","DOI":"10.3390\/e20020051","type":"journal-article","created":{"date-parts":[[2018,1,23]],"date-time":"2018-01-23T13:06:51Z","timestamp":1516712811000},"page":"51","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Minimising the Kullback\u2013Leibler Divergence for Model Selection in Distributed Nonlinear Systems"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5041-4090","authenticated-orcid":false,"given":"Oliver","family":"Cliff","sequence":"first","affiliation":[{"name":"Australian Centre for Field Robotics, The University of Sydney, Sydney NSW 2006, Australia"},{"name":"Complex Systems Research Group, The University of Sydney, Sydney NSW 2006, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4215-0344","authenticated-orcid":false,"given":"Mikhail","family":"Prokopenko","sequence":"additional","affiliation":[{"name":"Complex Systems Research Group, The University of Sydney, Sydney NSW 2006, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"Fitch","sequence":"additional","affiliation":[{"name":"Australian Centre for Field Robotics, The University of Sydney, Sydney NSW 2006, Australia"},{"name":"Centre for Autonomous Systems, University of Technology Sydney, Ultimo NSW 2007, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,1,23]]},"reference":[{"unstructured":"Akaike, H. (1971, January 2\u20138). Information theory and an extension of the maximum likelihood principle. Proceedings of the Second International Symposium on Information Theory, Tsahkadsor, Armenia, USSR.","key":"ref_1"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1111\/j.1467-8640.1994.tb00166.x","article-title":"Learning Bayesian belief networks: An approach based on the MDL principle","volume":"10","author":"Lam","year":"1994","journal-title":"Comput. Intell."},{"key":"ref_3","first-page":"2149","article-title":"A Scoring Function for Learning Bayesian Networks Based on Mutual Information and Conditional Independence Tests","volume":"7","year":"2006","journal-title":"J. Mach. Learn. Res."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1126\/science.1227079","article-title":"Detecting causality in complex ecosystems","volume":"338","author":"Sugihara","year":"2012","journal-title":"Science"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1007\/s10827-010-0262-3","article-title":"Transfer entropy\u2014A model-free measure of effective connectivity for the neurosciences","volume":"30","author":"Vicente","year":"2011","journal-title":"J. Comput. Neurosci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1555","DOI":"10.1162\/NECO_a_00756","article-title":"A statistical framework to infer delay and direction of information flow from measurements of complex systems","volume":"27","author":"Schumacher","year":"2015","journal-title":"Neural Comput."},{"unstructured":"Best, G., Cliff, O.M., Patten, T., Mettu, R.R., and Fitch, R. (2016, January 18\u201320). Decentralised Monte Carlo Tree Search for Active Perception. Proceedings of the International Workshop on the Algorithmic Foundations of Robotics (WAFR), San Francisco, CA, USA.","key":"ref_7"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1162\/ARTL_a_00221","article-title":"Delayed Spatio-Temporal Interactions and Coherent Structure in Multi-Agent Team Dynamics","volume":"23","author":"Cliff","year":"2017","journal-title":"Art. Life"},{"doi-asserted-by":"crossref","unstructured":"Best, G., Forrai, M., Mettu, R.R., and Fitch, R. (2018, January 21). Planning-aware communication for decentralised multi-robot coordination. Proceedings of the International Conference on Robotics and Automation, Brisbane, Australia.","key":"ref_9","DOI":"10.1109\/ICRA.2018.8460617"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1016\/j.physrep.2005.10.009","article-title":"Complex networks: Structure and dynamics","volume":"424","author":"Boccaletti","year":"2006","journal-title":"Phys. Rep."},{"unstructured":"Mortveit, H., and Reidys, C. (2007). An Introduction to Sequential Dynamical Systems, Springer Science & Business Media.","key":"ref_11"},{"doi-asserted-by":"crossref","unstructured":"Cliff, O.M., Prokopenko, M., and Fitch, R. (2016). An Information Criterion for Inferring Coupling in Distributed Dynamical Systems. Front. Robot. AI, 3.","key":"ref_12","DOI":"10.3389\/frobt.2016.00071"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1017\/S0269888910000251","article-title":"Learning Bayesian networks: Approaches and issues","volume":"26","author":"Daly","year":"2011","journal-title":"Knowl. Eng. Rev."},{"key":"ref_14","first-page":"445","article-title":"Learning equivalence classes of Bayesian-network structures","volume":"2","author":"Chickering","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1214\/aos\/1176344136","article-title":"Estimating the dimension of a model","volume":"6","author":"Schwarz","year":"1978","journal-title":"Ann. Stat."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1016\/0005-1098(78)90005-5","article-title":"Modeling by shortest data description","volume":"14","author":"Rissanen","year":"1978","journal-title":"Automatica"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1016\/S0925-2312(02)00732-4","article-title":"Temporal infomax leads to almost deterministic dynamical systems","volume":"52","author":"Ay","year":"2003","journal-title":"Neurocomputing"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2432","DOI":"10.3390\/e17042432","article-title":"Information geometry on complexity and stochastic interaction","volume":"17","author":"Ay","year":"2015","journal-title":"Entropy"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"037109","DOI":"10.1063\/1.3486801","article-title":"Information modification and particle collisions in distributed computation","volume":"20","author":"Lizier","year":"2010","journal-title":"Chaos"},{"unstructured":"Murphy, K. (2002). Dynamic Bayesian Networks: Representation, Inference and Learning. [Ph.D. Thesis, UC Berkeley].","key":"ref_20"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1816","DOI":"10.1103\/PhysRevLett.76.1816","article-title":"Generalized synchronization, predictability, and equivalence of unidirectionally coupled dynamical systems","volume":"76","author":"Kocarev","year":"1996","journal-title":"Phys. Rev. Lett."},{"doi-asserted-by":"crossref","unstructured":"Kantz, H., and Schreiber, T. (2004). Nonlinear Time Series Analysis, Cambridge University Press.","key":"ref_22","DOI":"10.1017\/CBO9780511755798"},{"unstructured":"Pearl, J. (2014). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann.","key":"ref_23"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"424","DOI":"10.2307\/1912791","article-title":"Investigating causal relations by econometric models and cross-spectral methods","volume":"37","author":"Granger","year":"1969","journal-title":"Econometrica"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1103\/PhysRevLett.85.461","article-title":"Measuring information transfer","volume":"85","author":"Schreiber","year":"2000","journal-title":"Phys. Rev. Lett."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e238701","DOI":"10.1103\/PhysRevLett.103.238701","article-title":"Granger Causality and Transfer Entropy Are Equivalent for Gaussian Variables","volume":"103","author":"Barnett","year":"2009","journal-title":"Phys. Rev. Lett."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"605","DOI":"10.1140\/epjb\/e2010-00034-5","article-title":"Differentiating information transfer and causal effect","volume":"73","author":"Lizier","year":"2010","journal-title":"Eur. Phys. J. B"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"042917","DOI":"10.1103\/PhysRevE.87.042917","article-title":"Spurious causalities with transfer entropy","volume":"87","author":"Smirnov","year":"2013","journal-title":"Phys. Rev. E"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"238701","DOI":"10.1103\/PhysRevLett.116.238701","article-title":"Information flows? A critique of transfer entropies","volume":"116","author":"James","year":"2016","journal-title":"Phys. Rev. Lett."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"052201","DOI":"10.1103\/PhysRevE.94.052201","article-title":"Information flow and causality as rigorous notions ab initio","volume":"94","author":"Liang","year":"2016","journal-title":"Phys. Rev. E"},{"key":"ref_31","first-page":"366","article-title":"Detecting strange attractors in turbulence","volume":"Volume 898","author":"Takens","year":"1981","journal-title":"Dynamical Systems and Turbulence"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1007\/s003329900072","article-title":"Delay embeddings for forced systems. I. Deterministic forcing","volume":"9","author":"Stark","year":"1999","journal-title":"J. Nonlinear Sci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1007\/s00332-003-0534-4","article-title":"Delay embeddings for forced systems. II. Stochastic forcing","volume":"13","author":"Stark","year":"2003","journal-title":"J. Nonlinear Sci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1016\/j.neuroimage.2011.03.058","article-title":"Effective connectivity: influence, causality and biophysical modeling","volume":"58","author":"Roebroeck","year":"2011","journal-title":"Neuroimage"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1016\/j.tics.2004.07.008","article-title":"Organization, development and function of complex brain networks","volume":"8","author":"Sporns","year":"2004","journal-title":"Trends Cogn. Sci."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1238411","DOI":"10.1126\/science.1238411","article-title":"Structural and functional brain networks: From connections to cognition","volume":"342","author":"Park","year":"2013","journal-title":"Science"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1016\/j.conb.2012.11.010","article-title":"Analysing connectivity with Granger causality and dynamic causal modelling","volume":"23","author":"Friston","year":"2013","journal-title":"Curr. Opin. Neurobiol."},{"unstructured":"Lizier, J.T., and Rubinov, M. (2012). Multivariate Construction of Effective Computational Networks from Observational Data, Max Planck Institute for Mathematics in the Sciences. Preprint 25\/2012.","key":"ref_38"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"4443","DOI":"10.3390\/e16084443","article-title":"Structure of a global network of financial companies based on transfer entropy","volume":"16","author":"Sandoval","year":"2014","journal-title":"Entropy"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.procs.2015.09.176","article-title":"Using Information-theoretic Principles to Analyze and Evaluate Complex Adaptive Supply Network Architectures","volume":"61","author":"Rodewald","year":"2015","journal-title":"Procedia Comput. Sci."},{"doi-asserted-by":"crossref","unstructured":"Crosato, E., Jiang, L., Lecheval, V., Lizier, J.T., Wang, X.R., Tichit, P., Theraulaz, G., and Prokopenko, M. (arXiv, 2017). Informative and misinformative interactions in a school of fish, arXiv.","key":"ref_41","DOI":"10.1007\/s11721-018-0157-x"},{"key":"ref_42","first-page":"9","article-title":"Sample estimate of the entropy of a random vector","volume":"23","author":"Kozachenko","year":"1987","journal-title":"Probl. Peredachi Inf."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"066138","DOI":"10.1103\/PhysRevE.69.066138","article-title":"Estimating mutual information","volume":"69","author":"Kraskov","year":"2004","journal-title":"Phys. Rev. E"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"5303","DOI":"10.1016\/S0362-546X(96)00149-6","article-title":"Takens embedding theorems for forced and stochastic systems","volume":"30","author":"Stark","year":"1997","journal-title":"Nonlinear Anal. Theory Methods Appl."},{"unstructured":"Friedman, N., Murphy, K., and Russell, S. (1998, January 24\u201326). Learning the structure of dynamic probabilistic networks. Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, Madison, WI, USA.","key":"ref_45"},{"unstructured":"Koller, D., and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques, MIT Press.","key":"ref_46"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1214\/aoms\/1177732360","article-title":"The large-sample distribution of the likelihood ratio for testing composite hypotheses","volume":"9","author":"Wilks","year":"1938","journal-title":"Ann. Math. Stat."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"138105","DOI":"10.1103\/PhysRevLett.109.138105","article-title":"Transfer entropy as a log-likelihood ratio","volume":"109","author":"Barnett","year":"2012","journal-title":"Phys. Rev. Lett."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"2765","DOI":"10.1093\/bioinformatics\/btr457","article-title":"GlobalMIT: Learning globally optimal dynamic Bayesian network with the mutual information test criterion","volume":"27","author":"Vinh","year":"2011","journal-title":"Bioinformatics"},{"doi-asserted-by":"crossref","unstructured":"Deyle, E.R., and Sugihara, G. (2011). Generalized theorems for nonlinear state space reconstruction. PLoS ONE, 6.","key":"ref_50","DOI":"10.1371\/journal.pone.0018295"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1006\/jtbi.1995.0058","article-title":"The coupled logistic map: a simple model for the effects of spatial heterogeneity on population dynamics","volume":"173","author":"Lloyd","year":"1995","journal-title":"J. Theor. Biol."},{"doi-asserted-by":"crossref","unstructured":"Lizier, J.T. (2014). JIDT: An information-theoretic toolkit for studying the dynamics of complex systems. Front. Robot. AI, 1.","key":"ref_52","DOI":"10.3389\/frobt.2014.00011"},{"unstructured":"Silander, T., and Myllymaki, P. (2006, January 13\u201316). A simple approach for finding the globally optimal Bayesian network structure. Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, Cambridge, MA, USA.","key":"ref_53"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"056201","DOI":"10.1103\/PhysRevE.65.056201","article-title":"Markov models from data by simple nonlinear time series predictors in delay embedding spaces","volume":"65","author":"Ragwitz","year":"2002","journal-title":"Phys. Rev. E"},{"key":"ref_55","first-page":"283","article-title":"Optimal embedding parameters: A modelling paradigm","volume":"194","author":"Small","year":"2004","journal-title":"Physica"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1175\/1520-0469(1963)020<0130:DNF>2.0.CO;2","article-title":"Deterministic nonperiodic flow","volume":"20","author":"Lorenz","year":"1963","journal-title":"J. Atmos. Sci."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1016\/0375-9601(76)90101-8","article-title":"An equation for continuous chaos","volume":"57","year":"1976","journal-title":"Phys. Lett. A"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/0375-9601(75)90353-9","article-title":"Analogy between higher instabilities in fluids and lasers","volume":"53","author":"Haken","year":"1975","journal-title":"Phys. Lett. A"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1103\/PhysRevLett.71.65","article-title":"Circuit implementation of synchronized chaos with applications to communications","volume":"71","author":"Cuomo","year":"1993","journal-title":"Phys. Rev. Lett."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"7387","DOI":"10.1103\/PhysRevA.46.7387","article-title":"Analysis and synthesis of synchronous periodic and chaotic systems","volume":"46","author":"He","year":"1992","journal-title":"Phys. Rev. A"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1143\/PTP.69.32","article-title":"Stability theory of synchronized motion in coupled-oscillator systems","volume":"69","author":"Fujisaka","year":"1983","journal-title":"Prog. Theor. Phys."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"980","DOI":"10.1103\/PhysRevE.51.980","article-title":"Generalized synchronization of chaos in directionally coupled chaotic systems","volume":"51","author":"Rulkov","year":"1995","journal-title":"Phys. Rev. E"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1613\/jair.1061","article-title":"Searching for Bayesian network structures in the space of restricted acyclic partially directed graphs","volume":"18","author":"Acid","year":"2003","journal-title":"J. Artif. Intell. Res."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.jphysparis.2006.10.001","article-title":"A free energy principle for the brain","volume":"100","author":"Friston","year":"2006","journal-title":"J. Physiol. Paris"},{"unstructured":"Williams, P.L., and Beer, R.D. (arXiv, 2011). Generalized measures of information transfer, arXiv.","key":"ref_65"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.jneumeth.2009.07.014","article-title":"Confounding effects of indirect connections on causality estimation","volume":"184","author":"Vakorin","year":"2009","journal-title":"J. Neurosci. Methods"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"032319","DOI":"10.1103\/PhysRevE.95.032319","article-title":"Transfer entropy in continuous time, with applications to jump and neural spiking processes","volume":"95","author":"Spinney","year":"2017","journal-title":"Phys. Rev. E"},{"unstructured":"Hefferan, B., Cliff, O.M., and Fitch, R. (2016, January 5\u20137). Adversarial Patrolling with Reactive Point Processes. Proceedings of the Australasian Conference on Robotics and Automation (ACRA), Brisbane, Australia.","key":"ref_68"},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"062143","DOI":"10.1103\/PhysRevE.91.062143","article-title":"Information thermodynamics of near-equilibrium computation","volume":"91","author":"Prokopenko","year":"2015","journal-title":"Phys. Rev. E"},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"022135","DOI":"10.1103\/PhysRevE.94.022135","article-title":"Transfer entropy in physical systems and the arrow of time","volume":"94","author":"Spinney","year":"2016","journal-title":"Phys. Rev. E"},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1007\/s005740200012","article-title":"The reconstruction theorem for endomorphisms","volume":"33","author":"Takens","year":"2002","journal-title":"Bull. Braz. Math. Soc."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"1483","DOI":"10.1016\/S0893-6080(03)00190-4","article-title":"Dynamical properties of strongly interacting Markov chains","volume":"16","author":"Ay","year":"2003","journal-title":"Neural Netw."},{"doi-asserted-by":"crossref","unstructured":"Edlund, J.A., Chaumont, N., Hintze, A., Koch, C., Tononi, G., and Adami, C. (2011). Integrated information increases with fitness in the evolution of animats. PLoS Comput. Biol., 7.","key":"ref_73","DOI":"10.1371\/journal.pcbi.1002236"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/20\/2\/51\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:52:19Z","timestamp":1760194339000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/20\/2\/51"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,1,23]]},"references-count":73,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2018,2]]}},"alternative-id":["e20020051"],"URL":"https:\/\/doi.org\/10.3390\/e20020051","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2018,1,23]]}}}