{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:43:42Z","timestamp":1760233422364,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,1,7]],"date-time":"2021-01-07T00:00:00Z","timestamp":1609977600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>If regularity in data takes the form of higher-order functions among groups of variables, models which are biased towards lower-order functions may easily mistake the data for noise. To distinguish whether this is the case, one must be able to quantify the contribution of different orders of dependence to the total information. Recent work in information theory attempts to do this through measures of multivariate mutual information (MMI) and information decomposition (ID). Despite substantial theoretical progress, practical issues related to tractability and learnability of higher-order functions are still largely unaddressed. In this work, we introduce a new approach to information decomposition\u2014termed Neural Information Decomposition (NID)\u2014which is both theoretically grounded, and can be efficiently estimated in practice using neural networks. We show on synthetic data that NID can learn to distinguish higher-order functions from noise, while many unsupervised probability models cannot. Additionally, we demonstrate the usefulness of this framework as a tool for exploring biological and artificial neural networks.<\/jats:p>","DOI":"10.3390\/e23010079","type":"journal-article","created":{"date-parts":[[2021,1,8]],"date-time":"2021-01-08T08:58:34Z","timestamp":1610096314000},"page":"79","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Discovering Higher-Order Interactions Through Neural Information Decomposition"],"prefix":"10.3390","volume":"23","author":[{"given":"Kyle","family":"Reing","sequence":"first","affiliation":[{"name":"Information Sciences Institute, University of Southern California, Los Angeles, CA 90292, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0793-141X","authenticated-orcid":false,"given":"Greg","family":"Ver Steeg","sequence":"additional","affiliation":[{"name":"Information Sciences Institute, University of Southern California, Los Angeles, CA 90292, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aram","family":"Galstyan","sequence":"additional","affiliation":[{"name":"Information Sciences Institute, University of Southern California, Los Angeles, CA 90292, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pcbi.1005268","article-title":"Could a Neuroscientist Understand a Microprocessor?","volume":"13","author":"Jonas","year":"2017","journal-title":"PLoS Comput. Biol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"052418","DOI":"10.1103\/PhysRevE.99.052418","article-title":"Searching for collective behavior in a small brain","volume":"99","author":"Chen","year":"2019","journal-title":"Phys. Rev. E"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1016\/j.cels.2017.08.014","article-title":"Gene Regulatory Network Inference from Single-Cell Data Using Multivariate Information Measures","volume":"5","author":"Chan","year":"2017","journal-title":"Cell Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5","DOI":"10.3389\/frobt.2015.00005","article-title":"Bits from Brains for Biologically Inspired Computing","volume":"2","author":"Wibral","year":"2015","journal-title":"Front. Robot. AI"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1147\/rd.41.0066","article-title":"Information Theoretical Analysis of Multivariate Correlation","volume":"4","author":"Watanabe","year":"1960","journal-title":"IBM J. Res. Dev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2683","DOI":"10.1109\/TIT.2019.2956144","article-title":"Maximizing Multivariate Information With Error-Correcting Codes","volume":"66","author":"Reing","year":"2020","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_7","unstructured":"Williams, P.L., and Beer, R.D. (2010). Nonnegative Decomposition of Multivariate Information. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wibral, M., Finn, C., Wollstadt, P., Lizier, J.T., and Priesemann, V. (2017). Quantifying Information Modification in Developing Neural Networks via Partial Information Decomposition. Entropy, 19.","DOI":"10.3390\/e19090494"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"James, R.G., Emenheiser, J., and Crutchfield, J.P. (2017). Unique Information via Dependency Constraints. arXiv.","DOI":"10.1088\/1751-8121\/aaed53"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3501","DOI":"10.3390\/e17053501","article-title":"Information decomposition and synergy","volume":"17","author":"Olbrich","year":"2015","journal-title":"Entropy"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s10827-013-0458-4","article-title":"Synergy, redundancy, and multivariate information measures: an experimentalist\u2019s perspective","volume":"36","author":"Timme","year":"2014","journal-title":"J. Comput. Neurosci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"11539","DOI":"10.1523\/JNEUROSCI.23-37-11539.2003","article-title":"Synergy, redundancy, and independence in population codes","volume":"23","author":"Schneidman","year":"2003","journal-title":"J. Neurosci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R.A., and Murphy, K. (2018, January 10\u201315). Fixing a Broken ELBO. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.","DOI":"10.1007\/978-3-319-67401-8_2"},{"key":"ref_14","unstructured":"Rezende, D.J., and Viola, F. (2018). Taming VAEs. arXiv."},{"key":"ref_15","unstructured":"Belghazi, M.I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R.D. (2018). Mine: Mutual information neural estimation. arXiv."},{"key":"ref_16","unstructured":"Poole, B., Ozair, S., van den Oord, A., Alemi, A.A., and Tucker, G. (2018, January 7). On variational lower bounds of mutual information. NeurIPS Workshop on Bayesian Deep Learning, Palais des Congres de Montreal, Montreal, QC, Canada."},{"key":"ref_17","unstructured":"Tsang, M., Liu, H., Purushotham, S., Murali, P., and Liu, Y. (, January 3\u20138). Neural Interaction Transparency (NIT): Disentangling Learned Interactions for Improved Interpretability. Proceedings of the Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018), Montr\u00e9al, QC, Canada."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"037103","DOI":"10.1063\/1.3638446","article-title":"A geometric approach to complexity","volume":"21","author":"Ay","year":"2011","journal-title":"Chaos Interdiscip. J. Nonlinear Sci."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ince, R.A.A. (2017). The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal. arXiv.","DOI":"10.3390\/e19070318"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Pica, G., Piasini, E., Chicharro, D., and Panzeri, S. (2017). Invariant components of synergy, redundancy, and unique information among three variables. arXiv.","DOI":"10.3390\/e19090451"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Finn, C., and Lizier, J. (2020). Generalised Measures of Multivariate Information Content. Entropy, 22.","DOI":"10.3390\/e22020216"},{"key":"ref_22","unstructured":"Kim, H., and Mnih, A. (2018, January 10\u201315). Disentangling by Factorising. Proceedings of the 2018 International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_23","unstructured":"Locatello, F., Bauer, S., Lucic, M., Gelly, S., Sch\u00f6lkopf, B., and Bachem, O. (2018). Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. arXiv."},{"key":"ref_24","unstructured":"Kingma, D.P., and Welling, M. (2014). Auto-Encoding Variational Bayes. arXiv."},{"key":"ref_25","unstructured":"Germain, M., Gregor, K., Murray, I., and Larochelle, H. (2015). MADE: Masked Autoencoder for Distribution Estimation. arXiv."},{"key":"ref_26","first-page":"7184","article-title":"Neural Autoregressive Distribution Estimation","volume":"17","author":"Uria","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref_27","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Masked Autoregressive Flow for Density Estimation. Advances in Neural Information Processing Systems 30, Curran Associates, Inc."},{"key":"ref_28","unstructured":"Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., and Garnett, R. (2016). Improved Variational Inference with Inverse Autoregressive Flow. Advances in Neural Information Processing Systems 29, Curran Associates, Inc."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"13:1","DOI":"10.1145\/2728167","article-title":"Finding Correlations in Subquadratic Time, with Applications to Learning Parities and the Closest Pair Problem","volume":"62","author":"Valiant","year":"2015","journal-title":"J. ACM"},{"key":"ref_30","unstructured":"Minsky, M., and Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry, MIT Press."},{"key":"ref_31","unstructured":"Brutzkus, A., and Globerson, A. (2018). Over-parameterization Improves Generalization in the XOR Detection Problem. arXiv."},{"key":"ref_32","unstructured":"Mhaskar, H., Liao, Q., and Poggio, T.A. (2016). Learning Real and Boolean Functions: When Is Deep Better Than Shallow. arXiv."},{"key":"ref_33","unstructured":"Nye, M., and Saxe, A. (2018). Are Efficient Deep Representations Learnable?. arXiv."},{"key":"ref_34","unstructured":"MacWilliams, F., and Sloane, N. (1978). The Theory of Error-Correcting Codes, North-holland Publishing Company. [2nd ed.]."},{"key":"ref_35","first-page":"1","article-title":"Searching for Collective Behavior in a Large Network of Sensory Neurons","volume":"10","author":"Marre","year":"2014","journal-title":"PLoS Comput. Biol."},{"key":"ref_36","unstructured":"Marre, O., Tkacik, G., Amodei, D., Schneidman, E., Bialek, W., and Berry, M. (2017). Multi-Electrode Array Recording from Salamander Retinal Ganglion Cells, IST Austria."},{"key":"ref_37","unstructured":"Koch, C. (2004). Biophysics of Computation: Information Processing in Single Neurons, Oxford University Press, Inc."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"e04250","DOI":"10.7554\/eLife.04250","article-title":"Automatic discovery of cell types and microcircuitry from neural connectomics","volume":"4","author":"Jonas","year":"2015","journal-title":"Elife"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"4207","DOI":"10.1523\/JNEUROSCI.4697-04.2005","article-title":"Redundancy and synergy of neuronal ensembles in motor cortex","volume":"25","author":"Narayanan","year":"2005","journal-title":"J. Neurosci."},{"key":"ref_40","unstructured":"Pica, G., Piasini, E., Safaai, H., Runyan, C.A., Diamond, M.E., Fellin, T., Kayser, C., Harvey, C.D., and Panzeri, S. (2017, January 4\u20139). Quantifying How Much Sensory Information in a Neural Code is Relevant for Behavior. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1985","DOI":"10.3390\/e16041985","article-title":"Intersection information based on common randomness","volume":"16","author":"Griffith","year":"2014","journal-title":"Entropy"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Bertschinger, N., Rauh, J., Olbrich, E., Jost, J., and Ay, N. (2013). Quantifying unique information. arXiv.","DOI":"10.3390\/e16042161"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Ince, R. (2017). Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal. Entropy, 19.","DOI":"10.3390\/e19070318"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Finn, C., and Lizier, J.T. (2018). Pointwise Information Decomposition Using the Specificity and Ambiguity Lattices. arXiv.","DOI":"10.3390\/e20040297"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Rauh, J. (2017). Secret Sharing and Shared Information. arXiv.","DOI":"10.3390\/e19110601"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Makkeh, A., Theis, D.O., and Vicente, R. (2017). Bivariate Partial Information Decomposition: The Optimization Perspective. Entropy, 19.","DOI":"10.3390\/e19100530"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Makkeh, A., Theis, D.O., and Vicente, R. (2018). BROJA-2PID: A Robust Estimator for Bivariate Partial Information Decomposition. Entropy, 20.","DOI":"10.3390\/e20040271"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Makkeh, A., Chicharro, D., Theis, D.O., and Vicente, R. (2019). MAXENT3D PID: An Estimator for the Maximum-Entropy Trivariate Partial Information Decomposition. Entropy, 21.","DOI":"10.3390\/e21090862"},{"key":"ref_49","unstructured":"Kolchinsky, A. (2020). A novel approach to multivariate redundancy and synergy. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/1\/79\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:08:14Z","timestamp":1760159294000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/1\/79"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,7]]},"references-count":49,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,1]]}},"alternative-id":["e23010079"],"URL":"https:\/\/doi.org\/10.3390\/e23010079","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2021,1,7]]}}}