{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T02:27:47Z","timestamp":1790044067713,"version":"4.0.1"},"reference-count":24,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2021,7,20]],"date-time":"2021-07-20T00:00:00Z","timestamp":1626739200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, known as the \u201credundant information\u201d. We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, we show that optimizing over functions enables the approximation of the redundant information for high-dimensional and continuous predictors. We demonstrate this on high-dimensional image classification and motor-neuroscience tasks.<\/jats:p>","DOI":"10.3390\/e23070922","type":"journal-article","created":{"date-parts":[[2021,7,20]],"date-time":"2021-07-20T11:26:10Z","timestamp":1626780370000},"page":"922","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Redundant Information Neural Estimation"],"prefix":"10.3390","volume":"23","author":[{"given":"Michael","family":"Kleinman","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Achille","sequence":"additional","affiliation":[{"name":"Department of Computational and Mathematical Sciences, Caltech, Pasadena, CA 91125, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefano","family":"Soatto","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of California, Los Angeles, CA 90095, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9298-0143","authenticated-orcid":false,"given":"Jonathan C.","family":"Kao","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90095, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,20]]},"reference":[{"key":"ref_1","unstructured":"Williams, P.L., and Beer, R.D. (2010). Nonnegative decomposition of multivariate information. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1985","DOI":"10.3390\/e16041985","article-title":"Intersection Information Based on Common Randomness","volume":"16","author":"Griffith","year":"2014","journal-title":"Entropy"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2161","DOI":"10.3390\/e16042161","article-title":"Quantifying Unique Information","volume":"16","author":"Bertschinger","year":"2014","journal-title":"Entropy"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"012130","DOI":"10.1103\/PhysRevE.87.012130","article-title":"Bivariate measure of redundant information","volume":"87","author":"Harder","year":"2013","journal-title":"Phys. Rev. E"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4644","DOI":"10.3390\/e17074644","article-title":"Quantifying Redundant Information in Predicting a Target Random Variable","volume":"17","author":"Griffith","year":"2015","journal-title":"Entropy"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Banerjee, P.K., Rauh, J., and Mont\u00fafar, G. (2018, January 17\u201322). Computing the Unique Information. Proceedings of the 2018 IEEE International Symposium on Information Theory (ISIT), Vail, CO, USA.","DOI":"10.1109\/ISIT.2018.8437757"},{"key":"ref_7","unstructured":"Kolchinsky, A. (2019). A novel approach to multivariate redundancy and synergy. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"052802","DOI":"10.1103\/PhysRevE.91.052802","article-title":"Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems","volume":"91","author":"Barrett","year":"2015","journal-title":"Phys. Rev. E"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Faes, L., Marinazzo, D., and Stramaglia, S. (2017). Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes. Entropy, 19.","DOI":"10.3390\/e19080408"},{"key":"ref_10","unstructured":"Banerjee, P.K., and Griffith, V. (2015). Synergy, redundancy and common information. arXiv."},{"key":"ref_11","unstructured":"Xu, Y., Zhao, S., Song, J., Stewart, R., and Ermon, S. (2020, January 26\u201330). A Theory of Usable Information under Computational Constraints. Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia."},{"key":"ref_12","first-page":"18674","article-title":"Learning Optimal Representations with the Decodable Information Bottleneck","volume":"Volume 33","author":"Larochelle","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_13","unstructured":"Kleinman, M., Achille, A., Idnani, D., and Kao, J. (2021, January 4). Usable Information and Evolution of Optimal Representations During Training. Proceedings of the International Conference on Learning Representations, Vienna, Austria."},{"key":"ref_14","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_16","unstructured":"Burgess, C.P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. (2018). Understanding disentangling in \u03b2-VAE. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1038\/nature04968","article-title":"A high-performance brain\u2014Computer interface","volume":"442","author":"Santhanam","year":"2006","journal-title":"Nature"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1146\/annurev-neuro-062111-150509","article-title":"Cortical control of arm movements: A dynamical systems perspective","volume":"36","author":"Shenoy","year":"2013","journal-title":"Annu. Rev. Neurosci."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1315","DOI":"10.1152\/jn.00097.2009","article-title":"Factor-analysis methods for higher-performance neural prostheses","volume":"102","author":"Santhanam","year":"2009","journal-title":"J. Neurophysiol."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Gao, P., Trautmann, E., Yu, B., Santhanam, G., Ryu, S., Shenoy, K., and Ganguli, S. (2017). A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, 214262.","DOI":"10.1101\/214262"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"066003","DOI":"10.1088\/1741-2560\/11\/6\/066003","article-title":"Performance sustaining intracortical neural prostheses","volume":"11","author":"Nuyujukian","year":"2014","journal-title":"J. Neural Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1038\/nature11129","article-title":"Neural population dynamics during reaching","volume":"487","author":"Churchland","year":"2012","journal-title":"Nature"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1145\/1968.1972","article-title":"A theory of the learnable","volume":"27","author":"Valiant","year":"1984","journal-title":"Commun. ACM"},{"key":"ref_24","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley-Interscience."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/7\/922\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:32:20Z","timestamp":1760164340000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/7\/922"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,20]]},"references-count":24,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,7]]}},"alternative-id":["e23070922"],"URL":"https:\/\/doi.org\/10.3390\/e23070922","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,20]]}}}