{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:31:56Z","timestamp":1760236316173,"version":"build-2065373602"},"reference-count":63,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,11,11]],"date-time":"2021-11-11T00:00:00Z","timestamp":1636588800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>In 1943, McCulloch and Pitts introduced a discrete recurrent neural network as a model for computation in brains. The work inspired breakthroughs such as the first computer design and the theory of finite automata. We focus on learning in Hopfield networks, a special case with symmetric weights and fixed-point attractor dynamics. Specifically, we explore minimum energy flow (MEF) as a scalable convex objective for determining network parameters. We catalog various properties of MEF, such as biological plausibility, and then compare to classical approaches in the theory of learning. Trained Hopfield networks can perform unsupervised clustering and define novel error-correcting coding schemes. They also efficiently find hidden structures (cliques) in graph theory. We extend this known connection from graphs to hypergraphs and discover n-node networks with robust storage of 2\u03a9(n1\u2212\u03f5) memories for any \u03f5&gt;0. In the case of graphs, we also determine a critical ratio of training samples at which networks generalize completely.<\/jats:p>","DOI":"10.3390\/e23111494","type":"journal-article","created":{"date-parts":[[2021,11,11]],"date-time":"2021-11-11T23:02:41Z","timestamp":1636671761000},"page":"1494","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Hidden Hypergraphs, Error-Correcting Codes, and Critical Learning in Hopfield Networks"],"prefix":"10.3390","volume":"23","author":[{"given":"Christopher","family":"Hillar","sequence":"first","affiliation":[{"name":"Awecom, Inc., San Francisco, CA 94103, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tenzin","family":"Chan","sequence":"additional","affiliation":[{"name":"Singapore University of Technology and Design, Singapore 487372, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rachel","family":"Taubman","sequence":"additional","affiliation":[{"name":"Awecom, Inc., San Francisco, CA 94103, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Rolnick","sequence":"additional","affiliation":[{"name":"School of Computer Science, McGill University, Montreal, QC H3A 0G4, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1007\/BF02478259","article-title":"A logical calculus of the ideas immanent in nervous activity","volume":"5","author":"McCulloch","year":"1943","journal-title":"Bull. Math. Biophys."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1023\/B:SYNT.0000043018.52445.3e","article-title":"The First computational theory of mind and brain: A close look at McCulloch and Pitts\u2019s \u201cA logical calculus of ideas immanent in nervous activity\u201d","volume":"141","author":"Piccinini","year":"2004","journal-title":"Synthese"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1109\/85.238389","article-title":"First draft of a report on the EDVAC","volume":"15","year":"1993","journal-title":"IEEE Ann. Hist. Comput."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2554","DOI":"10.1073\/pnas.79.8.2554","article-title":"Neural networks and physical systems with emergent collective computational abilities","volume":"79","author":"Hopfield","year":"1982","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/0025-5564(74)90031-5","article-title":"The existence of persistent states in the brain","volume":"19","author":"Little","year":"1974","journal-title":"Math. Biosci."},{"key":"ref_6","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G. (2021, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the Twenty-Sixth Annual Conference on Neural Information Processing Systems (NIPS), Lake Tahoe, NV, USA."},{"key":"ref_7","first-page":"1","article-title":"Do deep nets really need to be deep?","volume":"27","author":"Ba","year":"2014","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"ref_8","unstructured":"Ramsauer, H., Sch\u00e4fl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlovi\u0107, M., and Sandve, G.K. (2021). Hopfield networks is all you need. arXiv."},{"key":"ref_9","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_10","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An image is worth 16 \u00d7 16 words: Transformers for image recognition at scale. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"974","DOI":"10.1038\/s41593-019-0392-5","article-title":"Evidence that recurrent circuits are critical to the ventral stream\u2019s execution of core object recognition behavior","volume":"22","author":"Kar","year":"2019","journal-title":"Nat. Neurosci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"21854","DOI":"10.1073\/pnas.1905544116","article-title":"Recurrence is required to capture the representational dynamics of the human visual system","volume":"116","author":"Kietzmann","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e00596","DOI":"10.1016\/j.heliyon.2018.e00596","article-title":"An introduction to the maximum entropy approach and its application to inference problems in biology","volume":"4","year":"2018","journal-title":"Heliyon"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1038\/nature04701","article-title":"Weak pairwise correlations imply strongly correlated network states in a neural population","volume":"440","author":"Schneidman","year":"2006","journal-title":"Nature"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"5022","DOI":"10.1523\/JNEUROSCI.5187-08.2009","article-title":"The structure of large-scale synchronized firing in primate retina","volume":"29","author":"Shlens","year":"2009","journal-title":"J. Neurosci."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/BF02980577","article-title":"Beitrag zur Theorie des Ferromagnetismus","volume":"31","author":"Ising","year":"1925","journal-title":"Z. Phys."},{"key":"ref_17","unstructured":"Fiete, I., Schwab, D.J., and Tran, N.M. (2014). A binary Hopfield network with 1\/log(n) information rate and applications to grid cell decoding. arXiv."},{"key":"ref_18","unstructured":"Chaudhuri, R., and Fiete, I. (2017). Associative content-addressable networks with exponentially many robust stable states. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13408-017-0056-2","article-title":"Robust exponential memory in Hopfield networks","volume":"8","author":"Hillar","year":"2018","journal-title":"J. Math. Neurosci."},{"key":"ref_20","first-page":"1","article-title":"Bipartite expander Hopfield networks as self-decoding high-capacity error correcting codes","volume":"32","author":"Chaudhuri","year":"2019","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"ref_21","unstructured":"Hillar, C., Sohl-Dickstein, J., and Koepsell, K. (2012). Efficient and optimal binary Hopfield associative memory storage using minimum probability flow. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2483","DOI":"10.1162\/0899766042321751","article-title":"How many clusters? An information-theoretic perspective","volume":"16","author":"Still","year":"2004","journal-title":"Neural Comput."},{"key":"ref_23","unstructured":"Li, Y., Hu, P., Liu, Z., Peng, D., Zhou, J.T., and Peng, X. (2021, January 2\u20139). Contrastive clustering. Proceedings of the 2021 AAAI Conference on Artificial Intelligence (AAAI), Vancouver, BC, Canada."},{"key":"ref_24","first-page":"697","article-title":"Clustering hidden Markov models with variational HEM","volume":"15","author":"Coviello","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_25","unstructured":"Lan, H., Liu, Z., Hsiao, J.H., Yu, D., and Chan, A.B. (2021). Clustering hidden Markov models with variational Bayesian hierarchical EM. IEEE Trans. Neural Netw. Learn. Syst., 1\u201315."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Andriyanov, N. (2021). Methods for preventing visual attacks in convolutional neural networks based on data discard and dimensionality reduction. Appl. Sci., 11.","DOI":"10.3390\/app11115235"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Andriyanov, N., and Andriyanov, D. (2021, January 13\u201315). Intelligent processing of voice messages in civil aviation: Message recognition and the emotional state of the speaker analysis. Proceedings of the 2021 International Siberian Conference on Control and Communications (SIBCON), Kazan, Russia.","DOI":"10.1109\/SIBCON50419.2021.9438881"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/584091.584093","article-title":"A mathematical theory of communication","volume":"5","author":"Shannon","year":"2001","journal-title":"ACM SIGMOBILE Mob. Comput. Commun. Rev."},{"key":"ref_29","first-page":"264","article-title":"Neural approach for TV image compression using a Hopfield type network","volume":"1","author":"Naillon","year":"1989","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Hillar, C., Mehta, R., and Koepsell, K. (2014, January 27\u201330). A Hopfield recurrent neural network trained on natural images performs state-of-the-art image compression. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.","DOI":"10.1109\/ICIP.2014.7025831"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hillar, C., and Marzen, S. (2017, January 4\u20137). Revisiting perceptual distortion for natural images: Mean discrete structural similarity index. Proceedings of the 2017 Data Compression Conference (DCC), Snowbird, UT, USA.","DOI":"10.1109\/DCC.2017.84"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mehta, R., Marzen, S., and Hillar, C. (September, January 31). Exploring discrete approaches to lossy compression schemes for natural image patches. Proceedings of the 2015 23rd European Signal Processing Conference (EUSIPCO), Nice, France.","DOI":"10.1109\/EUSIPCO.2015.7362782"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1090\/conm\/685\/13814","article-title":"Neural network coding of natural images with applications to pure mathematics","volume":"Volume 685","author":"Hillar","year":"2017","journal-title":"Contemporary Mathematics"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1016\/j.procs.2015.07.313","article-title":"Robust discovery of temporal structure in multi-neuron recordings using Hopfield networks","volume":"53","author":"Hillar","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Effenberger, F., and Hillar, C. (2015, January 12\u201314). Discovery of salient low-dimensional dynamical structure in neuronal population activity using Hopfield networks. Proceedings of the International Workshop on Similarity-Based Pattern Recognition, Copenhagen, Denmark.","DOI":"10.1007\/978-3-319-24261-3_16"},{"key":"ref_36","unstructured":"Hillar, C., and Effenberger, F. (2015, July 15). hdnet\u2014A Python Package for Parallel Spike Train Analysis. Available online: https:\/\/github.com\/team-hdnet\/hdnet."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1007\/BF00339943","article-title":"\u201cNeural\u201d computation of decisions in optimization problems","volume":"52","author":"Hopfield","year":"1985","journal-title":"Biol. Cybern."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"5","DOI":"10.3389\/fphy.2014.00005","article-title":"Ising formulations of many NP problems","volume":"2","author":"Lucas","year":"2014","journal-title":"Front. Phys."},{"key":"ref_39","unstructured":"Boothby, K., Bunyk, P., Raymond, J., and Roy, A. (2020). Next-generation topology of d-wave quantum processors. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1017\/S096354831300045X","article-title":"Finding hidden cliques in linear time with high probability","volume":"23","author":"Dekel","year":"2014","journal-title":"Comb. Probab. Comput."},{"key":"ref_41","unstructured":"Hebb, D. (1949). The Organization of Behavior, Wiley."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1197","DOI":"10.1109\/T-C.1972.223477","article-title":"Learning patterns and pattern sequences by self-organizing nets of threshold elements","volume":"100","author":"Amari","year":"1972","journal-title":"IEEE Trans. Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1037\/h0042519","article-title":"The perceptron: A probabilistic model for information storage and organization in the brain","volume":"65","author":"Rosenblatt","year":"1958","journal-title":"Psychol. Rev."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Widrow, B., and Hoff, M.E. (1960). Adaptive Switching Circuits, Stanford University Ca Stanford Electronics Labs. Technical Report.","DOI":"10.21236\/AD0241531"},{"key":"ref_45","unstructured":"Black, A.H., and Prokasy, W.F. (1972). A theory of Pavlovian conditioning: Variations on the effectiveness of reinforcement and non-reinforcement. Classical Conditioning II: Current Research and Theory, Appleton-Century-Crofts."},{"key":"ref_46","first-page":"282","article-title":"Learning and relearning in Boltzmann machines","volume":"1","author":"Hinton","year":"1986","journal-title":"Parallel Distrib. Process. Explor. Microstruct. Cogn."},{"key":"ref_47","unstructured":"Cover, T., and Thomas, J. (2006). Elements of Information Theory, Wiley-Interscience. [2nd ed.]."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1109\/TPAMI.1984.4767596","article-title":"Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images","volume":"6","author":"Geman","year":"1984","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1400","DOI":"10.1214\/10-AAP728","article-title":"Random graphs with a given degree sequence","volume":"21","author":"Chatterjee","year":"2011","journal-title":"Ann. Appl. Probab."},{"key":"ref_50","unstructured":"Hillar, C., and Wibisono, A. (2013). Maximum entropy distributions on graphs. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Boyd, S., Boyd, S.P., and Vandenberghe, L. (2004). Convex Optimization, Cambridge University Press.","DOI":"10.1017\/CBO9780511804441"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1007\/s10994-007-5016-8","article-title":"Logarithmic regret algorithms for online convex optimization","volume":"69","author":"Hazan","year":"2007","journal-title":"Mach. Learn."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"892","DOI":"10.1038\/36103","article-title":"Activity-dependent scaling of quantal amplitude in neocortical neurons","volume":"391","author":"Turrigiano","year":"1998","journal-title":"Nature"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1017\/S0305004100027419","article-title":"Some generalized order-disorder transformations","volume":"Volume 48","author":"Potts","year":"1952","journal-title":"Mathematical Proceedings of the Cambridge Philosophical Society"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"220601","DOI":"10.1103\/PhysRevLett.107.220601","article-title":"New method for parameter estimation in probabilistic models: Minimum probability flow","volume":"107","author":"Battaglino","year":"2011","journal-title":"Phys. Rev. Lett."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"2987","DOI":"10.1152\/jn.01023.2004","article-title":"Polytrodes: High-density silicon electrode arrays for large-scale multiunit recording","volume":"93","author":"Blanche","year":"2005","journal-title":"J. Neurophysiol."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pcbi.1006283","article-title":"Unsupervised clustering of temporal patterns in high-dimensional neuronal ensembles using a novel dissimilarity measure","volume":"14","author":"Grossberger","year":"2018","journal-title":"PLoS Comput. Biol."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1080\/01621459.1963.10500830","article-title":"Probability inequalities for sums of bounded random variables","volume":"58","author":"Hoeffding","year":"1963","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_59","first-page":"357","article-title":"Weighted sums of certain dependent random variables","volume":"19","author":"Azuma","year":"1967","journal-title":"Tohoku Math. J. Second. Ser."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Liu, Z., Chotibut, T., Hillar, C., and Lin, S. (2020, January 7\u201312). Biologically plausible sequence learning with spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i02.5487"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1145\/1968.1972","article-title":"A theory of the learnable","volume":"27","author":"Valiant","year":"1984","journal-title":"Commun. ACM"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1007\/s10955-011-0229-4","article-title":"Are biological systems poised at criticality?","volume":"144","author":"Mora","year":"2011","journal-title":"J. Stat. Phys."},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Del Papa, B., Priesemann, V., and Triesch, J. (2017). Criticality meets learning: Criticality signatures in a self-organizing recurrent neural network. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0178683"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/11\/1494\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:28:40Z","timestamp":1760167720000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/11\/1494"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,11]]},"references-count":63,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["e23111494"],"URL":"https:\/\/doi.org\/10.3390\/e23111494","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2021,11,11]]}}}