{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T22:20:51Z","timestamp":1774650051120,"version":"3.50.1"},"reference-count":48,"publisher":"MIT Press - Journals","issue":"10","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,9,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We study the type of distributions that restricted Boltzmann machines (RBMs) with different activation functions can express by investigating the effect of the activation function of the hidden nodes on the marginal distribution they impose on observed binary nodes. We report an exact expression for these marginals in the form of a model of interacting binary variables with the explicit form of the interactions depending on the hidden node activation function. We study the properties of these interactions in detail and evaluate how the accuracy with which the RBM approximates distributions over binary variables depends on the hidden node activation function and the number of hidden nodes. When the inferred RBM parameters are weak, an intuitive pattern is found for the expression of the interaction terms, which reduces substantially the differences across activation functions. We show that the weak parameter approximation is a good approximation for different RBMs trained on the MNIST data set. Interestingly, in these cases, the mapping reveals that the inferred models are essentially low order interaction models.<\/jats:p>","DOI":"10.1162\/neco_a_01420","type":"journal-article","created":{"date-parts":[[2021,7,19]],"date-time":"2021-07-19T20:44:48Z","timestamp":1626727488000},"page":"2646-2681","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":13,"title":["Restricted Boltzmann Machines as Models of Interacting Variables"],"prefix":"10.1162","volume":"33","author":[{"given":"Nicola","family":"Bulso","sequence":"first","affiliation":[{"name":"Kavli Institute for Systems Neuroscience and Centre for Neural Computation, Norwegian University of Science and Technology, 7491 Trondheim, Norway, and SISSA\u2014Cognitive Neuroscience, 34136 Trieste, Italy nicola.bulso@ntnu.no"}]},{"given":"Yasser","family":"Roudi","sequence":"additional","affiliation":[{"name":"Kavli Institute for Systems Neuroscience and Centre for Neural Computation, Norwegian University of Science and Technology, 7491 Trondheim, Norway yasser.roudi@ntnu.no"}]}],"member":"281","published-online":{"date-parts":[[2021,9,16]]},"reference":[{"issue":"1","key":"2021091622574985500_B1","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1207\/s15516709cog0901_7","article-title":"A learning algorithm for Boltzmann machines","volume":"9","author":"Ackley","year":"1985","journal-title":"Cognitive Science"},{"issue":"5","key":"2021091622574985500_B2","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1038\/nrn1888","article-title":"Neural correlations, population coding and computation","volume":"7","author":"Averbeck","year":"2006","journal-title":"Nature Reviews Neuroscience"},{"issue":"1","key":"2021091622574985500_B3","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1073\/pnas.1908636117","article-title":"Shaping the learning landscape in neural networks around wide flat minima","volume":"117","author":"Baldassi","year":"2020","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2021091622574985500_B4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neunet.2012.06.003","article-title":"On the equivalence of Hopfield networks and Boltzmann machines","volume":"34","author":"Barra","year":"2012","journal-title":"Neural Networks"},{"issue":"4","key":"2021091622574985500_B5","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevE.96.042156","article-title":"Phase transitions in restricted Boltzmann machines with generic priors","volume":"96","author":"Barra","year":"2017","journal-title":"Physical Review E"},{"issue":"2","key":"2021091622574985500_B6","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevE.97.022310","article-title":"Phase diagram of restricted Boltzmann machines and generalized Hopfield networks with arbitrary priors","volume":"97","author":"Barra","year":"2018","journal-title":"Physical Review E"},{"key":"2021091622574985500_B7","author":"Barrat","year":"1997","journal-title":"The p-spin spherical spin glass model"},{"key":"2021091622574985500_B8","doi-asserted-by":"crossref","first-page":"1721","DOI":"10.1145\/2783258.2788613","article-title":"Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.","author":"Caruana","year":"2015","journal-title":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining"},{"issue":"3","key":"2021091622574985500_B9","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1137\/060661569","article-title":"Symmetric tensors and symmetric tensor rank","volume":"30","author":"Comon","year":"2008","journal-title":"SIAM Journal on Matrix Analysis and Applications"},{"issue":"3","key":"2021091622574985500_B10","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1145\/765568.765570","article-title":"A differential approach to inference in Bayesian networks","volume":"50","author":"Darwiche","year":"2003","journal-title":"Journal of the ACM"},{"issue":"3","key":"2021091622574985500_B11","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/BF02288367","article-title":"The approximation of one matrix by another of lower rank","volume":"1","author":"Eckart","year":"1936","journal-title":"Psychometrika"},{"issue":"1","key":"2021091622574985500_B12","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.patcog.2013.05.025","article-title":"Training restricted Boltzmann machines: An introduction","volume":"47","author":"Fischer","year":"2014","journal-title":"Pattern Recognition"},{"key":"2021091622574985500_B13","first-page":"1136","article-title":"Input switched affine networks: An RNN architecture designed for interpretability.","volume":"70","author":"Foerster","year":"2017","journal-title":"Proceedings of the 34th International Conference on Machine Learning"},{"key":"2021091622574985500_B14","first-page":"912","volume-title":"Advances in neural information processing systems","author":"Freund","year":"1992"},{"key":"2021091622574985500_B15","doi-asserted-by":"crossref","first-page":"3267","DOI":"10.1073\/pnas.1718710115","article-title":"Blindfold learning of an accurate neural metric","volume":"115","author":"Gardella","year":"2018","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2021091622574985500_B16","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1145\/1143844.1143887","article-title":"The rate adapting Poisson model for information retrieval and object recognition.","author":"Gehler","year":"2006","journal-title":"Proceedings of the 23rd International Conference on Machine Learning"},{"key":"2021091622574985500_B17","author":"Hertz","year":"1991","journal-title":"Introduction to the theory of neural computation"},{"issue":"8","key":"2021091622574985500_B18","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1162\/089976602760128018","article-title":"Training products of experts by minimizing contrastive divergence","volume":"14","author":"Hinton","year":"2002","journal-title":"Neural Computation"},{"key":"2021091622574985500_B19","first-page":"1","volume-title":"Encyclopedia of machine learning and data mining","author":"Hinton","year":"2014"},{"key":"2021091622574985500_B20","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1007\/978-3-642-35289-8_32","volume-title":"Neural networks: Tricks of the trade","author":"Hinton","year":"2012"},{"issue":"7","key":"2021091622574985500_B21","doi-asserted-by":"crossref","first-page":"1527","DOI":"10.1162\/neco.2006.18.7.1527","article-title":"A fast learning algorithm for deep belief nets","volume":"18","author":"Hinton","year":"2006","journal-title":"Neural Computation"},{"issue":"5","key":"2021091622574985500_B22","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pbio.2003663","article-title":"A novel unsupervised analysis of electrophysiological signals reveals new sleep substages in mice","volume":"16","author":"Katsageorgiou","year":"2018","journal-title":"PLOS Biology"},{"issue":"7","key":"2021091622574985500_B23","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1003684","article-title":"Modeling higher-order correlations within cortical microcolumns","volume":"10","author":"K\u00f6ster","year":"2014","journal-title":"PLOS Computational Biology"},{"key":"2021091622574985500_B24","first-page":"643","article-title":"Learning algorithms for the classification restricted Boltzmann machine","volume":"13","author":"Larochelle","year":"2012","journal-title":"Journal of Machine Learning Research"},{"key":"2021091622574985500_B25","volume-title":"Principles of neural coding","author":"Latham","year":"2013"},{"issue":"6","key":"2021091622574985500_B26","doi-asserted-by":"crossref","first-page":"1631","DOI":"10.1162\/neco.2008.04-07-510","article-title":"Representational power of restricted Boltzmann machines and deep belief networks","volume":"20","author":"Le Roux","year":"2008","journal-title":"Neural Computation"},{"issue":"10","key":"2021091622574985500_B27","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1145\/3233231","article-title":"The mythos of model interpretability.","volume":"61","author":"Lipton","year":"2018","journal-title":"Commun. ACM"},{"issue":"1","key":"2021091622574985500_B28","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1007\/BF01210677","article-title":"Sur une propri\u00e9t\u00e9 de la loi de gauss","volume":"44","author":"Marcinkiewicz","year":"1939","journal-title":"Mathematische Zeitschrift"},{"key":"2021091622574985500_B29","first-page":"2877","volume-title":"Advances in neural information processing systems","author":"Martens","year":"2013"},{"key":"2021091622574985500_B30","first-page":"517","article-title":"Parallelizable sampling of Markov random fields.","author":"Martens","year":"2010","journal-title":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics"},{"issue":"1","key":"2021091622574985500_B31","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/BF00199057","article-title":"Detecting higher-order interactions among the spiking events in a group of neurons","volume":"73","author":"Martignon","year":"1995","journal-title":"Biological Cybernetics"},{"issue":"1","key":"2021091622574985500_B32","doi-asserted-by":"crossref","DOI":"10.3390\/e23010034","article-title":"Boltzmann machines as generalized Hopfield networks: A review of recent results and outlooks","volume":"23","author":"Marullo","year":"2021","journal-title":"Entropy"},{"key":"2021091622574985500_B33","author":"Murphy","year":"2012","journal-title":"Machine learning: A probabilistic perspective"},{"key":"2021091622574985500_B34","first-page":"807","article-title":"Rectified linear units improve restricted Boltzmann machines.","author":"Nair","year":"2010","journal-title":"Proceedings of the 27th International Conference on Machine Learning"},{"issue":"2","key":"2021091622574985500_B35","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1023\/A:1008923215028","article-title":"Annealed importance sampling","volume":"11","author":"Neal","year":"2001","journal-title":"Statistics and Computing"},{"key":"2021091622574985500_B36","article-title":"Effect of activation functions on the training of overparameterized neural nets.","author":"Panigrahi","year":"2019","journal-title":"Proceedings of the International Conference on Learning Representations."},{"issue":"5","key":"2021091622574985500_B37","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevA.10.1852","article-title":"Some generalizations of the Marcinkiewicz theorem and its implications to certain approximation schemes in many-particle physics","volume":"10","author":"Rajagopal","year":"1974","journal-title":"Physical Review A"},{"key":"2021091622574985500_B38","first-page":"809","article-title":"Stochastic neural networks with monotonic activation functions.","author":"Ravanbakhsh","year":"2016","journal-title":"Proceedings of the 19th International Conference on Artificial Intelligence and Statistics"},{"key":"2021091622574985500_B39","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.conb.2015.07.006","article-title":"Learning with hidden variables","volume":"35","author":"Roudi","year":"2015","journal-title":"Current Opinion in Neurobiology"},{"key":"2021091622574985500_B40","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1145\/1273496.1273596","article-title":"Restricted Boltzmann machines for collaborative filtering.","author":"Salakhutdinov","year":"2007","journal-title":"Proceedings of the 24th International Conference on Machine Learning"},{"key":"2021091622574985500_B41","first-page":"872","article-title":"On the quantitative analysis of deep belief networks.","author":"Salakhutdinov","year":"2008","journal-title":"Proceedings of the 25th International Conference on Machine Learning"},{"issue":"1","key":"2021091622574985500_B42","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevLett.126.018301","article-title":"Efficiency of local learning rules in threshold-linear associative networks","volume":"126","author":"Sch\u00f6nsberg","year":"2021","journal-title":"Physical Review Letters"},{"key":"2021091622574985500_B43","author":"Smolensky","year":"1986","journal-title":"Information processing in dynamical systems: Foundations of harmony theory"},{"issue":"8","key":"2021091622574985500_B44","doi-asserted-by":"crossref","first-page":"1671","DOI":"10.1162\/neco_a_01210","article-title":"Learning compositional representations of interacting systems with restricted Boltzmann machines: Comparative study of lattice proteins","volume":"31","author":"Tubiana","year":"2019","journal-title":"Neural Computation"},{"issue":"13","key":"2021091622574985500_B45","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevLett.118.138301","article-title":"Emergence of compositional representations in restricted Boltzmann machines","volume":"118","author":"Tubiana","year":"2017","journal-title":"Physical Review Letters"},{"key":"2021091622574985500_B46","first-page":"1481","volume-title":"Advances in neural information processing systems, 17","author":"Welling","year":"2005"},{"issue":"3","key":"2021091622574985500_B47","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevE.99.032113","article-title":"Transforming generalized Ising models into Boltzmann machines","volume":"99","author":"Yoshioka","year":"2019","journal-title":"Physical Review E"},{"key":"2021091622574985500_B48","author":"Zanotto","year":"2017","journal-title":"Modeling retinal ganglion cell population activity with restricted Boltzmann machines."}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/10\/2646\/1963307\/neco_a_01420.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/10\/2646\/1963307\/neco_a_01420.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,16]],"date-time":"2021-09-16T22:58:47Z","timestamp":1631833127000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/33\/10\/2646\/103014\/Restricted-Boltzmann-Machines-as-Models-of"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,16]]},"references-count":48,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,9,16]]},"published-print":{"date-parts":[[2021,9,16]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01420","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,10]]},"published":{"date-parts":[[2021,9,16]]}}}