{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T08:56:03Z","timestamp":1784105763694,"version":"3.55.0"},"reference-count":45,"publisher":"MIT Press","issue":"8","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,14]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We describe a stochastic, dynamical system capable of inference and learning in a probabilistic latent variable model. The most challenging problem in such models\u2014sampling the posterior distribution over latent variables\u2014is proposed to be solved by harnessing natural sources of stochasticity inherent in electronic and neural systems. We demonstrate this idea for a sparse coding model by deriving a continuous-time equation for inferring its latent variables via Langevin dynamics. The model parameters are learned by simultaneously evolving according to another continuous-time equation, thus bypassing the need for digital accumulators or a global clock. Moreover, we show that Langevin dynamics lead to an efficient procedure for sampling from the posterior distribution in the L0 sparse regime, where latent variables are encouraged to be set to zero as opposed to having a small L1 norm. This allows the model to properly incorporate the notion of sparsity rather than having to resort to a relaxed version of sparsity to make optimization tractable. Simulations of the proposed dynamical system on both synthetic and natural image data sets demonstrate that the model is capable of probabilistically correct inference, enabling learning of the dictionary as well as parameters of the prior.<\/jats:p>","DOI":"10.1162\/neco_a_01505","type":"journal-article","created":{"date-parts":[[2022,7,7]],"date-time":"2022-07-07T23:46:07Z","timestamp":1657237567000},"page":"1676-1700","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["Learning and Inference in Sparse Coding Models With Langevin Dynamics"],"prefix":"10.1162","volume":"34","author":[{"given":"Michael Y.-S.","family":"Fang","sequence":"first","affiliation":[{"name":"Department of Physics, University of California, Berkeley, Berkeley, CA 94720, U.S.A."},{"name":"Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA 94720, U.S.A. michaelfang@berkeley.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mayur","family":"Mudigonda","sequence":"additional","affiliation":[{"name":"Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA 94720, U.S.A. mudigonda@berkeley.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryan","family":"Zarcone","sequence":"additional","affiliation":[{"name":"Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA 94720, U.S.A."},{"name":"Biophysics Graduate Group, University of California, Berkeley, Berkeley, CA 94720, U.S.A. ryanzarcone@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amir","family":"Khosrowshahi","sequence":"additional","affiliation":[{"name":"Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA 94720, U.S.A."},{"name":"Intel Corporation, Santa Clara, CA 95054, U.S.A. khosra@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bruno A.","family":"Olshausen","sequence":"additional","affiliation":[{"name":"Redwood Center for Theoretical Neuroscience, University of California, Berkeley, Berkeley, CA, U.S.A."},{"name":"Helen Wills Neuroscience Institute and School of Optometry, University of California, Berkeley, Berkeley, CA 94720, U.S.A. baolshausen@berkeley.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,7,14]]},"reference":[{"issue":"1","key":"2022071522333189900_B1","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1207\/s15516709cog0901_7","article-title":"A learning algorithm for Boltzmann machines","volume":"9","author":"Ackley","year":"1985","journal-title":"Cognitive Science"},{"issue":"6","key":"2022071522333189900_B2","doi-asserted-by":"publisher","first-page":"751","DOI":"10.2140\/involve.2014.7.751","article-title":"Zero-inflated Poisson (zip) distribution: Parameter estimation and applications to model data from natural calamities","volume":"7","author":"Beckett","year":"2014","journal-title":"Involve: A Journal of Mathematics"},{"issue":"6013","key":"2022071522333189900_B3","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1126\/science.1195870","article-title":"Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment","volume":"331","author":"Berkes","year":"2011","journal-title":"Science"},{"key":"2022071522333189900_B4","first-page":"89","volume-title":"Advances in neural information processing systems, 21","author":"Berkes","year":"2008"},{"issue":"7774","key":"2022071522333189900_B5","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1038\/s41586-019-1557-9","article-title":"Integer factorization using stochastic magnetic tunnel junctions","volume":"573","author":"Borders","year":"2019","journal-title":"Nature"},{"issue":"11","key":"2022071522333189900_B6","doi-asserted-by":"publisher","first-page":"2279","DOI":"10.1162\/neco_a_01325","article-title":"Effect of top-down connections in hierarchical sparse coding","volume":"32","author":"Boutin","year":"2020","journal-title":"Neural Computation"},{"issue":"5","key":"2022071522333189900_B7","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.75.056707","article-title":"Accurate sampling using Langevin dynamics","volume":"75","author":"Bussi","year":"2007","journal-title":"Physical Review E"},{"key":"2022071522333189900_B8","first-page":"300","article-title":"Underdamped Langevin MCMC: A non-asymptotic analysis","volume-title":"Proceedings of the 31st Conference on Learning Theory","author":"Cheng","year":"2018"},{"issue":"5","key":"2022071522333189900_B9","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1109\/JPROC.2021.3067593","article-title":"Advancing neuromorphic computing with Loihi: A survey of results and outlook","volume":"109","author":"Davies","year":"2021","journal-title":"Proceedings of the IEEE"},{"issue":"4","key":"2022071522333189900_B10","doi-asserted-by":"publisher","first-page":"1289","DOI":"10.1109\/TIT.2006.871582","article-title":"Compressed sensing","volume":"52","author":"Donoho","year":"2006","journal-title":"IEEE Transactions on Information Theory"},{"issue":"9","key":"2022071522333189900_B11","doi-asserted-by":"publisher","first-page":"1138","DOI":"10.1038\/s41593-020-0671-1","article-title":"Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference","volume":"23","author":"Echeveste","year":"2020","journal-title":"Nature Neuroscience"},{"key":"2022071522333189900_B12","first-page":"505","volume-title":"Advances in neural information processing systems","author":"Garrigues","year":"2007"},{"key":"2022071522333189900_B13","first-page":"676","volume-title":"Advances in neural information processing systems, 23","author":"Garrigues","year":"2010"},{"key":"2022071522333189900_B14","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The elements of statistical learning: Data mining, inference, and prediction","author":"Hastie","year":"2009"},{"key":"2022071522333189900_B15","volume-title":"Advances in neural information processing systems","author":"Hennequin","year":"2014"},{"issue":"5786","key":"2022071522333189900_B16","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"2022071522333189900_B17","article-title":"Optimal perceptual inference","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Hinton","year":"1983"},{"key":"2022071522333189900_B18","first-page":"293","volume-title":"Advances in neural information processing systems, 16","author":"Hoyer","year":"2003"},{"key":"2022071522333189900_B19","author":"Kingma","year":"2013","journal-title":"Auto-encoding variational Bayes."},{"issue":"1","key":"2022071522333189900_B20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-020-62676-7","article-title":"Memristive GAN in analog","volume":"10","author":"Krestinskaya","year":"2020","journal-title":"Scientific Reports"},{"issue":"7","key":"2022071522333189900_B21","doi-asserted-by":"publisher","first-page":"1434","DOI":"10.1364\/JOSAA.20.001434","article-title":"Hierarchical Bayesian inference in the visual cortex","volume":"20","author":"Lee","year":"2003","journal-title":"JOSA A"},{"issue":"7","key":"2022071522333189900_B22","doi-asserted-by":"publisher","first-page":"1587","DOI":"10.1364\/JOSAA.16.001587","article-title":"Probabilistic framework for the adaptation and comparison of image codes","volume":"16","author":"Lewicki","year":"1999","journal-title":"JOSA A"},{"key":"2022071522333189900_B23","author":"Mansinghka","year":"2014","journal-title":"Building fast Bayesian computing machines out of intentionally stochastic, digital parts."},{"key":"2022071522333189900_B24","volume-title":"Stochastic digital circuits for probabilistic inference","author":"Mansinghka","year":"2008"},{"issue":"1","key":"2022071522333189900_B25","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1016\/0893-6080(88)90024-X","article-title":"A silicon model of early visual processing","volume":"1","author":"Mead","year":"1988","journal-title":"Neural Networks"},{"issue":"404","key":"2022071522333189900_B26","doi-asserted-by":"publisher","first-page":"1023","DOI":"10.1080\/01621459.1988.10478694","article-title":"Bayesian variable selection in linear regression","volume":"83","author":"Mitchell","year":"1988","journal-title":"Journal of the American Statistical Association"},{"issue":"42","key":"2022071522333189900_B27","first-page":"1","article-title":"High-order Langevin diffusion yields an accelerated MCMC algorithm","volume":"22","author":"Mou","year":"2021","journal-title":"J. Mach. Learn. Res."},{"issue":"2","key":"2022071522333189900_B28","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1023\/A:1008923215028","article-title":"Annealed importance sampling","volume":"11","author":"Neal","year":"2001","journal-title":"Statistics and Computing"},{"key":"2022071522333189900_B29","doi-asserted-by":"publisher","first-page":"86510S","DOI":"10.1117\/12.2013504","article-title":"Highly overcomplete sparse coding","volume-title":"Proceedings of Human Vision and Electronic Imaging XVIII","author":"Olshausen","year":"2013"},{"key":"2022071522333189900_B30","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9504.003.0037","article-title":"Perception as an inference problem","volume-title":"The cognitive neurosciences","author":"Olshausen","year":"2014"},{"issue":"23","key":"2022071522333189900_B31","doi-asserted-by":"publisher","first-page":"3311","DOI":"10.1016\/S0042-6989(97)00169-7","article-title":"Sparse coding with an overcomplete basis set: A strategy employed by V1?","volume":"37","author":"Olshausen","year":"1997","journal-title":"Vision Research"},{"key":"2022071522333189900_B32","first-page":"841","volume-title":"Advances in neural information processing systems, 13","author":"Olshausen","year":"2000"},{"issue":"2","key":"2022071522333189900_B33","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1016\/j.neuron.2016.09.038","article-title":"Neural variability and sampling-based probabilistic representations in the visual cortex","volume":"92","author":"Orb\u00e1n","year":"2016","journal-title":"Neuron"},{"key":"2022071522333189900_B34","doi-asserted-by":"crossref","DOI":"10.1364\/CLEO_QELS.2019.FTu4C.2","article-title":"Photonic recurrent Ising sampler","volume-title":"Proceedings of the Conference on Lasters and Electro-Optics","author":"Roques-Carmes","year":"2019"},{"issue":"10","key":"2022071522333189900_B35","doi-asserted-by":"publisher","first-page":"2526","DOI":"10.1162\/neco.2008.03-07-486","article-title":"Sparse coding via thresholding and local competition in neural circuits","volume":"20","author":"Rozell","year":"2008","journal-title":"Neural Computation"},{"issue":"3","key":"2022071522333189900_B36","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1109\/JETCAS.2012.2214615","article-title":"Low power sparse approximation on reconfigurable analog hardware","volume":"2","author":"Shapero","year":"2012","journal-title":"IEEE Journal on Emerging and Selected Topics in Circuits and Systems"},{"key":"2022071522333189900_B37","first-page":"2618","volume-title":"Advances in neural information processing systems","author":"Shelton","year":"2011"},{"issue":"8","key":"2022071522333189900_B38","doi-asserted-by":"publisher","DOI":"10.1038\/nnano.2017.83","article-title":"Sparse coding with memristor networks","volume":"12","author":"Sheridan","year":"2017","journal-title":"Nature Nanotechnology"},{"key":"2022071522333189900_B39","first-page":"719","article-title":"Hamiltonian Monte Carlo without detailed balance","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Sohl-Dickstein","year":"2014"},{"issue":"1","key":"2022071522333189900_B40","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"issue":"3","key":"2022071522333189900_B41","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1016\/j.sigpro.2005.05.031","article-title":"Algorithms for simultaneous sparse approximation. Part II: Convex relaxation","volume":"86","author":"Tropp","year":"2006","journal-title":"Signal Processing"},{"issue":"1394","key":"2022071522333189900_B42","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1098\/rspb.1998.0303","article-title":"Independent component filters of natural images compared with simple cells in primary visual cortex","volume":"265","author":"van Hateren","year":"1998","journal-title":"Proceedings of Biological Sciences"},{"key":"2022071522333189900_B47","doi-asserted-by":"crossref","DOI":"10.1142\/9815","volume-title":"Sparse coding and its applications in computer vision","author":"Wang","year":"2015"},{"key":"2022071522333189900_B43","first-page":"681","article-title":"Bayesian learning via stochastic gradient Langevin dynamics","volume-title":"Proceedings of the 28th International Conference on Machine Learning","author":"Welling","year":"2011"},{"issue":"6","key":"2022071522333189900_B44","doi-asserted-by":"publisher","first-page":"1031","DOI":"10.1109\/JPROC.2010.2044470","article-title":"Sparse representation for computer vision and pattern recognition","volume":"98","author":"Wright","year":"2010","journal-title":"Proceedings of the IEEE"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/34\/8\/1676\/2034932\/neco_a_01505.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/34\/8\/1676\/2034932\/neco_a_01505.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,11]],"date-time":"2023-02-11T03:44:49Z","timestamp":1676087089000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/34\/8\/1676\/111785\/Learning-and-Inference-in-Sparse-Coding-Models"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,14]]},"references-count":45,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2022,7,14]]},"published-print":{"date-parts":[[2022,7,14]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01505","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,8]]},"published":{"date-parts":[[2022,7,14]]}}}