{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T14:06:33Z","timestamp":1724853993161},"reference-count":55,"publisher":"MIT Press","issue":"9","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,8,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We propose an optimization algorithm for variational inference (VI) in complex models. Our approach relies on natural gradient updates where the variational space is a Riemann manifold. We develop an efficient algorithm for gaussian variational inference whose updates satisfy the positive definite constraint on the variational covariance matrix. Our manifold gaussian variational Bayes on the precision matrix (MGVBP) solution provides simple update rules, is straightforward to implement, and the use of the precision matrix parameterization has a significant computational advantage. Due to its black-box nature, MGVBP stands as a ready-to-use solution for VI in complex models. Over five data sets, we empirically validate our feasible approach on different statistical and econometric models, discussing its performance with respect to baseline methods.<\/jats:p>","DOI":"10.1162\/neco_a_01686","type":"journal-article","created":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T18:25:31Z","timestamp":1724178331000},"page":"1744-1798","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Manifold Gaussian Variational Bayes on the Precision Matrix"],"prefix":"10.1162","volume":"36","author":[{"given":"Martin","family":"Magris","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Aarhus University, Aarhus 8200, Denmark"},{"name":"Instituto Tecnol\u00f3gico Aut\u00f3nomo de M\u00e9xico (ITAM), 01080 Ciudad de M\u00e9xico, M\u00e9xico magris@ece.au.dk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mostafa","family":"Shabani","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Aarhus University, Aarhus 8200, Denmark mshabani@ece.au.dk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexandros","family":"Iosifidis","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Aarhus University, Aarhus 8200, Denmark ai@ece.au.dk"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,8,19]]},"reference":[{"key":"2024082018250199900_bib1","doi-asserted-by":"crossref","DOI":"10.1515\/9781400830244","volume-title":"Optimization algorithms on matrix manifolds","author":"Absil","year":"2008"},{"issue":"3","key":"2024082018250199900_bib2","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1007\/s11222-013-9375-7","article-title":"A novel hybrid RBF neural networks model as a forecaster","volume":"24","author":"Akbilgic","year":"2014","journal-title":"Statistics and Computing"},{"key":"2024082018250199900_bib3","author":"Barfoot","year":"2020","journal-title":"Multivariate gaussian variational inference by natural gradient descent"},{"issue":"2","key":"2024082018250199900_bib4","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1016\/j.exmath.2018.01.002","article-title":"On the Bures\u2013Wasserstein distance between positive definite matrices","volume":"37","author":"Bhatia","year":"2019","journal-title":"Expositiones Mathematicae"},{"issue":"518","key":"2024082018250199900_bib5","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1080\/01621459.2017.1285773","article-title":"Variational inference: A review for statisticians","volume":"112","author":"Blei","year":"2017","journal-title":"Journal of the American Statistical Association"},{"key":"2024082018250199900_bib6","first-page":"1613","article-title":"Weight uncertainty in neural network","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Blundell","year":"2015"},{"issue":"42","key":"2024082018250199900_bib7","first-page":"1455","article-title":"Manopt, a Matlab toolbox for optimization on manifolds","volume":"15","author":"Boumal","year":"2014","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2024082018250199900_bib8","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1214\/12-BA703","article-title":"Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies","volume":"7","author":"Carbonetto","year":"2012","journal-title":"Bayesian Analysis"},{"issue":"2","key":"2024082018250199900_bib9","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1093\/jjfinec\/nbp001","article-title":"A simple approximate long-memory model of realized volatility","volume":"7","author":"Corsi","year":"2009","journal-title":"Journal of Financial Econometrics"},{"key":"2024082018250199900_bib10","author":"Ganguly","year":"2021","journal-title":"An introduction to variational inference."},{"key":"2024082018250199900_bib11","article-title":"Practical variational inference for neural networks","volume-title":"Advances in neural information processing systems","author":"Graves","year":"2011"},{"key":"2024082018250199900_bib12","first-page":"8940","article-title":"On Riemannian optimization over positive definite matrices with the Bures-Wasserstein geometry","volume-title":"Advances in neural information processing systems","author":"Han","year":"2021"},{"issue":"40","key":"2024082018250199900_bib13","first-page":"1303","article-title":"Stochastic variational inference","volume":"14","author":"Hoffman","year":"2013","journal-title":"Journal of Machine Learning Research"},{"key":"2024082018250199900_bib14","article-title":"Matrix manifold optimization for gaussian mixtures","volume-title":"Advances in neural information processing systems","author":"Hosseini","year":"2015"},{"key":"2024082018250199900_bib15","first-page":"379","article-title":"A survey and comparison of contemporary algorithms for computing the matrix geometric mean","volume":"39","author":"Jeuris","year":"2012","journal-title":"Electronic Transactions on Numerical Analysis"},{"key":"2024082018250199900_bib16","first-page":"878","article-title":"Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models","volume-title":"Artificial Intelligence and Statistics","author":"Khan","year":"2017"},{"key":"2024082018250199900_bib17","first-page":"31","article-title":"Fast yet simple natural-gradient descent for variational inference in complex models","volume-title":"Proceedings of the 2018 International Symposium on Information Theory and Its Applications","author":"Khan","year":"2018"},{"key":"2024082018250199900_bib18","first-page":"2611","article-title":"Fast and scalable Bayesian deep learning by weight-perturbation in Adam","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Khan","year":"2018"},{"key":"2024082018250199900_bib19","author":"Kingma","year":"2014","journal-title":"Adam: A method for stochastic optimization."},{"key":"2024082018250199900_bib20","author":"Kingma","year":"2013","journal-title":"Auto-encoding variational Bayes"},{"key":"2024082018250199900_bib21","article-title":"Automatic variational inference in Stan","volume-title":"Advances in neural information processing systems","author":"Kucukelbir","year":"2015"},{"issue":"3","key":"2024082018250199900_bib22","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1016\/S0893-6080(00)00098-8","article-title":"Bayesian approach for neural networks: Review and case studies","volume":"14","author":"Lampinen","year":"2001","journal-title":"Neural Networks"},{"key":"2024082018250199900_bib23","author":"Lin","year":"2021","journal-title":"Structured second-order methods via natural gradient descent."},{"key":"2024082018250199900_bib24","first-page":"6116","article-title":"Handling the positive-definite constraint in the Bayesian learning rule","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Lin","year":"2020"},{"key":"2024082018250199900_bib25","first-page":"217","article-title":"Black-box optimizer with stochastic implicit natural gradient","volume-title":"Machine learning and knowledge discovery in databases","author":"Lyu","year":"2021"},{"key":"2024082018250199900_bib26","author":"Mackay","year":"1992","journal-title":"Bayesian methods for adaptive models"},{"key":"2024082018250199900_bib27","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1088\/0954-898X_6_3_011","article-title":"Probable networks and plausible predictions: A review of practical Bayesian methods for supervised neural networks","volume":"6","author":"Mackay","year":"1995","journal-title":"Network: Computation in Neural Systems"},{"issue":"10","key":"2024082018250199900_bib28","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-023-10443-1","article-title":"Bayesian learning for neural networks: An algorithmic survey","volume":"56","author":"Magris","year":"2023","journal-title":"Artificial Intelligence Review"},{"key":"2024082018250199900_bib29","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1145\/3604237.3626863","article-title":"Variational inference for GARCH-family models","volume-title":"Proceedings of the Fourth ACM International Conference on AI in Finance","author":"Magris","year":"2023"},{"key":"2024082018250199900_bib30","author":"Magris","year":"2022","journal-title":"Quasi black-box variational inference with natural gradients for Bayesian learning"},{"issue":"6","key":"2024082018250199900_bib31","doi-asserted-by":"publisher","first-page":"1407","DOI":"10.1002\/for.2955","article-title":"Bayesian bilinear neural network for predicting the mid-price dynamics in limit-order book markets","volume":"42","author":"Magris","year":"2023","journal-title":"Journal of Forecasting"},{"issue":"1","key":"2024082018250199900_bib32","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1093\/biomet\/71.1.135","article-title":"Maximum likelihood estimation of models for residual covariance in spatial regression","volume":"71","author":"Mardia","year":"1984","journal-title":"Biometrika"},{"issue":"132","key":"2024082018250199900_bib33","first-page":"1","article-title":"Monte Carlo gradient estimation in machine learning","volume":"21","author":"Mohamed","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2024082018250199900_bib34","author":"Mroz","year":"1984","journal-title":"The sensitivity of an empirical model of married women\u2019s hours of work to economic and statistical assumptions"},{"issue":"8","key":"2024082018250199900_bib35","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1002\/for.2543","article-title":"Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods","volume":"37","author":"Ntakaris","year":"2018","journal-title":"Journal of Forecasting"},{"key":"2024082018250199900_bib36","first-page":"1","article-title":"Practical deep learning with Bayesian principles","volume-title":"Advances in neural information processing systems","author":"Osawa","year":"2019"},{"key":"2024082018250199900_bib37","doi-asserted-by":"publisher","DOI":"10.5555\/3042573.3042748","article-title":"Variational Bayesian inference with stochastic search","author":"Paisley","year":"2012","journal-title":"International Conference on Machine Learning"},{"issue":"1","key":"2024082018250199900_bib38","doi-asserted-by":"publisher","first-page":"246","DOI":"10.1016\/j.jeconom.2010.03.034","article-title":"Volatility forecast comparison using imperfect volatility proxies","volume":"160","author":"Patton","year":"2011","journal-title":"Journal of Econometrics"},{"key":"2024082018250199900_bib39","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/B978-0-12-814725-2.00010-8","article-title":"Manifold-valued image processing with SPD matrices","volume-title":"Riemannian geometric statistics in medical image analysis","author":"Pennec","year":"2020"},{"key":"2024082018250199900_bib40","first-page":"814","article-title":"Black box variational inference","volume-title":"Proceedings of the 17th International Conference on Artificial Intelligence and Statistics","author":"Ranganath","year":"2014"},{"issue":"3","key":"2024082018250199900_bib41","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1214\/aoms\/1177729586","article-title":"A stochastic approximation method","volume":"22","author":"Robbins","year":"1951","journal-title":"Annals of Mathematical Statistics"},{"key":"2024082018250199900_bib42","first-page":"1218","article-title":"Markov chain Monte Carlo and variational inference: Bridging the gap","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Salimans","year":"2015"},{"key":"2024082018250199900_bib43","author":"Salimans","year":"2014","journal-title":"On using control variates with stochastic approximation for variational Bayes and its connection to stochastic linear regression."},{"key":"2024082018250199900_bib44","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1613\/jair.251","article-title":"Mean field theory for sigmoid belief networks","volume":"4","author":"Saul","year":"1996","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2024082018250199900_bib45","author":"Tan","year":"2021","journal-title":"Natural gradient updates for Cholesky factor in gaussian and structured variational inference."},{"key":"2024082018250199900_bib46","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/978-3-540-71297-8_1","article-title":"An introduction to univariate Garch models","volume-title":"Handbook of financial time series","author":"Ter\u00e4svirta","year":"2009"},{"issue":"5","key":"2024082018250199900_bib47","doi-asserted-by":"publisher","first-page":"1407","DOI":"10.1109\/TNNLS.2018.2869225","article-title":"Temporal attention-augmented bilinear network for financial time-series data analysis","volume":"30","author":"Tran","year":"2019","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"2024082018250199900_bib48","author":"Tran","year":"2021","journal-title":"A practical tutorial on variational Bayes."},{"issue":"6","key":"2024082018250199900_bib49","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1007\/s11222-021-10047-1","article-title":"Variational Bayes on manifolds","volume":"31","author":"Tran","year":"2021","journal-title":"Statistics and Computing"},{"key":"2024082018250199900_bib50","first-page":"19","article-title":"Boosting black-box variational inference by incorporating the natural gradient","volume-title":"Proceedings of the 24th International Conference on Pattern Recognition","author":"Trusheim","year":"2018"},{"issue":"1\u20132","key":"2024082018250199900_bib51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/2200000001","article-title":"Graphical models, exponential families, and variational inference","volume":"1","author":"Wainwright","year":"2008","journal-title":"Foundations and Trends in Machine Learning"},{"issue":"1","key":"2024082018250199900_bib52","first-page":"949","article-title":"Natural evolution strategies","volume":"15","author":"Wierstra","year":"2014","journal-title":"Journal of Machine Learning Research"},{"key":"2024082018250199900_bib53","first-page":"2711","article-title":"Variance reduction properties of the reparameterization trick","volume-title":"Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics","author":"Xu","year":"2019"},{"key":"2024082018250199900_bib54","first-page":"5852","article-title":"Noisy natural gradient as variational inference","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zhang","year":"2018"},{"issue":"472","key":"2024082018250199900_bib55","doi-asserted-by":"publisher","first-page":"1394","DOI":"10.1198\/016214505000000169","article-title":"A tale of two time scales: Determining integrated volatility with noisy high-frequency data","volume":"100","author":"Zhang","year":"2005","journal-title":"Journal of the American Statistical Association"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/9\/1744\/2466011\/neco_a_01686.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/9\/1744\/2466011\/neco_a_01686.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T18:26:05Z","timestamp":1724178365000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/36\/9\/1744\/124062\/Manifold-Gaussian-Variational-Bayes-on-the"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,19]]},"references-count":55,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,8,19]]},"published-print":{"date-parts":[[2024,8,19]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01686","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,8,19]]}}}