{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T19:27:47Z","timestamp":1773775667365,"version":"3.50.1"},"reference-count":51,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T00:00:00Z","timestamp":1734307200000},"content-version":"vor","delay-in-days":15,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T00:00:00Z","timestamp":1734307200000},"content-version":"tdm","delay-in-days":15,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"crossref","award":["22K11934"],"award-info":[{"award-number":["22K11934"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Variational inference with normalizing flows is an increasingly popular alternative to MCMC methods. In particular, normalizing flows based on affine coupling layers (Real NVPs) are frequently used due to their good empirical performance. In theory, increasing the depth of normalizing flows should lead to more accurate posterior approximations. However, in practice, training deep normalizing flows for approximating high-dimensional posterior distributions is often infeasible due to the high variance of the stochastic gradients. In this work, we show that previous methods for stabilizing the variance of stochastic gradient descent can be insufficient to achieve stable training of Real NVPs. As the source of the problem, we identify that, during training, samples often exhibit unusual high values. As a remedy, we propose a combination of two methods: (1) soft-thresholding of the scale in Real NVPs, and (2) a bijective soft log transformation of the samples. We evaluate these and other previously proposed modification on several challenging target distributions, including a high-dimensional horseshoe logistic regression model. Our experiments show that with our modifications, stable training of Real NVPs for posteriors with several thousand dimensions and heavy tails is possible, allowing for more accurate marginal likelihood estimation via importance sampling. Moreover, we evaluate several common training techniques and architecture choices and provide practical advise for training Real NVPs for high-dimensional variational inference. Finally, we also provide new empirical and theoretical justification that optimizing the evidence lower bound of normalizing flows leads to good posterior distribution coverage.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad9a39","type":"journal-article","created":{"date-parts":[[2024,12,3]],"date-time":"2024-12-03T23:01:27Z","timestamp":1733266887000},"page":"045066","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Stabilizing training of affine coupling layers for high-dimensional variational inference"],"prefix":"10.1088","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1123-4369","authenticated-orcid":true,"given":"Daniel","family":"Andrade","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2024,12,16]]},"reference":[{"key":"mlstad9a39bib1","doi-asserted-by":"publisher","first-page":"6745","DOI":"10.1073\/pnas.96.12.6745","article-title":"Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays","volume":"96","author":"Alon","year":"1999","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstad9a39bib2","article-title":"Loft-stable training of normalizing flows for variational inference","author":"Andrade","year":"2023"},{"key":"mlstad9a39bib3","first-page":"pp 318","article-title":"Annealed flow transport Monte Carlo","author":"Arbel","year":"2021"},{"key":"mlstad9a39bib4","article-title":"Guided image generation with conditional invertible neural networks","author":"Ardizzone","year":"2019"},{"key":"mlstad9a39bib5","first-page":"pp 573","article-title":"Invertible residual networks","author":"Behrmann","year":"2019"},{"key":"mlstad9a39bib6","first-page":"pp 1792","article-title":"Understanding and mitigating exploding inverses in invertible neural networks","author":"Behrmann","year":"2021"},{"key":"mlstad9a39bib7","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1080\/01621459.2017.1285773","article-title":"Variational inference: a review for statisticians","volume":"112","author":"Blei","year":"2017","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstad9a39bib8","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1093\/biomet\/asq017","article-title":"The horseshoe estimator for sparse signals","volume":"97","author":"Carvalho","year":"2010","journal-title":"Biometrika"},{"key":"mlstad9a39bib9","article-title":"Neural ordinary differential equations","volume":"vol 31","author":"Chen","year":"2018"},{"key":"mlstad9a39bib10","first-page":"65","article-title":"The practical implementation of Bayesian model selection","volume":"38","author":"Chipman","year":"2001","journal-title":"Lect. Notes-Monogr. Ser."},{"key":"mlstad9a39bib11","doi-asserted-by":"publisher","first-page":"1587","DOI":"10.1080\/01621459.2022.2087659","article-title":"An invitation to sequential Monte Carlo samplers","volume":"117","author":"Dai","year":"2022","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstad9a39bib12","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1080\/01621459.2020.1811100","article-title":"Monte Carlo approximation of Bayes factors via mixing with surrogate distributions","volume":"117","author":"Dai","year":"2022","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstad9a39bib13","first-page":"pp 10961","article-title":"Robust, accurate stochastic optimization for variational inference","volume":"vol 33","author":"Dhaka","year":"2020"},{"key":"mlstad9a39bib14","first-page":"pp 7787","article-title":"Challenges and opportunities in high dimensional variational inference","volume":"vol 34","author":"Dhaka","year":"2021"},{"key":"mlstad9a39bib15","article-title":"Density estimation using Real NVP","author":"Dinh","year":"2017"},{"key":"mlstad9a39bib16","article-title":"On the universality of volume-preserving and coupling-based normalizing flows","author":"Draxler","year":"2024"},{"key":"mlstad9a39bib17","article-title":"Neural spline flows","volume":"vol 32","author":"Durkan","year":"2019"},{"key":"mlstad9a39bib18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10182-024-00497-z","article-title":"Bernstein flows for flexible posteriors in variational Bayes","volume":"108","author":"D\u00fcrr","year":"2024","journal-title":"AStA Adv. Stat. Anal."},{"key":"mlstad9a39bib19","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6420\/abe928","article-title":"Stabilizing invertible neural networks using mixture models","volume":"37","author":"Hagemann","year":"2021","journal-title":"Inverse Problems"},{"key":"mlstad9a39bib20","article-title":"Augmented normalizing flows: bridging the gap between generative flows and latent variable models","author":"Huang","year":"2020"},{"key":"mlstad9a39bib21","first-page":"1","article-title":"universal approximation property of invertible neural networks","volume":"24","author":"Ishikawa","year":"2023","journal-title":"J. Mach. Learn. Res."},{"key":"mlstad9a39bib22","first-page":"pp 4673","article-title":"Tails of Lipschitz triangular flows","author":"Jaini","year":"2020"},{"key":"mlstad9a39bib23","article-title":"Variational refinement for importance sampling using the forward Kullback-Leibler divergence","author":"Jerfel","year":"2021"},{"key":"mlstad9a39bib24","doi-asserted-by":"publisher","first-page":"773","DOI":"10.1080\/01621459.1995.10476572","article-title":"Bayes factors","volume":"90","author":"Kass","year":"1995","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstad9a39bib25","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2015"},{"key":"mlstad9a39bib26","article-title":"Auto-encoding variational {Bayes }","author":"Kingma","year":"2014"},{"key":"mlstad9a39bib27","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1198\/tast.2009.0030","article-title":"On the assessment of Monte Carlo error in simulation-based statistical analyses","volume":"63","author":"Koehler","year":"2009","journal-title":"Am. Stat."},{"key":"mlstad9a39bib28","first-page":"pp 5628","article-title":"Representational aspects of depth and conditioning in normalizing flows","author":"Koehler","year":"2021"},{"key":"mlstad9a39bib29","doi-asserted-by":"publisher","first-page":"3918","DOI":"10.1093\/bioinformatics\/btac416","article-title":"Variational Bayes for high-dimensional proportional hazards models with applications within gene expression","volume":"38","author":"Komodromos","year":"2022","journal-title":"Bioinformatics"},{"key":"mlstad9a39bib30","first-page":"1","article-title":"Automatic differentiation variational inference","volume":"18","author":"Kucukelbir","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"mlstad9a39bib31","first-page":"pp 12700","article-title":"universal approximation using well-conditioned normalizing flows","volume":"vol 34","author":"Lee","year":"2021"},{"key":"mlstad9a39bib32","first-page":"pp 13257","article-title":"Fat\u2013tailed variational inference with anisotropic tail adaptive flows","author":"Liang","year":"2022"},{"key":"mlstad9a39bib33","first-page":"1","article-title":"Empirical priors for prediction in sparse high-dimensional linear regression","volume":"21","author":"Martin","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"mlstad9a39bib34","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1023\/A:1008923215028","article-title":"Annealed importance sampling","volume":"11","author":"Neal","year":"2001","journal-title":"Stat. Comput."},{"key":"mlstad9a39bib35","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1214\/aos\/1056562461","article-title":"Slice sampling","volume":"31","author":"Neal","year":"2003","journal-title":"Ann. Stat."},{"key":"mlstad9a39bib36","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.108.114501","article-title":"Detecting and mitigating mode-collapse for flow-based sampling of lattice field theories","volume":"108","author":"Nicoli","year":"2023","journal-title":"Phys. Rev. D"},{"key":"mlstad9a39bib37","first-page":"1","article-title":"Normalizing flows for probabilistic modeling and inference","volume":"22","author":"Papamakarios","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"mlstad9a39bib38","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1198\/016214508000000337","article-title":"The Bayesian Lasso","volume":"103","author":"Park","year":"2008","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstad9a39bib39","volume":"vol 32","author":"Paszke","year":"2019"},{"key":"mlstad9a39bib40","article-title":"Composable effects for flexible and accelerated probabilistic programming in NumPyro","author":"Phan","year":"2019"},{"key":"mlstad9a39bib41","article-title":"Robust normalizing flows using Bernstein-type polynomials","author":"Ramasinghe","year":"2021"},{"key":"mlstad9a39bib42","first-page":"pp 814","article-title":"Black box variational inference","author":"Ranganath","year":"2014"},{"key":"mlstad9a39bib43","first-page":"pp 1530","article-title":"Variational inference with normalizing flows","author":"Rezende","year":"2015"},{"key":"mlstad9a39bib44","article-title":"Sticking the landing: simple, lower-variance gradient estimators for variational inference","volume":"vol 30","author":"Roeder","year":"2017"},{"key":"mlstad9a39bib45","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005752","article-title":"mixOmics: an R package for \u2018omics feature selection and multiple data integration","volume":"13","author":"Rohart","year":"2017","journal-title":"PLoS Comput. Biol."},{"key":"mlstad9a39bib46","first-page":"pp 10766","article-title":"Can push-forward generative models fit multimodal distributions?","volume":"vol 35","author":"Salmona","year":"2022"},{"key":"mlstad9a39bib47","doi-asserted-by":"crossref","DOI":"10.21105\/joss.05361","article-title":"normflows: a PyTorch package for normalizing flows","author":"Stimper","year":"2023"},{"key":"mlstad9a39bib48","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ac9455","article-title":"Gradients should stay on path: better estimators of the reverse-and forward KL divergence for normalizing flows","volume":"3","author":"Vaitl","year":"2022a","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad9a39bib49","first-page":"pp 21945","article-title":"Path-gradient estimators for continuous normalizing flows","author":"Vaitl","year":"2022b"},{"key":"mlstad9a39bib50","article-title":"Fast and unified path gradient estimators for normalizing flows","author":"Vaitl","year":"2024"},{"key":"mlstad9a39bib51","doi-asserted-by":"publisher","first-page":"2008","DOI":"10.1109\/TPAMI.2018.2889774","article-title":"Advances in variational inference","volume":"41","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T13:11:09Z","timestamp":1734354669000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad9a39"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,1]]},"references-count":51,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,12,16]]},"published-print":{"date-parts":[[2024,12,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad9a39","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,1]]},"assertion":[{"value":"Stabilizing training of affine coupling layers for high-dimensional variational inference","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2024-06-10","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-12-03","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-12-16","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}